tidbstandalone/TIDB_CLOUD_MIGRATION.md

TiDB Cloud to Local TiDB Migration Guide

This guide provides officially recommended approaches for migrating data from TiDB Cloud to a local TiDB instance, since TiDB Data Migration (DM) cannot be used with TiDB Cloud Serverless.

Why DM Doesn't Work with TiDB Cloud

TiDB Data Migration (DM) fails with TiDB Cloud because:

1. No MySQL binlog support - TiDB Cloud Serverless doesn't expose binlog in the traditional MySQL way
2. binlog_format is STATEMENT - DM requires ROW format
3. TiDB explicitly not supported as upstream - DM is designed for MySQL/MariaDB → TiDB, not TiDB → TiDB

Approach 1: Console Export + SQL Import (Simplest)

Export from TiDB Cloud

1. Using TiDB Cloud Console:

   • Navigate to your cluster in the TiDB Cloud Console
   • Go to Data > Import
   • Click "Export Data to" > "Local File"
   • Select databases/tables to export
   • Choose format (SQL recommended for small datasets)
   • Click "Export"

2. Using TiDB Cloud CLI:

   # Create export task
   ticloud serverless export create -c <cluster-id>
   
   # Download exported data
   ticloud serverless export download -c <cluster-id> -e <export-id>
   

Import to Local TiDB

# Import SQL file
mysql -h 127.0.0.1 -P 4000 -u root < exported_data.sql

# Or for CSV files
mysql -h 127.0.0.1 -P 4000 -u root -e "
LOAD DATA LOCAL INFILE 'table_data.csv' 
INTO TABLE your_table 
FIELDS TERMINATED BY ',' 
ENCLOSED BY '\"' 
LINES TERMINATED BY '\n'
IGNORE 1 ROWS;"

Approach 2: Dumpling + TiDB Lightning (For Larger Datasets)

Prerequisites

Install TiDB tools:

# Install TiUP
curl --proto '=https' --tlsv1.2 -sSf https://tiup-mirrors.pingcap.com/install.sh | sh
source ~/.bash_profile

# Install tools
tiup install dumpling tidb-lightning

Export with Dumpling

# Export data from TiDB Cloud
dumpling \
  -u {TEST_DB_USER} \
  -p {TEST_DB_PASSWORD} \
  -P {TEST_DB_PORT} \
  -h {TEST_DB_HOST} \
  -o /tmp/tidb-export \
  --filetype sql \
  -r 20000 \
  -F 256MiB

Import with TiDB Lightning

1. Create configuration file (lightning.toml):

   [lightning]
   level = "info"
   file = "tidb-lightning.log"
   
   [tikv-importer]
   backend = "local"
   sorted-kv-dir = "/tmp/sorted-kv-dir"
   
   [mydumper]
   data-source-dir = "/tmp/tidb-export"
   no-schema = false
   
   [tidb]
   host = "127.0.0.1"
   port = 4000
   user = "root"
   password = ""
   status-port = 10080
   pd-addr = "127.0.0.1:2379"
   

2. Run TiDB Lightning:

   tidb-lightning -config lightning.toml
   

Approach 3: Periodic Sync Script

Create a script for periodic data sync:

Export Script (export-cloud.sh)

#!/bin/bash

# Source .env file
source .env

# Export data using mysqldump (built-in tool)
mysqldump \
  -h $TEST_DB_HOST \
  -P $TEST_DB_PORT \
  -u $TEST_DB_USER \
  -p$TEST_DB_PASSWORD \
  --single-transaction \
  --routines \
  --triggers \
  $DATABASE_NAME \
  $TABLES > /tmp/cloud-export.sql

echo "Export completed: /tmp/cloud-export.sql"

Import Script (import-local.sh)

#!/bin/bash

# Import to local TiDB
mysql -h 127.0.0.1 -P 4000 -u root < /tmp/cloud-export.sql

echo "Import completed to local TiDB"

Combined Sync Script (sync-data.sh)

#!/bin/bash

echo "🔄 Syncing data from TiDB Cloud to local TiDB..."

# Export from cloud
./export-cloud.sh

# Import to local
./import-local.sh

echo "✅ Sync completed!"

Approach 4: Application-Level Sync (For Continuous Updates)

For real-time sync, implement in your application:

# Example Python script for selective sync
import mysql.connector

# Connect to both databases
cloud_db = mysql.connector.connect(
    host="gateway01.ap-northeast-1.prod.aws.tidbcloud.com",
    port=4000,
    user="3mmwxY44wQF4L6P.root",
    password="JQ8wbPsXfx7xJOR5",
    database="workflow_local"
)

local_db = mysql.connector.connect(
    host="127.0.0.1",
    port=4000,
    user="root",
    password="",
    database="workflow_local"
)

# Sync specific tables
def sync_table(table_name):
    # Get data from cloud
    cloud_cursor = cloud_db.cursor()
    cloud_cursor.execute(f"SELECT * FROM {table_name}")
    rows = cloud_cursor.fetchall()
    
    # Clear and insert into local
    local_cursor = local_db.cursor()
    local_cursor.execute(f"DELETE FROM {table_name}")
    
    if rows:
        placeholders = ','.join(['%s'] * len(rows[0]))
        local_cursor.executemany(
            f"INSERT INTO {table_name} VALUES ({placeholders})", 
            rows
        )
    
    local_db.commit()
    print(f"Synced {len(rows)} rows to {table_name}")

# Sync your tables
sync_table("plans")

Recommended Solution for Your Setup

For development purposes, I recommend:

1. Use Approach 1 (Console Export + SQL Import) for simplicity
2. Create helper scripts for periodic sync
3. Consider application-level sync for real-time needs

Quick Setup

Create these helper scripts in your project:

# Make scripts executable
chmod +x sync-data.sh export-cloud.sh import-local.sh

# Run sync
./sync-data.sh

Limitations and Considerations

TiDB Cloud Serverless Limitations

• No traditional MySQL binlog access
• Limited to export/import methods
• No direct replication support in most plans

Performance Considerations

• Full table exports can be slow for large datasets
• Network bandwidth affects sync speed
• Consider incremental exports for large tables

Security Notes

• Store credentials securely (use .env file)
• Use TLS connections when possible
• Rotate credentials regularly

Troubleshooting

Connection Issues

# Test connection to TiDB Cloud
mysql -h $TEST_DB_HOST -P $TEST_DB_PORT -u $TEST_DB_USER -p

# Test connection to local TiDB
mysql -h 127.0.0.1 -P 4000 -u root

Export Errors

• Ensure user has SELECT privileges
• Check network connectivity
• Verify table existence

Import Errors

• Check schema compatibility
• Ensure sufficient disk space
• Verify TiDB is running

References

• TiDB Cloud Export Documentation
• TiDB Migration Tools Overview
• Dumpling Documentation
• TiDB Lightning Documentation

For production use cases, contact TiDB Cloud Support to discuss available replication options for your specific plan.