275 lines
6.4 KiB
Markdown
275 lines
6.4 KiB
Markdown
# TiDB Cloud to Local TiDB Migration Guide
|
|
|
|
This guide provides officially recommended approaches for migrating data from TiDB Cloud to a local TiDB instance, since TiDB Data Migration (DM) cannot be used with TiDB Cloud Serverless.
|
|
|
|
## Why DM Doesn't Work with TiDB Cloud
|
|
|
|
TiDB Data Migration (DM) fails with TiDB Cloud because:
|
|
|
|
1. **No MySQL binlog support** - TiDB Cloud Serverless doesn't expose binlog in the traditional MySQL way
|
|
2. **binlog_format is STATEMENT** - DM requires ROW format
|
|
3. **TiDB explicitly not supported as upstream** - DM is designed for MySQL/MariaDB → TiDB, not TiDB → TiDB
|
|
|
|
## Approach 1: Console Export + SQL Import (Simplest)
|
|
|
|
### Export from TiDB Cloud
|
|
|
|
1. **Using TiDB Cloud Console**:
|
|
- Navigate to your cluster in the TiDB Cloud Console
|
|
- Go to Data > Import
|
|
- Click "Export Data to" > "Local File"
|
|
- Select databases/tables to export
|
|
- Choose format (SQL recommended for small datasets)
|
|
- Click "Export"
|
|
|
|
2. **Using TiDB Cloud CLI**:
|
|
```bash
|
|
# Create export task
|
|
ticloud serverless export create -c <cluster-id>
|
|
|
|
# Download exported data
|
|
ticloud serverless export download -c <cluster-id> -e <export-id>
|
|
```
|
|
|
|
### Import to Local TiDB
|
|
|
|
```bash
|
|
# Import SQL file
|
|
mysql -h 127.0.0.1 -P 4000 -u root < exported_data.sql
|
|
|
|
# Or for CSV files
|
|
mysql -h 127.0.0.1 -P 4000 -u root -e "
|
|
LOAD DATA LOCAL INFILE 'table_data.csv'
|
|
INTO TABLE your_table
|
|
FIELDS TERMINATED BY ','
|
|
ENCLOSED BY '\"'
|
|
LINES TERMINATED BY '\n'
|
|
IGNORE 1 ROWS;"
|
|
```
|
|
|
|
## Approach 2: Dumpling + TiDB Lightning (For Larger Datasets)
|
|
|
|
### Prerequisites
|
|
|
|
Install TiDB tools:
|
|
```bash
|
|
# Install TiUP
|
|
curl --proto '=https' --tlsv1.2 -sSf https://tiup-mirrors.pingcap.com/install.sh | sh
|
|
source ~/.bash_profile
|
|
|
|
# Install tools
|
|
tiup install dumpling tidb-lightning
|
|
```
|
|
|
|
### Export with Dumpling
|
|
|
|
```bash
|
|
# Export data from TiDB Cloud
|
|
dumpling \
|
|
-u {TEST_DB_USER} \
|
|
-p {TEST_DB_PASSWORD} \
|
|
-P {TEST_DB_PORT} \
|
|
-h {TEST_DB_HOST} \
|
|
-o /tmp/tidb-export \
|
|
--filetype sql \
|
|
-r 20000 \
|
|
-F 256MiB
|
|
```
|
|
|
|
### Import with TiDB Lightning
|
|
|
|
1. **Create configuration file** (`lightning.toml`):
|
|
```toml
|
|
[lightning]
|
|
level = "info"
|
|
file = "tidb-lightning.log"
|
|
|
|
[tikv-importer]
|
|
backend = "local"
|
|
sorted-kv-dir = "/tmp/sorted-kv-dir"
|
|
|
|
[mydumper]
|
|
data-source-dir = "/tmp/tidb-export"
|
|
no-schema = false
|
|
|
|
[tidb]
|
|
host = "127.0.0.1"
|
|
port = 4000
|
|
user = "root"
|
|
password = ""
|
|
status-port = 10080
|
|
pd-addr = "127.0.0.1:2379"
|
|
```
|
|
|
|
2. **Run TiDB Lightning**:
|
|
```bash
|
|
tidb-lightning -config lightning.toml
|
|
```
|
|
|
|
## Approach 3: Periodic Sync Script
|
|
|
|
Create a script for periodic data sync:
|
|
|
|
### Export Script (`export-cloud.sh`)
|
|
```bash
|
|
#!/bin/bash
|
|
|
|
# Source .env file
|
|
source .env
|
|
|
|
# Export data using mysqldump (built-in tool)
|
|
mysqldump \
|
|
-h $TEST_DB_HOST \
|
|
-P $TEST_DB_PORT \
|
|
-u $TEST_DB_USER \
|
|
-p$TEST_DB_PASSWORD \
|
|
--single-transaction \
|
|
--routines \
|
|
--triggers \
|
|
$DATABASE_NAME \
|
|
$TABLES > /tmp/cloud-export.sql
|
|
|
|
echo "Export completed: /tmp/cloud-export.sql"
|
|
```
|
|
|
|
### Import Script (`import-local.sh`)
|
|
```bash
|
|
#!/bin/bash
|
|
|
|
# Import to local TiDB
|
|
mysql -h 127.0.0.1 -P 4000 -u root < /tmp/cloud-export.sql
|
|
|
|
echo "Import completed to local TiDB"
|
|
```
|
|
|
|
### Combined Sync Script (`sync-data.sh`)
|
|
```bash
|
|
#!/bin/bash
|
|
|
|
echo "🔄 Syncing data from TiDB Cloud to local TiDB..."
|
|
|
|
# Export from cloud
|
|
./export-cloud.sh
|
|
|
|
# Import to local
|
|
./import-local.sh
|
|
|
|
echo "✅ Sync completed!"
|
|
```
|
|
|
|
## Approach 4: Application-Level Sync (For Continuous Updates)
|
|
|
|
For real-time sync, implement in your application:
|
|
|
|
```python
|
|
# Example Python script for selective sync
|
|
import mysql.connector
|
|
|
|
# Connect to both databases
|
|
cloud_db = mysql.connector.connect(
|
|
host="gateway01.ap-northeast-1.prod.aws.tidbcloud.com",
|
|
port=4000,
|
|
user="3mmwxY44wQF4L6P.root",
|
|
password="JQ8wbPsXfx7xJOR5",
|
|
database="workflow_local"
|
|
)
|
|
|
|
local_db = mysql.connector.connect(
|
|
host="127.0.0.1",
|
|
port=4000,
|
|
user="root",
|
|
password="",
|
|
database="workflow_local"
|
|
)
|
|
|
|
# Sync specific tables
|
|
def sync_table(table_name):
|
|
# Get data from cloud
|
|
cloud_cursor = cloud_db.cursor()
|
|
cloud_cursor.execute(f"SELECT * FROM {table_name}")
|
|
rows = cloud_cursor.fetchall()
|
|
|
|
# Clear and insert into local
|
|
local_cursor = local_db.cursor()
|
|
local_cursor.execute(f"DELETE FROM {table_name}")
|
|
|
|
if rows:
|
|
placeholders = ','.join(['%s'] * len(rows[0]))
|
|
local_cursor.executemany(
|
|
f"INSERT INTO {table_name} VALUES ({placeholders})",
|
|
rows
|
|
)
|
|
|
|
local_db.commit()
|
|
print(f"Synced {len(rows)} rows to {table_name}")
|
|
|
|
# Sync your tables
|
|
sync_table("plans")
|
|
```
|
|
|
|
## Recommended Solution for Your Setup
|
|
|
|
For development purposes, I recommend:
|
|
|
|
1. **Use Approach 1** (Console Export + SQL Import) for simplicity
|
|
2. **Create helper scripts** for periodic sync
|
|
3. **Consider application-level sync** for real-time needs
|
|
|
|
### Quick Setup
|
|
|
|
Create these helper scripts in your project:
|
|
|
|
```bash
|
|
# Make scripts executable
|
|
chmod +x sync-data.sh export-cloud.sh import-local.sh
|
|
|
|
# Run sync
|
|
./sync-data.sh
|
|
```
|
|
|
|
## Limitations and Considerations
|
|
|
|
### TiDB Cloud Serverless Limitations
|
|
- No traditional MySQL binlog access
|
|
- Limited to export/import methods
|
|
- No direct replication support in most plans
|
|
|
|
### Performance Considerations
|
|
- Full table exports can be slow for large datasets
|
|
- Network bandwidth affects sync speed
|
|
- Consider incremental exports for large tables
|
|
|
|
### Security Notes
|
|
- Store credentials securely (use .env file)
|
|
- Use TLS connections when possible
|
|
- Rotate credentials regularly
|
|
|
|
## Troubleshooting
|
|
|
|
### Connection Issues
|
|
```bash
|
|
# Test connection to TiDB Cloud
|
|
mysql -h $TEST_DB_HOST -P $TEST_DB_PORT -u $TEST_DB_USER -p
|
|
|
|
# Test connection to local TiDB
|
|
mysql -h 127.0.0.1 -P 4000 -u root
|
|
```
|
|
|
|
### Export Errors
|
|
- Ensure user has SELECT privileges
|
|
- Check network connectivity
|
|
- Verify table existence
|
|
|
|
### Import Errors
|
|
- Check schema compatibility
|
|
- Ensure sufficient disk space
|
|
- Verify TiDB is running
|
|
|
|
## References
|
|
|
|
- [TiDB Cloud Export Documentation](https://docs.pingcap.com/tidbcloud/serverless-export/)
|
|
- [TiDB Migration Tools Overview](https://docs.pingcap.com/tidb/stable/migration-tools)
|
|
- [Dumpling Documentation](https://docs.pingcap.com/tidb/stable/dumpling-overview)
|
|
- [TiDB Lightning Documentation](https://docs.pingcap.com/tidb/stable/tidb-lightning-overview)
|
|
|
|
For production use cases, contact TiDB Cloud Support to discuss available replication options for your specific plan. |