docs(readme): remove DM from TiDB local dev environment and simplify setup
- Update README to reflect minimal TiDB standalone mode without DM components - Remove DM master and worker service definitions, commands, and configs - Delete all sync scripts, guides, and troubleshooting related to DM - Simplify architecture diagram and component explanations accordingly - Adjust quick start instructions to focus on TiDB standalone usage only - Remove dependency on .env and sync configuration files - Clean up docker-compose.yml to run only TiDB service in standalone mode - Remove all references to data synchronization from TiDB Cloud or test environments - Delete SYNC_GUIDE.md, TIDB_CLOUD_MIGRATION.md, and TROUBLESHOOTING.md files as obsolete
This commit is contained in:
parent
9eb1428779
commit
3e5524e1a3
149
README.md
149
README.md
|
|
@ -1,43 +1,22 @@
|
||||||
# TiDB Local Development Environment
|
# TiDB Local Development Environment
|
||||||
|
|
||||||
A minimal TiDB instance with Data Migration (DM) for syncing data from test environments.
|
A minimal TiDB instance for local development.
|
||||||
|
|
||||||
## Architecture
|
## Architecture
|
||||||
|
|
||||||
```
|
```
|
||||||
┌─────────────────────────────────────────────────────────────┐
|
┌─────────────────────────────────────────────────┐
|
||||||
│ Your macOS (OrbStack) │
|
│ Your macOS (OrbStack) │
|
||||||
│ │
|
│ │
|
||||||
│ ┌──────────────┐ ┌──────────────┐ │
|
│ ┌──────────────┐ ┌──────────────┐ │
|
||||||
│ │ DataGrip │──────▶│ TiDB │ │
|
│ │ DataGrip │──────▶│ TiDB │ │
|
||||||
│ │ (port 4000) │ │ (Standalone) │ │
|
│ │ (port 4000) │ │ (Standalone) │ │
|
||||||
│ └──────────────┘ └───────▲───────┘ │
|
│ └──────────────┘ └──────────────┘ │
|
||||||
│ │ │
|
└─────────────────────────────────────────────────┘
|
||||||
│ ┌───────┴────────┐ │
|
|
||||||
│ │ DM Worker │ │
|
|
||||||
│ │ (Sync Engine) │ │
|
|
||||||
│ └───────▲────────┘ │
|
|
||||||
│ │ │
|
|
||||||
│ ┌───────┴────────┐ │
|
|
||||||
│ │ DM Master │ │
|
|
||||||
│ │ (Orchestrator) │ │
|
|
||||||
│ └───────▲────────┘ │
|
|
||||||
│ │ │
|
|
||||||
└────────────────────────────────┼─────────────────────────────┘
|
|
||||||
│
|
|
||||||
(Continuous Sync)
|
|
||||||
│
|
|
||||||
┌────────────▼─────────────┐
|
|
||||||
│ Test TiDB Instance │
|
|
||||||
│ (Remote Environment) │
|
|
||||||
└──────────────────────────┘
|
|
||||||
```
|
```
|
||||||
|
|
||||||
**Components:**
|
**Components:**
|
||||||
- **TiDB (Standalone Mode)**: Runs with embedded storage (unistore), no separate PD/TiKV needed
|
- **TiDB (Standalone Mode)**: Runs with embedded storage (unistore), no separate PD/TiKV needed
|
||||||
- **DM Master**: Manages data migration tasks
|
|
||||||
- **DM Worker**: Executes the actual data synchronization from test to local
|
|
||||||
- **DataGrip/MySQL Clients**: Connect directly to TiDB on port 4000
|
|
||||||
|
|
||||||
## Quick Reference
|
## Quick Reference
|
||||||
|
|
||||||
|
|
@ -45,24 +24,15 @@ A minimal TiDB instance with Data Migration (DM) for syncing data from test envi
|
||||||
| Service | Host | Port | User | Password |
|
| Service | Host | Port | User | Password |
|
||||||
|---------|------|------|------|----------|
|
|---------|------|------|------|----------|
|
||||||
| TiDB | `127.0.0.1` | `4000` | `root` | _(empty)_ |
|
| TiDB | `127.0.0.1` | `4000` | `root` | _(empty)_ |
|
||||||
| DM Master | `127.0.0.1` | `8261` | - | - |
|
|
||||||
|
|
||||||
### Useful Commands
|
### Useful Commands
|
||||||
```bash
|
``bash
|
||||||
# Start environment (TiDB only, no working sync)
|
# Start environment
|
||||||
./start.sh
|
./start.sh
|
||||||
|
|
||||||
# Test connection
|
# Test connection
|
||||||
./test-connection.sh
|
./test-connection.sh
|
||||||
|
|
||||||
# NEW: Sync data from TiDB Cloud to local
|
|
||||||
./sync-data.sh
|
|
||||||
|
|
||||||
# Check sync status (deprecated - DM doesn't work with TiDB Cloud)
|
|
||||||
# ./status.sh
|
|
||||||
# or use the sync control script:
|
|
||||||
# ./sync-control.sh status
|
|
||||||
|
|
||||||
# Connect with MySQL client
|
# Connect with MySQL client
|
||||||
mysql -h 127.0.0.1 -P 4000 -u root
|
mysql -h 127.0.0.1 -P 4000 -u root
|
||||||
|
|
||||||
|
|
@ -79,7 +49,6 @@ For DataGrip/DBeaver setup, see [DATAGRIP_SETUP.md](DATAGRIP_SETUP.md)
|
||||||
|
|
||||||
- macOS with OrbStack (or Docker Desktop)
|
- macOS with OrbStack (or Docker Desktop)
|
||||||
- Docker Compose v2 (command: `docker compose`, not `docker-compose`)
|
- Docker Compose v2 (command: `docker compose`, not `docker-compose`)
|
||||||
- Access to test TiDB instance
|
|
||||||
|
|
||||||
**Check your setup:**
|
**Check your setup:**
|
||||||
```bash
|
```bash
|
||||||
|
|
@ -90,52 +59,17 @@ For DataGrip/DBeaver setup, see [DATAGRIP_SETUP.md](DATAGRIP_SETUP.md)
|
||||||
|
|
||||||
## Configuration
|
## Configuration
|
||||||
|
|
||||||
1. **Copy and edit `.env` file**:
|
No configuration needed for the basic setup.
|
||||||
```bash
|
|
||||||
cp .env.example .env
|
|
||||||
# Edit .env with your test database credentials
|
|
||||||
```
|
|
||||||
|
|
||||||
Required variables:
|
|
||||||
- `TEST_DB_HOST`: Your test TiDB host
|
|
||||||
- `TEST_DB_PORT`: Test TiDB port (default: 4000)
|
|
||||||
- `TEST_DB_USER`: Test database username
|
|
||||||
- `TEST_DB_PASSWORD`: Test database password
|
|
||||||
- `DATABASE_NAME`: Database to sync
|
|
||||||
- `TABLES`: Comma-separated list of tables (e.g., "table1,table2,table3")
|
|
||||||
|
|
||||||
## Usage
|
## Usage
|
||||||
|
|
||||||
### How the Sync Works
|
|
||||||
|
|
||||||
**Important Note**: The original DM-based sync approach doesn't work with TiDB Cloud Serverless because TiDB Cloud doesn't support the MySQL replication features that DM requires.
|
|
||||||
|
|
||||||
### Officially Recommended Approaches
|
|
||||||
|
|
||||||
See [TIDB_CLOUD_MIGRATION.md](TIDB_CLOUD_MIGRATION.md) for officially supported migration methods:
|
|
||||||
|
|
||||||
1. **Console Export + SQL Import** (simplest for development)
|
|
||||||
2. **Dumpling + TiDB Lightning** (for larger datasets)
|
|
||||||
3. **Periodic Sync Scripts** (created in this project)
|
|
||||||
4. **Application-Level Sync** (for real-time needs)
|
|
||||||
|
|
||||||
**For detailed sync operations, see [SYNC_GUIDE.md](SYNC_GUIDE.md)**
|
|
||||||
|
|
||||||
### Start the environment
|
### Start the environment
|
||||||
```bash
|
``bash
|
||||||
docker compose up -d
|
docker compose up -d
|
||||||
```
|
```
|
||||||
|
|
||||||
This will:
|
This will:
|
||||||
1. Start TiDB in standalone mode
|
1. Start TiDB in standalone mode
|
||||||
2. Start DM master and worker
|
|
||||||
3. Automatically configure the data source and sync task
|
|
||||||
4. Begin syncing data from test to local
|
|
||||||
|
|
||||||
### Check sync status
|
|
||||||
```bash
|
|
||||||
docker exec dm-master /dmctl --master-addr=dm-master:8261 query-status test-to-local
|
|
||||||
```
|
|
||||||
|
|
||||||
### Connect to local TiDB
|
### Connect to local TiDB
|
||||||
|
|
||||||
|
|
@ -153,17 +87,16 @@ mysql -h 127.0.0.1 -P 4000 -u root
|
||||||
See [DATAGRIP_SETUP.md](DATAGRIP_SETUP.md) for detailed client setup instructions.
|
See [DATAGRIP_SETUP.md](DATAGRIP_SETUP.md) for detailed client setup instructions.
|
||||||
|
|
||||||
### View logs
|
### View logs
|
||||||
```bash
|
```
|
||||||
# All services
|
# All services
|
||||||
docker compose logs -f
|
docker compose logs -f
|
||||||
|
|
||||||
# Specific service
|
# Specific service
|
||||||
docker compose logs -f tidb
|
docker compose logs -f tidb
|
||||||
docker compose logs -f dm-worker
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Stop the environment
|
### Stop the environment
|
||||||
```bash
|
``bash
|
||||||
docker compose down
|
docker compose down
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -172,76 +105,16 @@ docker compose down
|
||||||
docker compose down -v
|
docker compose down -v
|
||||||
```
|
```
|
||||||
|
|
||||||
## Manual DM Operations
|
|
||||||
|
|
||||||
### Check source configuration
|
|
||||||
```bash
|
|
||||||
docker exec dm-master /dmctl --master-addr=dm-master:8261 operate-source show
|
|
||||||
```
|
|
||||||
|
|
||||||
### Stop sync task
|
|
||||||
```bash
|
|
||||||
docker exec dm-master /dmctl --master-addr=dm-master:8261 stop-task test-to-local
|
|
||||||
```
|
|
||||||
|
|
||||||
### Start sync task
|
|
||||||
```bash
|
|
||||||
docker exec dm-master /dmctl --master-addr=dm-master:8261 start-task /configs/task.yaml
|
|
||||||
```
|
|
||||||
|
|
||||||
### Pause sync task
|
|
||||||
```bash
|
|
||||||
docker exec dm-master /dmctl --master-addr=dm-master:8261 pause-task test-to-local
|
|
||||||
```
|
|
||||||
|
|
||||||
### Resume sync task
|
|
||||||
```bash
|
|
||||||
docker exec dm-master /dmctl --master-addr=dm-master:8261 resume-task test-to-local
|
|
||||||
```
|
|
||||||
|
|
||||||
## Troubleshooting
|
## Troubleshooting
|
||||||
|
|
||||||
### Common Issues
|
|
||||||
|
|
||||||
For detailed troubleshooting, see [TROUBLESHOOTING.md](TROUBLESHOOTING.md)
|
|
||||||
|
|
||||||
### Quick Checks
|
|
||||||
|
|
||||||
#### TiDB health check failing
|
|
||||||
```bash
|
|
||||||
# Check if TiDB is healthy
|
|
||||||
docker ps | grep tidb
|
|
||||||
|
|
||||||
# Should show: (healthy)
|
|
||||||
# If not, check logs:
|
|
||||||
docker logs tidb
|
|
||||||
```
|
|
||||||
|
|
||||||
#### DM task fails to start
|
|
||||||
- Check if test database is accessible from container
|
|
||||||
- Verify credentials in `.env`
|
|
||||||
- Check logs: `docker-compose logs dm-worker`
|
|
||||||
|
|
||||||
### Tables not syncing
|
|
||||||
- Ensure tables exist in source database
|
|
||||||
- Verify table names in `TABLES` variable
|
|
||||||
- Check task status for specific errors
|
|
||||||
|
|
||||||
### TiDB connection issues
|
### TiDB connection issues
|
||||||
- Verify TiDB is running: `docker ps | grep tidb`
|
- Verify TiDB is running: `docker ps | grep tidb`
|
||||||
- Check health: `docker exec tidb mysql -h 127.0.0.1 -P 4000 -u root -e "SELECT 1"`
|
- Check health: `docker exec tidb mysql -h 127.0.0.1 -P 4000 -u root -e "SELECT 1"`
|
||||||
|
|
||||||
### Re-initialize DM configuration
|
|
||||||
```bash
|
|
||||||
docker compose up -d dm-init
|
|
||||||
```
|
|
||||||
|
|
||||||
## Resource Usage
|
## Resource Usage
|
||||||
|
|
||||||
Default resource limits (suitable for local development):
|
Default resource limits (suitable for local development):
|
||||||
- TiDB: 2 CPU, 2GB RAM
|
- TiDB: 2 CPU, 2GB RAM
|
||||||
- DM Worker: 1 CPU, 1GB RAM
|
|
||||||
- DM Master: 0.5 CPU, 512MB RAM
|
|
||||||
|
|
||||||
Adjust in `docker-compose.yml` if needed.
|
Adjust in `docker-compose.yml` if needed.
|
||||||
|
|
||||||
|
|
@ -250,5 +123,3 @@ Adjust in `docker-compose.yml` if needed.
|
||||||
- **Docker Compose v2**: This project uses `docker compose` (v2 syntax). If you have v1, either upgrade or create an alias: `alias docker-compose='docker compose'`
|
- **Docker Compose v2**: This project uses `docker compose` (v2 syntax). If you have v1, either upgrade or create an alias: `alias docker-compose='docker compose'`
|
||||||
- **Standalone Mode**: TiDB runs without distributed storage, suitable for development only
|
- **Standalone Mode**: TiDB runs without distributed storage, suitable for development only
|
||||||
- **Data Persistence**: Data is stored in Docker volumes, persists across restarts
|
- **Data Persistence**: Data is stored in Docker volumes, persists across restarts
|
||||||
- **Sync Mode**: Configured for full + incremental sync ("all" mode)
|
|
||||||
- **OrbStack DNS**: Uses `.orb.local` hostnames for container networking
|
|
||||||
|
|
|
||||||
354
SYNC_GUIDE.md
354
SYNC_GUIDE.md
|
|
@ -1,354 +0,0 @@
|
||||||
# Data Sync Guide
|
|
||||||
|
|
||||||
## How Sync Works
|
|
||||||
|
|
||||||
Your TiDB Data Migration (DM) setup continuously syncs data from your test environment to the local TiDB instance.
|
|
||||||
|
|
||||||
```
|
|
||||||
Test TiDB ──────┐
|
|
||||||
│
|
|
||||||
(DM reads changes)
|
|
||||||
│
|
|
||||||
▼
|
|
||||||
DM Worker
|
|
||||||
│
|
|
||||||
(Applies to local)
|
|
||||||
│
|
|
||||||
▼
|
|
||||||
Local TiDB
|
|
||||||
```
|
|
||||||
|
|
||||||
## Automatic Sync Setup
|
|
||||||
|
|
||||||
When you run `./start.sh`, the sync is **automatically configured and started**:
|
|
||||||
|
|
||||||
1. ✅ Reads your `.env` file for credentials and table list
|
|
||||||
2. ✅ Generates the DM task configuration
|
|
||||||
3. ✅ Configures the source connection (test TiDB)
|
|
||||||
4. ✅ Starts the sync task
|
|
||||||
5. ✅ Begins syncing data (full + incremental)
|
|
||||||
|
|
||||||
**You don't need to do anything manually!**
|
|
||||||
|
|
||||||
## Sync Modes
|
|
||||||
|
|
||||||
The sync is configured with `task-mode: "all"`:
|
|
||||||
- **Full sync**: Initial copy of all existing data
|
|
||||||
- **Incremental sync**: Continuous replication of changes (INSERT, UPDATE, DELETE)
|
|
||||||
|
|
||||||
## Managing Sync
|
|
||||||
|
|
||||||
### Easy Way (Recommended)
|
|
||||||
|
|
||||||
Use the [`sync-control.sh`](sync-control.sh) script:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Check sync status
|
|
||||||
./sync-control.sh status
|
|
||||||
|
|
||||||
# Stop sync
|
|
||||||
./sync-control.sh stop
|
|
||||||
|
|
||||||
# Start sync
|
|
||||||
./sync-control.sh start
|
|
||||||
|
|
||||||
# Pause sync (temporarily)
|
|
||||||
./sync-control.sh pause
|
|
||||||
|
|
||||||
# Resume sync
|
|
||||||
./sync-control.sh resume
|
|
||||||
|
|
||||||
# Restart sync (stop + start)
|
|
||||||
./sync-control.sh restart
|
|
||||||
|
|
||||||
# Re-initialize configuration
|
|
||||||
./sync-control.sh reinit
|
|
||||||
```
|
|
||||||
|
|
||||||
### Advanced Way (dmctl)
|
|
||||||
|
|
||||||
Use `dmctl` directly:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Check status
|
|
||||||
docker exec dm-master /dmctl --master-addr=dm-master:8261 query-status test-to-local
|
|
||||||
|
|
||||||
# Stop task
|
|
||||||
docker exec dm-master /dmctl --master-addr=dm-master:8261 stop-task test-to-local
|
|
||||||
|
|
||||||
# Start task
|
|
||||||
docker exec dm-master /dmctl --master-addr=dm-master:8261 start-task test-to-local
|
|
||||||
|
|
||||||
# Pause task
|
|
||||||
docker exec dm-master /dmctl --master-addr=dm-master:8261 pause-task test-to-local
|
|
||||||
|
|
||||||
# Resume task
|
|
||||||
docker exec dm-master /dmctl --master-addr=dm-master:8261 resume-task test-to-local
|
|
||||||
```
|
|
||||||
|
|
||||||
## Checking Sync Status
|
|
||||||
|
|
||||||
### Quick Check
|
|
||||||
|
|
||||||
```bash
|
|
||||||
./status.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
This shows:
|
|
||||||
- Source configuration
|
|
||||||
- Task status (running, paused, stopped)
|
|
||||||
- Current sync position
|
|
||||||
- Error messages (if any)
|
|
||||||
- Local databases
|
|
||||||
|
|
||||||
### Detailed Status
|
|
||||||
|
|
||||||
```bash
|
|
||||||
./sync-control.sh status
|
|
||||||
```
|
|
||||||
|
|
||||||
### Verify Data Sync
|
|
||||||
|
|
||||||
Connect to local TiDB and verify:
|
|
||||||
|
|
||||||
```sql
|
|
||||||
-- Connect
|
|
||||||
mysql -h 127.0.0.1 -P 4000 -u root
|
|
||||||
|
|
||||||
-- Check databases
|
|
||||||
SHOW DATABASES;
|
|
||||||
|
|
||||||
-- Switch to your database
|
|
||||||
USE your_database;
|
|
||||||
|
|
||||||
-- Check tables
|
|
||||||
SHOW TABLES;
|
|
||||||
|
|
||||||
-- Verify row count
|
|
||||||
SELECT COUNT(*) FROM table1;
|
|
||||||
|
|
||||||
-- Compare with source (if you have access)
|
|
||||||
-- Run the same query on test environment
|
|
||||||
```
|
|
||||||
|
|
||||||
## Configuration Files
|
|
||||||
|
|
||||||
### Environment Variables (`.env`)
|
|
||||||
|
|
||||||
This is where you configure what to sync:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Source database
|
|
||||||
TEST_DB_HOST=your-test-tidb-host
|
|
||||||
TEST_DB_PORT=4000
|
|
||||||
TEST_DB_USER=root
|
|
||||||
TEST_DB_PASSWORD=your-password
|
|
||||||
|
|
||||||
# What to sync
|
|
||||||
DATABASE_NAME=your_database
|
|
||||||
TABLES="table1,table2,table3"
|
|
||||||
```
|
|
||||||
|
|
||||||
### Task Template (`configs/task.yaml`)
|
|
||||||
|
|
||||||
This is just a **template for reference**. The actual task config is generated by [`scripts/init-dm.sh`](scripts/init-dm.sh).
|
|
||||||
|
|
||||||
### Source Config (`configs/source.yaml`)
|
|
||||||
|
|
||||||
Template for source database connection. Also generated dynamically.
|
|
||||||
|
|
||||||
## Common Scenarios
|
|
||||||
|
|
||||||
### Adding/Removing Tables
|
|
||||||
|
|
||||||
1. Edit `.env` and update `TABLES` variable:
|
|
||||||
```bash
|
|
||||||
TABLES="table1,table2,table3,new_table4"
|
|
||||||
```
|
|
||||||
|
|
||||||
2. Re-initialize:
|
|
||||||
```bash
|
|
||||||
./sync-control.sh reinit
|
|
||||||
```
|
|
||||||
|
|
||||||
### Changing Source Database
|
|
||||||
|
|
||||||
1. Edit `.env` with new credentials
|
|
||||||
2. Restart everything:
|
|
||||||
```bash
|
|
||||||
docker compose down
|
|
||||||
./start.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
### Resetting Sync (Fresh Start)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Stop and remove everything
|
|
||||||
docker compose down -v
|
|
||||||
|
|
||||||
# Start fresh
|
|
||||||
./start.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
### Pausing Sync Temporarily
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Pause (without stopping containers)
|
|
||||||
./sync-control.sh pause
|
|
||||||
|
|
||||||
# Resume when ready
|
|
||||||
./sync-control.sh resume
|
|
||||||
```
|
|
||||||
|
|
||||||
## Monitoring
|
|
||||||
|
|
||||||
### View Logs
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# All services
|
|
||||||
docker compose logs -f
|
|
||||||
|
|
||||||
# DM Worker only
|
|
||||||
docker compose logs -f dm-worker
|
|
||||||
|
|
||||||
# DM Master only
|
|
||||||
docker compose logs -f dm-master
|
|
||||||
|
|
||||||
# Init script logs
|
|
||||||
docker logs dm-init
|
|
||||||
```
|
|
||||||
|
|
||||||
### Check DM Master Status
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docker exec dm-master /dmctl --master-addr=dm-master:8261 operate-source show
|
|
||||||
```
|
|
||||||
|
|
||||||
### Check DM Worker Status
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docker ps | grep dm-worker
|
|
||||||
```
|
|
||||||
|
|
||||||
## Troubleshooting
|
|
||||||
|
|
||||||
### Sync Not Starting
|
|
||||||
|
|
||||||
**Check init logs:**
|
|
||||||
```bash
|
|
||||||
docker logs dm-init
|
|
||||||
```
|
|
||||||
|
|
||||||
**Common issues:**
|
|
||||||
- Wrong credentials in `.env`
|
|
||||||
- Test database not accessible
|
|
||||||
- Tables don't exist in source
|
|
||||||
|
|
||||||
**Solution:**
|
|
||||||
```bash
|
|
||||||
# Fix .env, then:
|
|
||||||
./sync-control.sh reinit
|
|
||||||
```
|
|
||||||
|
|
||||||
### Sync Stopped with Errors
|
|
||||||
|
|
||||||
**Check error message:**
|
|
||||||
```bash
|
|
||||||
./sync-control.sh status
|
|
||||||
```
|
|
||||||
|
|
||||||
**Common errors:**
|
|
||||||
- Network connectivity issues
|
|
||||||
- Permission denied on source
|
|
||||||
- Table schema mismatch
|
|
||||||
|
|
||||||
**Solution:**
|
|
||||||
```bash
|
|
||||||
# Fix the underlying issue, then:
|
|
||||||
./sync-control.sh restart
|
|
||||||
```
|
|
||||||
|
|
||||||
### Data Not Syncing
|
|
||||||
|
|
||||||
**Verify task is running:**
|
|
||||||
```bash
|
|
||||||
./sync-control.sh status
|
|
||||||
```
|
|
||||||
|
|
||||||
**Check if tables exist:**
|
|
||||||
```bash
|
|
||||||
# On source
|
|
||||||
mysql -h $TEST_DB_HOST -P 4000 -u root -p -e "SHOW TABLES FROM your_database;"
|
|
||||||
|
|
||||||
# On local
|
|
||||||
mysql -h 127.0.0.1 -P 4000 -u root -e "SHOW TABLES FROM your_database;"
|
|
||||||
```
|
|
||||||
|
|
||||||
**Compare row counts:**
|
|
||||||
```bash
|
|
||||||
# Create a verification script
|
|
||||||
mysql -h 127.0.0.1 -P 4000 -u root -e "SELECT COUNT(*) FROM your_database.table1;"
|
|
||||||
```
|
|
||||||
|
|
||||||
## Performance Tuning
|
|
||||||
|
|
||||||
### Adjust Sync Speed
|
|
||||||
|
|
||||||
Edit `docker-compose.yml`:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
dm-worker:
|
|
||||||
deploy:
|
|
||||||
resources:
|
|
||||||
limits:
|
|
||||||
cpus: '2' # Increase CPU
|
|
||||||
memory: 2G # Increase memory
|
|
||||||
```
|
|
||||||
|
|
||||||
Then restart:
|
|
||||||
```bash
|
|
||||||
docker compose down
|
|
||||||
docker compose up -d
|
|
||||||
```
|
|
||||||
|
|
||||||
### Monitor Resource Usage
|
|
||||||
|
|
||||||
```bash
|
|
||||||
docker stats
|
|
||||||
```
|
|
||||||
|
|
||||||
## Best Practices
|
|
||||||
|
|
||||||
1. **Always check status** after starting: `./status.sh`
|
|
||||||
2. **Monitor logs** during initial sync: `docker compose logs -f dm-worker`
|
|
||||||
3. **Verify data** in local TiDB after sync completes
|
|
||||||
4. **Use pause/resume** instead of stop/start for temporary halts
|
|
||||||
5. **Keep `.env` secure** - it contains credentials
|
|
||||||
6. **Test connectivity** before sync: `./test-connection.sh`
|
|
||||||
|
|
||||||
## FAQ
|
|
||||||
|
|
||||||
**Q: Is the sync real-time?**
|
|
||||||
A: Near real-time. Changes are replicated with minimal delay (usually seconds).
|
|
||||||
|
|
||||||
**Q: What happens if my laptop sleeps?**
|
|
||||||
A: Sync will resume automatically when containers restart.
|
|
||||||
|
|
||||||
**Q: Can I sync from multiple sources?**
|
|
||||||
A: Yes, but requires manual DM configuration. This setup is for single source.
|
|
||||||
|
|
||||||
**Q: Does it sync schema changes?**
|
|
||||||
A: Yes, DDL statements are replicated (CREATE, ALTER, DROP).
|
|
||||||
|
|
||||||
**Q: Can I sync to a different database name locally?**
|
|
||||||
A: Requires custom task configuration. Default syncs to same database name.
|
|
||||||
|
|
||||||
**Q: How do I exclude certain tables?**
|
|
||||||
A: Remove them from `TABLES` in `.env` and run `./sync-control.sh reinit`.
|
|
||||||
|
|
||||||
## See Also
|
|
||||||
|
|
||||||
- [README.md](README.md) - Main documentation
|
|
||||||
- [DATAGRIP_SETUP.md](DATAGRIP_SETUP.md) - Connect with GUI clients
|
|
||||||
- [scripts/init-dm.sh](scripts/init-dm.sh) - Initialization script
|
|
||||||
- [TiDB DM Documentation](https://docs.pingcap.com/tidb-data-migration/stable)
|
|
||||||
|
|
@ -1,275 +0,0 @@
|
||||||
# TiDB Cloud to Local TiDB Migration Guide
|
|
||||||
|
|
||||||
This guide provides officially recommended approaches for migrating data from TiDB Cloud to a local TiDB instance, since TiDB Data Migration (DM) cannot be used with TiDB Cloud Serverless.
|
|
||||||
|
|
||||||
## Why DM Doesn't Work with TiDB Cloud
|
|
||||||
|
|
||||||
TiDB Data Migration (DM) fails with TiDB Cloud because:
|
|
||||||
|
|
||||||
1. **No MySQL binlog support** - TiDB Cloud Serverless doesn't expose binlog in the traditional MySQL way
|
|
||||||
2. **binlog_format is STATEMENT** - DM requires ROW format
|
|
||||||
3. **TiDB explicitly not supported as upstream** - DM is designed for MySQL/MariaDB → TiDB, not TiDB → TiDB
|
|
||||||
|
|
||||||
## Approach 1: Console Export + SQL Import (Simplest)
|
|
||||||
|
|
||||||
### Export from TiDB Cloud
|
|
||||||
|
|
||||||
1. **Using TiDB Cloud Console**:
|
|
||||||
- Navigate to your cluster in the TiDB Cloud Console
|
|
||||||
- Go to Data > Import
|
|
||||||
- Click "Export Data to" > "Local File"
|
|
||||||
- Select databases/tables to export
|
|
||||||
- Choose format (SQL recommended for small datasets)
|
|
||||||
- Click "Export"
|
|
||||||
|
|
||||||
2. **Using TiDB Cloud CLI**:
|
|
||||||
```bash
|
|
||||||
# Create export task
|
|
||||||
ticloud serverless export create -c <cluster-id>
|
|
||||||
|
|
||||||
# Download exported data
|
|
||||||
ticloud serverless export download -c <cluster-id> -e <export-id>
|
|
||||||
```
|
|
||||||
|
|
||||||
### Import to Local TiDB
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Import SQL file
|
|
||||||
mysql -h 127.0.0.1 -P 4000 -u root < exported_data.sql
|
|
||||||
|
|
||||||
# Or for CSV files
|
|
||||||
mysql -h 127.0.0.1 -P 4000 -u root -e "
|
|
||||||
LOAD DATA LOCAL INFILE 'table_data.csv'
|
|
||||||
INTO TABLE your_table
|
|
||||||
FIELDS TERMINATED BY ','
|
|
||||||
ENCLOSED BY '\"'
|
|
||||||
LINES TERMINATED BY '\n'
|
|
||||||
IGNORE 1 ROWS;"
|
|
||||||
```
|
|
||||||
|
|
||||||
## Approach 2: Dumpling + TiDB Lightning (For Larger Datasets)
|
|
||||||
|
|
||||||
### Prerequisites
|
|
||||||
|
|
||||||
Install TiDB tools:
|
|
||||||
```bash
|
|
||||||
# Install TiUP
|
|
||||||
curl --proto '=https' --tlsv1.2 -sSf https://tiup-mirrors.pingcap.com/install.sh | sh
|
|
||||||
source ~/.bash_profile
|
|
||||||
|
|
||||||
# Install tools
|
|
||||||
tiup install dumpling tidb-lightning
|
|
||||||
```
|
|
||||||
|
|
||||||
### Export with Dumpling
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Export data from TiDB Cloud
|
|
||||||
dumpling \
|
|
||||||
-u {TEST_DB_USER} \
|
|
||||||
-p {TEST_DB_PASSWORD} \
|
|
||||||
-P {TEST_DB_PORT} \
|
|
||||||
-h {TEST_DB_HOST} \
|
|
||||||
-o /tmp/tidb-export \
|
|
||||||
--filetype sql \
|
|
||||||
-r 20000 \
|
|
||||||
-F 256MiB
|
|
||||||
```
|
|
||||||
|
|
||||||
### Import with TiDB Lightning
|
|
||||||
|
|
||||||
1. **Create configuration file** (`lightning.toml`):
|
|
||||||
```toml
|
|
||||||
[lightning]
|
|
||||||
level = "info"
|
|
||||||
file = "tidb-lightning.log"
|
|
||||||
|
|
||||||
[tikv-importer]
|
|
||||||
backend = "local"
|
|
||||||
sorted-kv-dir = "/tmp/sorted-kv-dir"
|
|
||||||
|
|
||||||
[mydumper]
|
|
||||||
data-source-dir = "/tmp/tidb-export"
|
|
||||||
no-schema = false
|
|
||||||
|
|
||||||
[tidb]
|
|
||||||
host = "127.0.0.1"
|
|
||||||
port = 4000
|
|
||||||
user = "root"
|
|
||||||
password = ""
|
|
||||||
status-port = 10080
|
|
||||||
pd-addr = "127.0.0.1:2379"
|
|
||||||
```
|
|
||||||
|
|
||||||
2. **Run TiDB Lightning**:
|
|
||||||
```bash
|
|
||||||
tidb-lightning -config lightning.toml
|
|
||||||
```
|
|
||||||
|
|
||||||
## Approach 3: Periodic Sync Script
|
|
||||||
|
|
||||||
Create a script for periodic data sync:
|
|
||||||
|
|
||||||
### Export Script (`export-cloud.sh`)
|
|
||||||
```bash
|
|
||||||
#!/bin/bash
|
|
||||||
|
|
||||||
# Source .env file
|
|
||||||
source .env
|
|
||||||
|
|
||||||
# Export data using mysqldump (built-in tool)
|
|
||||||
mysqldump \
|
|
||||||
-h $TEST_DB_HOST \
|
|
||||||
-P $TEST_DB_PORT \
|
|
||||||
-u $TEST_DB_USER \
|
|
||||||
-p$TEST_DB_PASSWORD \
|
|
||||||
--single-transaction \
|
|
||||||
--routines \
|
|
||||||
--triggers \
|
|
||||||
$DATABASE_NAME \
|
|
||||||
$TABLES > /tmp/cloud-export.sql
|
|
||||||
|
|
||||||
echo "Export completed: /tmp/cloud-export.sql"
|
|
||||||
```
|
|
||||||
|
|
||||||
### Import Script (`import-local.sh`)
|
|
||||||
```bash
|
|
||||||
#!/bin/bash
|
|
||||||
|
|
||||||
# Import to local TiDB
|
|
||||||
mysql -h 127.0.0.1 -P 4000 -u root < /tmp/cloud-export.sql
|
|
||||||
|
|
||||||
echo "Import completed to local TiDB"
|
|
||||||
```
|
|
||||||
|
|
||||||
### Combined Sync Script (`sync-data.sh`)
|
|
||||||
```bash
|
|
||||||
#!/bin/bash
|
|
||||||
|
|
||||||
echo "🔄 Syncing data from TiDB Cloud to local TiDB..."
|
|
||||||
|
|
||||||
# Export from cloud
|
|
||||||
./export-cloud.sh
|
|
||||||
|
|
||||||
# Import to local
|
|
||||||
./import-local.sh
|
|
||||||
|
|
||||||
echo "✅ Sync completed!"
|
|
||||||
```
|
|
||||||
|
|
||||||
## Approach 4: Application-Level Sync (For Continuous Updates)
|
|
||||||
|
|
||||||
For real-time sync, implement in your application:
|
|
||||||
|
|
||||||
```python
|
|
||||||
# Example Python script for selective sync
|
|
||||||
import mysql.connector
|
|
||||||
|
|
||||||
# Connect to both databases
|
|
||||||
cloud_db = mysql.connector.connect(
|
|
||||||
host="gateway01.ap-northeast-1.prod.aws.tidbcloud.com",
|
|
||||||
port=4000,
|
|
||||||
user="3mmwxY44wQF4L6P.root",
|
|
||||||
password="JQ8wbPsXfx7xJOR5",
|
|
||||||
database="workflow_local"
|
|
||||||
)
|
|
||||||
|
|
||||||
local_db = mysql.connector.connect(
|
|
||||||
host="127.0.0.1",
|
|
||||||
port=4000,
|
|
||||||
user="root",
|
|
||||||
password="",
|
|
||||||
database="workflow_local"
|
|
||||||
)
|
|
||||||
|
|
||||||
# Sync specific tables
|
|
||||||
def sync_table(table_name):
|
|
||||||
# Get data from cloud
|
|
||||||
cloud_cursor = cloud_db.cursor()
|
|
||||||
cloud_cursor.execute(f"SELECT * FROM {table_name}")
|
|
||||||
rows = cloud_cursor.fetchall()
|
|
||||||
|
|
||||||
# Clear and insert into local
|
|
||||||
local_cursor = local_db.cursor()
|
|
||||||
local_cursor.execute(f"DELETE FROM {table_name}")
|
|
||||||
|
|
||||||
if rows:
|
|
||||||
placeholders = ','.join(['%s'] * len(rows[0]))
|
|
||||||
local_cursor.executemany(
|
|
||||||
f"INSERT INTO {table_name} VALUES ({placeholders})",
|
|
||||||
rows
|
|
||||||
)
|
|
||||||
|
|
||||||
local_db.commit()
|
|
||||||
print(f"Synced {len(rows)} rows to {table_name}")
|
|
||||||
|
|
||||||
# Sync your tables
|
|
||||||
sync_table("plans")
|
|
||||||
```
|
|
||||||
|
|
||||||
## Recommended Solution for Your Setup
|
|
||||||
|
|
||||||
For development purposes, I recommend:
|
|
||||||
|
|
||||||
1. **Use Approach 1** (Console Export + SQL Import) for simplicity
|
|
||||||
2. **Create helper scripts** for periodic sync
|
|
||||||
3. **Consider application-level sync** for real-time needs
|
|
||||||
|
|
||||||
### Quick Setup
|
|
||||||
|
|
||||||
Create these helper scripts in your project:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Make scripts executable
|
|
||||||
chmod +x sync-data.sh export-cloud.sh import-local.sh
|
|
||||||
|
|
||||||
# Run sync
|
|
||||||
./sync-data.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
## Limitations and Considerations
|
|
||||||
|
|
||||||
### TiDB Cloud Serverless Limitations
|
|
||||||
- No traditional MySQL binlog access
|
|
||||||
- Limited to export/import methods
|
|
||||||
- No direct replication support in most plans
|
|
||||||
|
|
||||||
### Performance Considerations
|
|
||||||
- Full table exports can be slow for large datasets
|
|
||||||
- Network bandwidth affects sync speed
|
|
||||||
- Consider incremental exports for large tables
|
|
||||||
|
|
||||||
### Security Notes
|
|
||||||
- Store credentials securely (use .env file)
|
|
||||||
- Use TLS connections when possible
|
|
||||||
- Rotate credentials regularly
|
|
||||||
|
|
||||||
## Troubleshooting
|
|
||||||
|
|
||||||
### Connection Issues
|
|
||||||
```bash
|
|
||||||
# Test connection to TiDB Cloud
|
|
||||||
mysql -h $TEST_DB_HOST -P $TEST_DB_PORT -u $TEST_DB_USER -p
|
|
||||||
|
|
||||||
# Test connection to local TiDB
|
|
||||||
mysql -h 127.0.0.1 -P 4000 -u root
|
|
||||||
```
|
|
||||||
|
|
||||||
### Export Errors
|
|
||||||
- Ensure user has SELECT privileges
|
|
||||||
- Check network connectivity
|
|
||||||
- Verify table existence
|
|
||||||
|
|
||||||
### Import Errors
|
|
||||||
- Check schema compatibility
|
|
||||||
- Ensure sufficient disk space
|
|
||||||
- Verify TiDB is running
|
|
||||||
|
|
||||||
## References
|
|
||||||
|
|
||||||
- [TiDB Cloud Export Documentation](https://docs.pingcap.com/tidbcloud/serverless-export/)
|
|
||||||
- [TiDB Migration Tools Overview](https://docs.pingcap.com/tidb/stable/migration-tools)
|
|
||||||
- [Dumpling Documentation](https://docs.pingcap.com/tidb/stable/dumpling-overview)
|
|
||||||
- [TiDB Lightning Documentation](https://docs.pingcap.com/tidb/stable/tidb-lightning-overview)
|
|
||||||
|
|
||||||
For production use cases, contact TiDB Cloud Support to discuss available replication options for your specific plan.
|
|
||||||
|
|
@ -1,419 +0,0 @@
|
||||||
# Troubleshooting Guide
|
|
||||||
|
|
||||||
## Common Issues and Solutions
|
|
||||||
|
|
||||||
### 1. TiDB Health Check Failing
|
|
||||||
|
|
||||||
#### Symptom
|
|
||||||
```
|
|
||||||
dependency failed to start: container tidb is unhealthy
|
|
||||||
```
|
|
||||||
|
|
||||||
Even though you can connect to TiDB from your host machine, Docker health check fails.
|
|
||||||
|
|
||||||
#### Root Cause
|
|
||||||
The original health check tried to use `mysql` command inside the TiDB container:
|
|
||||||
```yaml
|
|
||||||
healthcheck:
|
|
||||||
test: ["CMD", "mysql", "-h", "127.0.0.1", "-P", "4000", "-u", "root", "-e", "SELECT 1"]
|
|
||||||
```
|
|
||||||
|
|
||||||
The TiDB Docker image doesn't include the MySQL client binary, so this check always failed.
|
|
||||||
|
|
||||||
#### Solution ✅
|
|
||||||
Use TiDB's built-in HTTP status endpoint instead:
|
|
||||||
```yaml
|
|
||||||
healthcheck:
|
|
||||||
test: ["CMD", "wget", "-q", "-O-", "http://127.0.0.1:10080/status"]
|
|
||||||
interval: 10s
|
|
||||||
timeout: 5s
|
|
||||||
retries: 3
|
|
||||||
start_period: 10s
|
|
||||||
```
|
|
||||||
|
|
||||||
**Why this works:**
|
|
||||||
- TiDB exposes a status endpoint on port 10080
|
|
||||||
- `wget` is available in the container
|
|
||||||
- Returns HTTP 200 when TiDB is ready
|
|
||||||
- `start_period` gives TiDB time to initialize before health checks begin
|
|
||||||
|
|
||||||
### 2. Docker Compose Version Warning
|
|
||||||
|
|
||||||
#### Symptom
|
|
||||||
```
|
|
||||||
WARN[0000] version is obsolete, it will be ignored
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Solution ✅
|
|
||||||
Remove the `version` field from `docker-compose.yml`. Modern Docker Compose (v2) doesn't need it.
|
|
||||||
|
|
||||||
**Before:**
|
|
||||||
```yaml
|
|
||||||
version: '3.8'
|
|
||||||
services:
|
|
||||||
...
|
|
||||||
```
|
|
||||||
|
|
||||||
**After:**
|
|
||||||
```yaml
|
|
||||||
services:
|
|
||||||
...
|
|
||||||
```
|
|
||||||
|
|
||||||
### 3. Service Dependencies Not Starting in Order
|
|
||||||
|
|
||||||
#### Symptom
|
|
||||||
Services fail because dependencies aren't ready yet.
|
|
||||||
|
|
||||||
#### Solution ✅
|
|
||||||
Use proper health checks and dependency conditions:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
dm-worker:
|
|
||||||
depends_on:
|
|
||||||
tidb:
|
|
||||||
condition: service_healthy
|
|
||||||
dm-master:
|
|
||||||
condition: service_healthy
|
|
||||||
```
|
|
||||||
|
|
||||||
**Important:**
|
|
||||||
- Each dependency must have a working health check
|
|
||||||
- `start_period` prevents false negatives during startup
|
|
||||||
|
|
||||||
### 4. dm-init Fails to Start
|
|
||||||
|
|
||||||
#### Symptom
|
|
||||||
```
|
|
||||||
Error: dm-init exits immediately
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Check:
|
|
||||||
```bash
|
|
||||||
docker logs dm-init
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Common Causes:
|
|
||||||
|
|
||||||
**a) .env not configured:**
|
|
||||||
```bash
|
|
||||||
# Check if .env exists and has real values
|
|
||||||
cat .env
|
|
||||||
```
|
|
||||||
|
|
||||||
**Solution:**
|
|
||||||
```bash
|
|
||||||
# Copy template and edit
|
|
||||||
cp .env.example .env
|
|
||||||
vim .env
|
|
||||||
```
|
|
||||||
|
|
||||||
**b) Test database not reachable:**
|
|
||||||
```bash
|
|
||||||
# Test from dm-init container
|
|
||||||
docker run --rm --network tidb-network pingcap/dm:latest \
|
|
||||||
sh -c "wget -q -O- http://tidb:10080/status"
|
|
||||||
```
|
|
||||||
|
|
||||||
**c) Script syntax error:**
|
|
||||||
```bash
|
|
||||||
# Check init script
|
|
||||||
sh -n scripts/init-dm.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
### 5. Containers Keep Restarting
|
|
||||||
|
|
||||||
#### Check Status:
|
|
||||||
```bash
|
|
||||||
docker ps -a
|
|
||||||
docker logs <container_name>
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Common Issues:
|
|
||||||
|
|
||||||
**a) Port already in use:**
|
|
||||||
```
|
|
||||||
Error: bind: address already in use
|
|
||||||
```
|
|
||||||
|
|
||||||
**Solution:** Change ports in `docker-compose.yml`:
|
|
||||||
```yaml
|
|
||||||
ports:
|
|
||||||
- "14000:4000" # Changed from 4000:4000
|
|
||||||
```
|
|
||||||
|
|
||||||
**b) Out of memory:**
|
|
||||||
```
|
|
||||||
Error: OOM killed
|
|
||||||
```
|
|
||||||
|
|
||||||
**Solution:** Increase memory limits or free up system resources.
|
|
||||||
|
|
||||||
**c) Permission issues:**
|
|
||||||
```
|
|
||||||
Error: permission denied
|
|
||||||
```
|
|
||||||
|
|
||||||
**Solution:** Check volume permissions or run:
|
|
||||||
```bash
|
|
||||||
docker compose down -v # Remove volumes
|
|
||||||
docker compose up -d # Recreate
|
|
||||||
```
|
|
||||||
|
|
||||||
### 6. Sync Task Not Running
|
|
||||||
|
|
||||||
#### Check Status:
|
|
||||||
```bash
|
|
||||||
./status.sh
|
|
||||||
# or
|
|
||||||
./sync-control.sh status
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Common Issues:
|
|
||||||
|
|
||||||
**a) Task not created:**
|
|
||||||
```bash
|
|
||||||
# Check if source is configured
|
|
||||||
docker exec dm-master /dmctl --master-addr=dm-master:8261 operate-source show
|
|
||||||
```
|
|
||||||
|
|
||||||
**Solution:**
|
|
||||||
```bash
|
|
||||||
./sync-control.sh reinit
|
|
||||||
```
|
|
||||||
|
|
||||||
**b) Wrong credentials:**
|
|
||||||
Check logs:
|
|
||||||
```bash
|
|
||||||
docker logs dm-worker
|
|
||||||
```
|
|
||||||
|
|
||||||
Fix `.env` and reinit:
|
|
||||||
```bash
|
|
||||||
vim .env
|
|
||||||
./sync-control.sh reinit
|
|
||||||
```
|
|
||||||
|
|
||||||
**c) Table doesn't exist:**
|
|
||||||
Verify tables exist on source database:
|
|
||||||
```bash
|
|
||||||
# Connect to test DB and check
|
|
||||||
SHOW TABLES FROM your_database;
|
|
||||||
```
|
|
||||||
|
|
||||||
### 7. Connection Refused to TiDB
|
|
||||||
|
|
||||||
#### Symptom
|
|
||||||
```
|
|
||||||
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (61)
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Checks:
|
|
||||||
|
|
||||||
**a) Is TiDB running?**
|
|
||||||
```bash
|
|
||||||
docker ps | grep tidb
|
|
||||||
```
|
|
||||||
|
|
||||||
**b) Is it healthy?**
|
|
||||||
```bash
|
|
||||||
docker ps
|
|
||||||
# Look for "(healthy)" status
|
|
||||||
```
|
|
||||||
|
|
||||||
**c) Is port exposed?**
|
|
||||||
```bash
|
|
||||||
docker port tidb
|
|
||||||
# Should show: 4000/tcp -> 0.0.0.0:4000
|
|
||||||
```
|
|
||||||
|
|
||||||
**d) Test from inside container:**
|
|
||||||
```bash
|
|
||||||
docker exec tidb wget -q -O- http://127.0.0.1:10080/status
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Solutions:
|
|
||||||
|
|
||||||
**If container not running:**
|
|
||||||
```bash
|
|
||||||
docker compose up -d tidb
|
|
||||||
docker logs tidb
|
|
||||||
```
|
|
||||||
|
|
||||||
**If unhealthy:**
|
|
||||||
```bash
|
|
||||||
# Wait for health check
|
|
||||||
sleep 15
|
|
||||||
docker ps
|
|
||||||
|
|
||||||
# If still unhealthy, check logs
|
|
||||||
docker logs tidb
|
|
||||||
```
|
|
||||||
|
|
||||||
**If port not exposed:**
|
|
||||||
```bash
|
|
||||||
# Recreate container
|
|
||||||
docker compose down
|
|
||||||
docker compose up -d
|
|
||||||
```
|
|
||||||
|
|
||||||
### 8. Data Not Syncing
|
|
||||||
|
|
||||||
#### Verify sync is running:
|
|
||||||
```bash
|
|
||||||
./sync-control.sh status
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Check sync lag:
|
|
||||||
Look for "syncer" section in status output.
|
|
||||||
|
|
||||||
#### Common Issues:
|
|
||||||
|
|
||||||
**a) Sync paused:**
|
|
||||||
```bash
|
|
||||||
./sync-control.sh resume
|
|
||||||
```
|
|
||||||
|
|
||||||
**b) Sync stopped with error:**
|
|
||||||
```bash
|
|
||||||
# Check error in status output
|
|
||||||
./sync-control.sh status
|
|
||||||
|
|
||||||
# Fix the issue, then restart
|
|
||||||
./sync-control.sh restart
|
|
||||||
```
|
|
||||||
|
|
||||||
**c) Network issues:**
|
|
||||||
```bash
|
|
||||||
# Test connectivity from dm-worker to source
|
|
||||||
docker exec dm-worker ping -c 3 your-test-db-host
|
|
||||||
```
|
|
||||||
|
|
||||||
**d) Binlog not enabled on source:**
|
|
||||||
Source database must have binlog enabled for incremental sync.
|
|
||||||
|
|
||||||
### 9. Slow Sync Performance
|
|
||||||
|
|
||||||
#### Check resource usage:
|
|
||||||
```bash
|
|
||||||
docker stats
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Solutions:
|
|
||||||
|
|
||||||
**a) Increase worker resources:**
|
|
||||||
Edit `docker-compose.yml`:
|
|
||||||
```yaml
|
|
||||||
dm-worker:
|
|
||||||
deploy:
|
|
||||||
resources:
|
|
||||||
limits:
|
|
||||||
cpus: '2'
|
|
||||||
memory: 2G
|
|
||||||
```
|
|
||||||
|
|
||||||
**b) Optimize batch size:**
|
|
||||||
See [TiDB DM Documentation](https://docs.pingcap.com/tidb-data-migration/stable/tune-configuration) for advanced tuning.
|
|
||||||
|
|
||||||
### 10. Docker Compose v1 vs v2 Issues
|
|
||||||
|
|
||||||
#### Symptom
|
|
||||||
```
|
|
||||||
docker: 'compose' is not a docker command
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Solution
|
|
||||||
See [DOCKER_COMPOSE_V2.md](DOCKER_COMPOSE_V2.md) for:
|
|
||||||
- Upgrading to v2
|
|
||||||
- Creating an alias
|
|
||||||
- Compatibility mode
|
|
||||||
|
|
||||||
## Diagnostic Commands
|
|
||||||
|
|
||||||
### Check everything at once:
|
|
||||||
```bash
|
|
||||||
# Service status
|
|
||||||
docker compose ps
|
|
||||||
|
|
||||||
# Health checks
|
|
||||||
docker ps
|
|
||||||
|
|
||||||
# Logs (all services)
|
|
||||||
docker compose logs --tail=50
|
|
||||||
|
|
||||||
# Logs (specific service)
|
|
||||||
docker compose logs --tail=50 tidb
|
|
||||||
|
|
||||||
# Resource usage
|
|
||||||
docker stats --no-stream
|
|
||||||
|
|
||||||
# Network connectivity
|
|
||||||
docker network inspect tidb-network
|
|
||||||
```
|
|
||||||
|
|
||||||
### Test connectivity:
|
|
||||||
```bash
|
|
||||||
# From host to TiDB
|
|
||||||
mysql -h 127.0.0.1 -P 4000 -u root -e "SELECT 1"
|
|
||||||
|
|
||||||
# From host (HTTP)
|
|
||||||
curl http://127.0.0.1:10080/status
|
|
||||||
|
|
||||||
# From container to TiDB
|
|
||||||
docker run --rm --network tidb-network pingcap/dm:latest \
|
|
||||||
wget -q -O- http://tidb:10080/status
|
|
||||||
```
|
|
||||||
|
|
||||||
### Reset everything:
|
|
||||||
```bash
|
|
||||||
# Stop and remove everything (including data)
|
|
||||||
docker compose down -v
|
|
||||||
|
|
||||||
# Start fresh
|
|
||||||
./start.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
## Getting Help
|
|
||||||
|
|
||||||
### Collect diagnostic information:
|
|
||||||
```bash
|
|
||||||
# Create a diagnostic report
|
|
||||||
echo "=== Docker Version ===" > diagnostic.txt
|
|
||||||
docker --version >> diagnostic.txt
|
|
||||||
docker compose version >> diagnostic.txt
|
|
||||||
|
|
||||||
echo -e "\n=== Container Status ===" >> diagnostic.txt
|
|
||||||
docker ps -a >> diagnostic.txt
|
|
||||||
|
|
||||||
echo -e "\n=== TiDB Logs ===" >> diagnostic.txt
|
|
||||||
docker logs tidb --tail=50 >> diagnostic.txt 2>&1
|
|
||||||
|
|
||||||
echo -e "\n=== DM Worker Logs ===" >> diagnostic.txt
|
|
||||||
docker logs dm-worker --tail=50 >> diagnostic.txt 2>&1
|
|
||||||
|
|
||||||
echo -e "\n=== DM Init Logs ===" >> diagnostic.txt
|
|
||||||
docker logs dm-init >> diagnostic.txt 2>&1
|
|
||||||
|
|
||||||
echo -e "\n=== Network Info ===" >> diagnostic.txt
|
|
||||||
docker network inspect tidb-network >> diagnostic.txt
|
|
||||||
|
|
||||||
echo "Report saved to diagnostic.txt"
|
|
||||||
```
|
|
||||||
|
|
||||||
### Useful resources:
|
|
||||||
- [TiDB Documentation](https://docs.pingcap.com/tidb/stable)
|
|
||||||
- [TiDB DM Documentation](https://docs.pingcap.com/tidb-data-migration/stable)
|
|
||||||
- Project documentation:
|
|
||||||
- [README.md](README.md)
|
|
||||||
- [SYNC_GUIDE.md](SYNC_GUIDE.md)
|
|
||||||
- [DATAGRIP_SETUP.md](DATAGRIP_SETUP.md)
|
|
||||||
- [DOCKER_COMPOSE_V2.md](DOCKER_COMPOSE_V2.md)
|
|
||||||
|
|
||||||
## Still Having Issues?
|
|
||||||
|
|
||||||
If none of these solutions work:
|
|
||||||
|
|
||||||
1. Check logs: `docker compose logs`
|
|
||||||
2. Create diagnostic report (see above)
|
|
||||||
3. Check if it's a known issue in TiDB/DM GitHub issues
|
|
||||||
4. Verify your environment meets prerequisites (see [README.md](README.md))
|
|
||||||
|
|
@ -1,12 +0,0 @@
|
||||||
source-id: "test-tidb"
|
|
||||||
enable-gtid: false
|
|
||||||
enable-relay: false
|
|
||||||
from:
|
|
||||||
host: "${TEST_DB_HOST}"
|
|
||||||
port: ${TEST_DB_PORT}
|
|
||||||
user: "${TEST_DB_USER}"
|
|
||||||
password: "${TEST_DB_PASSWORD}"
|
|
||||||
security:
|
|
||||||
ssl-ca: ""
|
|
||||||
ssl-cert: ""
|
|
||||||
ssl-key: ""
|
|
||||||
|
|
@ -1,52 +0,0 @@
|
||||||
# ==============================================================================
|
|
||||||
# DM Task Configuration Template
|
|
||||||
# ==============================================================================
|
|
||||||
#
|
|
||||||
# This file is a TEMPLATE and is NOT used directly by DM.
|
|
||||||
#
|
|
||||||
# The actual task configuration is dynamically generated by:
|
|
||||||
# scripts/init-dm.sh
|
|
||||||
#
|
|
||||||
# The script reads environment variables from .env and creates the real task.yaml
|
|
||||||
# with your specific database name and table list.
|
|
||||||
#
|
|
||||||
# HOW TO RUN THE SYNC:
|
|
||||||
# --------------------
|
|
||||||
# 1. Configure .env file with your credentials
|
|
||||||
# 2. Run: ./start.sh (auto-generates and starts sync)
|
|
||||||
# 3. Check status: ./status.sh or ./sync-control.sh status
|
|
||||||
#
|
|
||||||
# For detailed guide, see: SYNC_GUIDE.md
|
|
||||||
#
|
|
||||||
# MANUAL CONTROL:
|
|
||||||
# ---------------
|
|
||||||
# - Start: ./sync-control.sh start
|
|
||||||
# - Stop: ./sync-control.sh stop
|
|
||||||
# - Pause: ./sync-control.sh pause
|
|
||||||
# - Resume: ./sync-control.sh resume
|
|
||||||
# - Restart: ./sync-control.sh restart
|
|
||||||
# - Reinit: ./sync-control.sh reinit
|
|
||||||
#
|
|
||||||
# ==============================================================================
|
|
||||||
|
|
||||||
# Template structure (for reference):
|
|
||||||
# name: "test-to-local"
|
|
||||||
# task-mode: "all" # full + incremental sync
|
|
||||||
# target-database:
|
|
||||||
# host: "tidb"
|
|
||||||
# port: 4000
|
|
||||||
# user: "root"
|
|
||||||
# password: ""
|
|
||||||
#
|
|
||||||
# mysql-instances:
|
|
||||||
# - source-id: "test-tidb"
|
|
||||||
# block-allow-list: "sync-tables"
|
|
||||||
#
|
|
||||||
# block-allow-list:
|
|
||||||
# sync-tables:
|
|
||||||
# do-dbs: ["${DATABASE_NAME}"]
|
|
||||||
# do-tables:
|
|
||||||
# - db-name: "${DATABASE_NAME}"
|
|
||||||
# tbl-name: "table1"
|
|
||||||
# - db-name: "${DATABASE_NAME}"
|
|
||||||
# tbl-name: "table2"
|
|
||||||
|
|
@ -1,15 +1,6 @@
|
||||||
# OrbStack optimized settings
|
# Minimal TiDB setup - Standalone mode with unistore
|
||||||
x-common-settings: &common
|
|
||||||
restart: unless-stopped
|
|
||||||
logging:
|
|
||||||
driver: "json-file"
|
|
||||||
options:
|
|
||||||
max-size: "10m"
|
|
||||||
max-file: "3"
|
|
||||||
|
|
||||||
services:
|
services:
|
||||||
tidb:
|
tidb:
|
||||||
<<: *common
|
|
||||||
image: pingcap/tidb:latest
|
image: pingcap/tidb:latest
|
||||||
container_name: tidb
|
container_name: tidb
|
||||||
hostname: tidb.orb.local # OrbStack DNS
|
hostname: tidb.orb.local # OrbStack DNS
|
||||||
|
|
@ -39,89 +30,11 @@ services:
|
||||||
reservations:
|
reservations:
|
||||||
cpus: '1'
|
cpus: '1'
|
||||||
memory: 1G
|
memory: 1G
|
||||||
|
restart: unless-stopped
|
||||||
dm-master:
|
|
||||||
<<: *common
|
|
||||||
image: pingcap/dm:latest
|
|
||||||
container_name: dm-master
|
|
||||||
hostname: dm-master.orb.local
|
|
||||||
command:
|
|
||||||
- /dm-master
|
|
||||||
- --master-addr=:8261
|
|
||||||
- --advertise-addr=dm-master:8261
|
|
||||||
ports:
|
|
||||||
- "8261:8261"
|
|
||||||
volumes:
|
|
||||||
- dm_master_data:/data
|
|
||||||
- ./configs:/configs:ro
|
|
||||||
healthcheck:
|
|
||||||
test: ["CMD", "wget", "-q", "-O-", "http://127.0.0.1:8261/status"]
|
|
||||||
interval: 10s
|
|
||||||
timeout: 5s
|
|
||||||
retries: 3
|
|
||||||
start_period: 5s
|
|
||||||
deploy:
|
|
||||||
resources:
|
|
||||||
limits:
|
|
||||||
cpus: '0.5'
|
|
||||||
memory: 512M
|
|
||||||
|
|
||||||
dm-worker:
|
|
||||||
<<: *common
|
|
||||||
image: pingcap/dm:latest
|
|
||||||
container_name: dm-worker
|
|
||||||
hostname: dm-worker.orb.local
|
|
||||||
command:
|
|
||||||
- /dm-worker
|
|
||||||
- --worker-addr=:8262
|
|
||||||
- --advertise-addr=dm-worker:8262
|
|
||||||
- --join=dm-master:8261
|
|
||||||
volumes:
|
|
||||||
- dm_worker_data:/data
|
|
||||||
- ./configs:/configs:ro
|
|
||||||
depends_on:
|
|
||||||
tidb:
|
|
||||||
condition: service_healthy
|
|
||||||
dm-master:
|
|
||||||
condition: service_healthy
|
|
||||||
healthcheck:
|
|
||||||
test: ["CMD", "wget", "-q", "-O-", "http://127.0.0.1:8262/status"]
|
|
||||||
interval: 10s
|
|
||||||
timeout: 5s
|
|
||||||
retries: 3
|
|
||||||
start_period: 5s
|
|
||||||
deploy:
|
|
||||||
resources:
|
|
||||||
limits:
|
|
||||||
cpus: '1'
|
|
||||||
memory: 1G
|
|
||||||
|
|
||||||
dm-init:
|
|
||||||
image: pingcap/dm:latest
|
|
||||||
container_name: dm-init
|
|
||||||
volumes:
|
|
||||||
- ./configs:/configs:ro
|
|
||||||
- ./scripts:/scripts:ro
|
|
||||||
environment:
|
|
||||||
- TEST_DB_HOST=${TEST_DB_HOST}
|
|
||||||
- TEST_DB_PORT=${TEST_DB_PORT}
|
|
||||||
- TEST_DB_USER=${TEST_DB_USER}
|
|
||||||
- TEST_DB_PASSWORD=${TEST_DB_PASSWORD}
|
|
||||||
- DATABASE_NAME=${DATABASE_NAME}
|
|
||||||
- TABLES=${TABLES}
|
|
||||||
depends_on:
|
|
||||||
dm-worker:
|
|
||||||
condition: service_healthy
|
|
||||||
command: ["/bin/sh", "/scripts/init-dm.sh"]
|
|
||||||
restart: "no"
|
|
||||||
|
|
||||||
volumes:
|
volumes:
|
||||||
tidb_data:
|
tidb_data:
|
||||||
driver: local
|
driver: local
|
||||||
dm_master_data:
|
|
||||||
driver: local
|
|
||||||
dm_worker_data:
|
|
||||||
driver: local
|
|
||||||
|
|
||||||
networks:
|
networks:
|
||||||
default:
|
default:
|
||||||
|
|
|
||||||
|
|
@ -1,86 +0,0 @@
|
||||||
#!/bin/bash
|
|
||||||
|
|
||||||
echo "☁️ Exporting data from TiDB Cloud..."
|
|
||||||
|
|
||||||
# Check if .env exists
|
|
||||||
if [ ! -f .env ]; then
|
|
||||||
echo "❌ .env file not found!"
|
|
||||||
echo "📝 Please create .env file with your TiDB Cloud credentials"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Source environment variables
|
|
||||||
source .env
|
|
||||||
|
|
||||||
# Validate required variables
|
|
||||||
if [ -z "$TEST_DB_HOST" ] || [ -z "$TEST_DB_USER" ] || [ -z "$TEST_DB_PASSWORD" ]; then
|
|
||||||
echo "❌ Missing database credentials in .env"
|
|
||||||
echo "📝 Required: TEST_DB_HOST, TEST_DB_USER, TEST_DB_PASSWORD"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Create export directory
|
|
||||||
EXPORT_DIR="/tmp/tidb-cloud-export"
|
|
||||||
mkdir -p $EXPORT_DIR
|
|
||||||
|
|
||||||
# Test connection
|
|
||||||
echo "🔍 Testing connection to TiDB Cloud..."
|
|
||||||
mysql -h $TEST_DB_HOST -P ${TEST_DB_PORT:-4000} -u $TEST_DB_USER -p$TEST_DB_PASSWORD -e "SELECT 1" >/dev/null 2>&1
|
|
||||||
if [ $? -ne 0 ]; then
|
|
||||||
echo "❌ Cannot connect to TiDB Cloud"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo "✅ Connected successfully"
|
|
||||||
|
|
||||||
# Export schema using SQL queries
|
|
||||||
echo "📦 Exporting schema..."
|
|
||||||
|
|
||||||
# Create database statement
|
|
||||||
echo "CREATE DATABASE IF NOT EXISTS \`${DATABASE_NAME:-workflow_local}\`;
|
|
||||||
USE \`${DATABASE_NAME:-workflow_local}\`;
|
|
||||||
" > $EXPORT_DIR/schema.sql
|
|
||||||
|
|
||||||
# Get table schemas
|
|
||||||
for table in ${TABLES//,/ }; do
|
|
||||||
echo "-- Table: $table"
|
|
||||||
mysql -h $TEST_DB_HOST -P ${TEST_DB_PORT:-4000} -u $TEST_DB_USER -p$TEST_DB_PASSWORD -e "SHOW CREATE TABLE \`${DATABASE_NAME:-workflow_local}\`.$table;" -N -s | cut -f2 >> $EXPORT_DIR/schema.sql
|
|
||||||
echo ";" >> $EXPORT_DIR/schema.sql
|
|
||||||
echo "" >> $EXPORT_DIR/schema.sql
|
|
||||||
done
|
|
||||||
|
|
||||||
# Check if export was successful
|
|
||||||
if [ ! -s "$EXPORT_DIR/schema.sql" ]; then
|
|
||||||
echo "❌ Schema export failed - empty file"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo "✅ Schema exported to $EXPORT_DIR/schema.sql"
|
|
||||||
|
|
||||||
# Export data using SQL
|
|
||||||
echo "📦 Exporting data..."
|
|
||||||
|
|
||||||
# Clear data file
|
|
||||||
> $EXPORT_DIR/data.sql
|
|
||||||
|
|
||||||
# Export data for each table
|
|
||||||
for table in ${TABLES//,/ }; do
|
|
||||||
echo "-- Data for table: $table"
|
|
||||||
# Simple approach: export as CSV and convert to INSERT statements
|
|
||||||
mysql -h $TEST_DB_HOST -P ${TEST_DB_PORT:-4000} -u $TEST_DB_USER -p$TEST_DB_PASSWORD -e "SELECT * FROM \`${DATABASE_NAME:-workflow_local}\`.$table;" | sed '1d' > $EXPORT_DIR/${table}.csv
|
|
||||||
|
|
||||||
# If we have data, convert to INSERT statements
|
|
||||||
if [ -s "$EXPORT_DIR/${table}.csv" ]; then
|
|
||||||
# This is a simplified approach - for production use, you'd want a more robust CSV to SQL converter
|
|
||||||
echo "-- Note: Data export for $table requires manual conversion from CSV" >> $EXPORT_DIR/data.sql
|
|
||||||
echo "-- CSV file location: $EXPORT_DIR/${table}.csv" >> $EXPORT_DIR/data.sql
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
|
|
||||||
echo "⚠️ Data export completed - CSV files created for manual import"
|
|
||||||
echo "📂 Export completed successfully!"
|
|
||||||
echo " Schema: $EXPORT_DIR/schema.sql"
|
|
||||||
echo " Data CSV files:"
|
|
||||||
for table in ${TABLES//,/ }; do
|
|
||||||
echo " $EXPORT_DIR/${table}.csv"
|
|
||||||
done
|
|
||||||
|
|
@ -1,91 +0,0 @@
|
||||||
#!/bin/bash
|
|
||||||
|
|
||||||
echo "🏠 Importing data to local TiDB..."
|
|
||||||
|
|
||||||
# Check if local TiDB is accessible
|
|
||||||
mysql -h 127.0.0.1 -P 4000 -u root -e "SELECT 1" >/dev/null 2>&1
|
|
||||||
if [ $? -ne 0 ]; then
|
|
||||||
echo "❌ Cannot connect to local TiDB"
|
|
||||||
echo "📝 Make sure TiDB is running: ./start.sh"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Check if export files exist
|
|
||||||
EXPORT_DIR="/tmp/tidb-cloud-export"
|
|
||||||
if [ ! -f "$EXPORT_DIR/schema.sql" ]; then
|
|
||||||
echo "❌ Export files not found!"
|
|
||||||
echo "📝 Run export-cloud.sh first"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Import schema
|
|
||||||
echo "🏗️ Importing schema..."
|
|
||||||
mysql -h 127.0.0.1 -P 4000 -u root < $EXPORT_DIR/schema.sql 2>/dev/null
|
|
||||||
|
|
||||||
if [ $? -ne 0 ]; then
|
|
||||||
echo "❌ Schema import failed"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo "✅ Schema imported successfully"
|
|
||||||
|
|
||||||
# Import data from CSV files
|
|
||||||
echo "📥 Importing data..."
|
|
||||||
for table in ${TABLES//,/ }; do
|
|
||||||
if [ -f "$EXPORT_DIR/${table}.csv" ]; then
|
|
||||||
echo " Importing data for table: $table"
|
|
||||||
# Count lines in CSV (excluding header if present)
|
|
||||||
line_count=$(wc -l < "$EXPORT_DIR/${table}.csv" | tr -d ' ')
|
|
||||||
|
|
||||||
if [ "$line_count" -gt 0 ]; then
|
|
||||||
# Use LOAD DATA LOCAL INFILE to import CSV
|
|
||||||
mysql -h 127.0.0.1 -P 4000 -u root --local-infile=1 -e "
|
|
||||||
USE ${DATABASE_NAME:-workflow_local};
|
|
||||||
LOAD DATA LOCAL INFILE '$EXPORT_DIR/${table}.csv'
|
|
||||||
INTO TABLE $table
|
|
||||||
FIELDS TERMINATED BY '\t'
|
|
||||||
LINES TERMINATED BY '\n'
|
|
||||||
IGNORE 0 LINES;
|
|
||||||
" 2>/dev/null
|
|
||||||
|
|
||||||
if [ $? -ne 0 ]; then
|
|
||||||
echo "⚠️ Warning: Failed to import data for table $table"
|
|
||||||
# Try alternative method - read CSV and generate INSERT statements
|
|
||||||
echo " Trying alternative import method..."
|
|
||||||
while IFS=$'\t' read -r col1 col2 col3 col4 col5; do
|
|
||||||
# Escape single quotes
|
|
||||||
col1_escaped=$(echo "$col1" | sed "s/'/''/g")
|
|
||||||
col2_escaped=$(echo "$col2" | sed "s/'/''/g")
|
|
||||||
col3_escaped=$(echo "$col3" | sed "s/'/''/g")
|
|
||||||
col4_escaped=$(echo "$col4" | sed "s/'/''/g")
|
|
||||||
col5_escaped=$(echo "$col5" | sed "s/'/''/g" | sed "s/NULL//")
|
|
||||||
|
|
||||||
# Handle NULL values
|
|
||||||
if [ "$col5_escaped" = "" ]; then
|
|
||||||
col5_sql="NULL"
|
|
||||||
else
|
|
||||||
col5_sql="'$col5_escaped'"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Only insert if we have data
|
|
||||||
if [ -n "$col1" ]; then
|
|
||||||
mysql -h 127.0.0.1 -P 4000 -u root -e "
|
|
||||||
USE ${DATABASE_NAME:-workflow_local};
|
|
||||||
INSERT INTO $table (id, name, description, type, parent_plan_id)
|
|
||||||
VALUES ('$col1_escaped', '$col2_escaped', '$col3_escaped', '$col4_escaped', $col5_sql);
|
|
||||||
" 2>/dev/null
|
|
||||||
fi
|
|
||||||
done < "$EXPORT_DIR/${table}.csv"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Count rows imported
|
|
||||||
row_count=$(mysql -h 127.0.0.1 -P 4000 -u root -e "USE ${DATABASE_NAME:-workflow_local}; SELECT COUNT(*) FROM $table;" -N -s 2>/dev/null)
|
|
||||||
echo " ✅ Imported $row_count rows into $table"
|
|
||||||
else
|
|
||||||
echo " ℹ️ No data to import for table $table"
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
|
|
||||||
echo "✅ Data import completed"
|
|
||||||
echo "🎉 Import completed!"
|
|
||||||
|
|
@ -1,75 +0,0 @@
|
||||||
#!/bin/sh
|
|
||||||
set -e
|
|
||||||
|
|
||||||
echo "Waiting for DM master to be ready..."
|
|
||||||
sleep 5
|
|
||||||
|
|
||||||
# Check if it's TiDB Cloud (requires TLS)
|
|
||||||
if echo "$TEST_DB_HOST" | grep -q "tidbcloud.com"; then
|
|
||||||
echo "Detected TiDB Cloud - downloading CA certificate for TLS..."
|
|
||||||
wget -q -O /tmp/isrgrootx1.pem https://letsencrypt.org/certs/isrgrootx1.pem
|
|
||||||
|
|
||||||
# Generate source.yaml with TLS for TiDB Cloud
|
|
||||||
cat > /tmp/source.yaml <<EOF
|
|
||||||
source-id: "test-tidb"
|
|
||||||
enable-gtid: false
|
|
||||||
enable-relay: false
|
|
||||||
server-id: 101
|
|
||||||
from:
|
|
||||||
host: "$TEST_DB_HOST"
|
|
||||||
port: $TEST_DB_PORT
|
|
||||||
user: "$TEST_DB_USER"
|
|
||||||
password: "$TEST_DB_PASSWORD"
|
|
||||||
security:
|
|
||||||
ssl-ca: "/tmp/isrgrootx1.pem"
|
|
||||||
EOF
|
|
||||||
else
|
|
||||||
# Generate source.yaml without TLS for regular TiDB
|
|
||||||
cat > /tmp/source.yaml <<EOF
|
|
||||||
source-id: "test-tidb"
|
|
||||||
enable-gtid: false
|
|
||||||
enable-relay: false
|
|
||||||
from:
|
|
||||||
host: "$TEST_DB_HOST"
|
|
||||||
port: $TEST_DB_PORT
|
|
||||||
user: "$TEST_DB_USER"
|
|
||||||
password: "$TEST_DB_PASSWORD"
|
|
||||||
EOF
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo "Creating DM source configuration..."
|
|
||||||
/dmctl --master-addr=dm-master:8261 operate-source create /tmp/source.yaml || true
|
|
||||||
|
|
||||||
# Generate task.yaml with multiple tables
|
|
||||||
echo "name: \"test-to-local\"
|
|
||||||
task-mode: \"all\"
|
|
||||||
target-database:
|
|
||||||
host: \"tidb\"
|
|
||||||
port: 4000
|
|
||||||
user: \"root\"
|
|
||||||
password: \"\"
|
|
||||||
|
|
||||||
mysql-instances:
|
|
||||||
- source-id: \"test-tidb\"
|
|
||||||
block-allow-list: \"sync-tables\"
|
|
||||||
|
|
||||||
block-allow-list:
|
|
||||||
sync-tables:
|
|
||||||
do-dbs: [\"$DATABASE_NAME\"]
|
|
||||||
do-tables:" > /tmp/task.yaml
|
|
||||||
|
|
||||||
# Add each table from TABLES variable
|
|
||||||
IFS=',' read -ra TABLE_ARRAY <<< "$TABLES"
|
|
||||||
for table in "${TABLE_ARRAY[@]}"; do
|
|
||||||
table=$(echo "$table" | xargs) # trim whitespace
|
|
||||||
echo " - db-name: \"$DATABASE_NAME\"
|
|
||||||
tbl-name: \"$table\"" >> /tmp/task.yaml
|
|
||||||
done
|
|
||||||
|
|
||||||
echo "Starting DM sync task..."
|
|
||||||
/dmctl --master-addr=dm-master:8261 start-task /tmp/task.yaml || true
|
|
||||||
|
|
||||||
echo "Checking task status..."
|
|
||||||
/dmctl --master-addr=dm-master:8261 query-status test-to-local
|
|
||||||
|
|
||||||
echo "DM initialization complete!"
|
|
||||||
26
start.sh
26
start.sh
|
|
@ -1,26 +1,7 @@
|
||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
set -e
|
set -e
|
||||||
|
|
||||||
echo "🚀 Starting TiDB Local Environment..."
|
echo "🚀 Starting Minimal TiDB Environment..."
|
||||||
|
|
||||||
# Check if .env exists
|
|
||||||
if [ ! -f .env ]; then
|
|
||||||
echo "⚠️ .env file not found!"
|
|
||||||
echo "📝 Creating .env from template..."
|
|
||||||
cp .env.example .env
|
|
||||||
echo "✏️ Please edit .env file with your test database credentials"
|
|
||||||
echo " Then run this script again."
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Source .env to check if configured
|
|
||||||
source .env
|
|
||||||
|
|
||||||
if [ "$TEST_DB_HOST" = "your-test-tidb-host" ]; then
|
|
||||||
echo "⚠️ .env file needs configuration!"
|
|
||||||
echo "✏️ Please edit .env file with your test database credentials"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo "🐳 Starting Docker containers..."
|
echo "🐳 Starting Docker containers..."
|
||||||
docker compose up -d
|
docker compose up -d
|
||||||
|
|
@ -36,15 +17,10 @@ echo "📊 Connection Info:"
|
||||||
echo " TiDB: mysql -h 127.0.0.1 -P 4000 -u root"
|
echo " TiDB: mysql -h 127.0.0.1 -P 4000 -u root"
|
||||||
echo " DataGrip: Host: 127.0.0.1, Port: 4000, User: root, Password: (empty)"
|
echo " DataGrip: Host: 127.0.0.1, Port: 4000, User: root, Password: (empty)"
|
||||||
echo ""
|
echo ""
|
||||||
echo "🔄 To sync data from TiDB Cloud:"
|
|
||||||
echo " ./sync-data.sh"
|
|
||||||
echo ""
|
|
||||||
echo "🔍 Useful commands:"
|
echo "🔍 Useful commands:"
|
||||||
echo " Test connection: ./test-connection.sh"
|
echo " Test connection: ./test-connection.sh"
|
||||||
echo " Sync data: ./sync-data.sh"
|
|
||||||
echo " View logs: docker compose logs -f"
|
echo " View logs: docker compose logs -f"
|
||||||
echo " Stop environment: docker compose down"
|
echo " Stop environment: docker compose down"
|
||||||
echo ""
|
echo ""
|
||||||
echo "📖 For DataGrip setup: see DATAGRIP_SETUP.md"
|
echo "📖 For DataGrip setup: see DATAGRIP_SETUP.md"
|
||||||
echo "📘 For TiDB Cloud migration: see TIDB_CLOUD_MIGRATION.md"
|
|
||||||
echo ""
|
echo ""
|
||||||
|
|
|
||||||
22
status.sh
22
status.sh
|
|
@ -1,22 +0,0 @@
|
||||||
#!/bin/bash
|
|
||||||
|
|
||||||
echo "🔍 Checking DM Sync Status..."
|
|
||||||
echo ""
|
|
||||||
|
|
||||||
# Check if containers are running
|
|
||||||
if ! docker ps | grep -q dm-master; then
|
|
||||||
echo "❌ DM Master is not running. Start the environment first:"
|
|
||||||
echo " ./start.sh"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo "📡 Source Configuration:"
|
|
||||||
docker exec dm-master /dmctl --master-addr=dm-master:8261 operate-source show
|
|
||||||
|
|
||||||
echo ""
|
|
||||||
echo "📊 Task Status:"
|
|
||||||
docker exec dm-master /dmctl --master-addr=dm-master:8261 query-status test-to-local
|
|
||||||
|
|
||||||
echo ""
|
|
||||||
echo "💾 Local TiDB Databases:"
|
|
||||||
docker exec tidb mysql -h 127.0.0.1 -P 4000 -u root -e "SHOW DATABASES;"
|
|
||||||
|
|
@ -1,81 +0,0 @@
|
||||||
#!/bin/bash
|
|
||||||
|
|
||||||
echo "⚠️ WARNING: TiDB Data Migration (DM) is not compatible with TiDB Cloud Serverless"
|
|
||||||
echo "⚠️ This script is deprecated. Use ./sync-data.sh instead."
|
|
||||||
echo ""
|
|
||||||
|
|
||||||
echo "For officially supported migration approaches, see TIDB_CLOUD_MIGRATION.md"
|
|
||||||
echo ""
|
|
||||||
|
|
||||||
TASK_NAME="test-to-local"
|
|
||||||
DMCTL="docker exec dm-master /dmctl --master-addr=dm-master:8261"
|
|
||||||
|
|
||||||
show_usage() {
|
|
||||||
echo "Usage: ./sync-control.sh [command]"
|
|
||||||
echo ""
|
|
||||||
echo "Commands:"
|
|
||||||
echo " status - Show sync task status"
|
|
||||||
echo " start - Start the sync task"
|
|
||||||
echo " stop - Stop the sync task"
|
|
||||||
echo " pause - Pause the sync task"
|
|
||||||
echo " resume - Resume the sync task"
|
|
||||||
echo " restart - Restart the sync task (stop + start)"
|
|
||||||
echo " reinit - Re-initialize DM configuration"
|
|
||||||
echo ""
|
|
||||||
}
|
|
||||||
|
|
||||||
check_dm() {
|
|
||||||
if ! docker ps | grep -q dm-master; then
|
|
||||||
echo "❌ DM Master is not running"
|
|
||||||
echo " Start with: ./start.sh"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
case "$1" in
|
|
||||||
status)
|
|
||||||
check_dm
|
|
||||||
echo "📊 Checking sync status for task: $TASK_NAME"
|
|
||||||
echo ""
|
|
||||||
$DMCTL query-status $TASK_NAME
|
|
||||||
;;
|
|
||||||
start)
|
|
||||||
check_dm
|
|
||||||
echo "▶️ Starting sync task: $TASK_NAME"
|
|
||||||
$DMCTL start-task $TASK_NAME
|
|
||||||
;;
|
|
||||||
stop)
|
|
||||||
check_dm
|
|
||||||
echo "⏹️ Stopping sync task: $TASK_NAME"
|
|
||||||
$DMCTL stop-task $TASK_NAME
|
|
||||||
;;
|
|
||||||
pause)
|
|
||||||
check_dm
|
|
||||||
echo "⏸️ Pausing sync task: $TASK_NAME"
|
|
||||||
$DMCTL pause-task $TASK_NAME
|
|
||||||
;;
|
|
||||||
resume)
|
|
||||||
check_dm
|
|
||||||
echo "▶️ Resuming sync task: $TASK_NAME"
|
|
||||||
$DMCTL resume-task $TASK_NAME
|
|
||||||
;;
|
|
||||||
restart)
|
|
||||||
check_dm
|
|
||||||
echo "🔄 Restarting sync task: $TASK_NAME"
|
|
||||||
$DMCTL stop-task $TASK_NAME
|
|
||||||
sleep 2
|
|
||||||
$DMCTL start-task $TASK_NAME
|
|
||||||
;;
|
|
||||||
reinit)
|
|
||||||
echo "🔄 Re-initializing DM configuration..."
|
|
||||||
docker compose up -d dm-init
|
|
||||||
echo ""
|
|
||||||
echo "⏳ Waiting for initialization..."
|
|
||||||
sleep 5
|
|
||||||
docker logs dm-init
|
|
||||||
;;
|
|
||||||
*)
|
|
||||||
show_usage
|
|
||||||
exit 1
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
29
sync-data.sh
29
sync-data.sh
|
|
@ -1,29 +0,0 @@
|
||||||
#!/bin/bash
|
|
||||||
|
|
||||||
echo "🔄 Syncing data from TiDB Cloud to local TiDB..."
|
|
||||||
echo ""
|
|
||||||
|
|
||||||
# Export from TiDB Cloud
|
|
||||||
echo "☁️ Step 1: Exporting from TiDB Cloud"
|
|
||||||
./export-cloud.sh
|
|
||||||
if [ $? -ne 0 ]; then
|
|
||||||
echo "❌ Export failed"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo ""
|
|
||||||
|
|
||||||
# Import to local TiDB
|
|
||||||
echo "🏠 Step 2: Importing to local TiDB"
|
|
||||||
./import-local.sh
|
|
||||||
if [ $? -ne 0 ]; then
|
|
||||||
echo "❌ Import failed"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo ""
|
|
||||||
echo "✅ Data sync completed successfully!"
|
|
||||||
echo ""
|
|
||||||
echo "📊 Verify data:"
|
|
||||||
echo " mysql -h 127.0.0.1 -P 4000 -u root -e 'USE ${DATABASE_NAME:-workflow_local}; SHOW TABLES;'"
|
|
||||||
echo " mysql -h 127.0.0.1 -P 4000 -u root -e 'USE ${DATABASE_NAME:-workflow_local}; SELECT COUNT(*) FROM ${TABLES%%,*};'"
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
|
|
||||||
echo "🔌 Testing TiDB Connection for DataGrip..."
|
echo "🔌 Testing TiDB Connection..."
|
||||||
echo ""
|
echo ""
|
||||||
|
|
||||||
# Check if TiDB container is running
|
# Check if TiDB container is running
|
||||||
|
|
@ -22,7 +22,7 @@ if docker exec tidb mysql -h 127.0.0.1 -P 4000 -u root -e "SELECT VERSION();" 2>
|
||||||
echo "📊 Available databases:"
|
echo "📊 Available databases:"
|
||||||
docker exec tidb mysql -h 127.0.0.1 -P 4000 -u root -e "SHOW DATABASES;" 2>/dev/null
|
docker exec tidb mysql -h 127.0.0.1 -P 4000 -u root -e "SHOW DATABASES;" 2>/dev/null
|
||||||
echo ""
|
echo ""
|
||||||
echo "🎯 DataGrip Connection Settings:"
|
echo "🎯 Connection Settings:"
|
||||||
echo " Host: 127.0.0.1"
|
echo " Host: 127.0.0.1"
|
||||||
echo " Port: 4000"
|
echo " Port: 4000"
|
||||||
echo " User: root"
|
echo " User: root"
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue