README.md

@@ -1,22 +1,43 @@
 # TiDB Local Development Environment
 
-A minimal TiDB instance for local development.
+A minimal TiDB instance with Data Migration (DM) for syncing data from test environments.
 
 ## Architecture
 
 ```
-┌─────────────────────────────────────────────────┐
+┌─────────────────────────────────────────────────────────────┐
-│              Your macOS (OrbStack)              │
+│                    Your macOS (OrbStack)                    │
-│                                                 │
+│                                                             │
-│  ┌──────────────┐      ┌──────────────┐        │
+│  ┌──────────────┐      ┌──────────────┐                   │
-│  │   DataGrip   │──────▶│     TiDB     │        │
+│  │   DataGrip   │──────▶│     TiDB     │                   │
-│  │  (port 4000) │      │  (Standalone) │        │
+│  │  (port 4000) │      │  (Standalone) │                   │
-│  └──────────────┘      └──────────────┘        │
+│  └──────────────┘      └───────▲───────┘                   │
-└─────────────────────────────────────────────────┘
+│                                │                             │
+│                        ┌───────┴────────┐                   │
+│                        │   DM Worker    │                   │
+│                        │  (Sync Engine) │                   │
+│                        └───────▲────────┘                   │
+│                                │                             │
+│                        ┌───────┴────────┐                   │
+│                        │   DM Master    │                   │
+│                        │ (Orchestrator) │                   │
+│                        └───────▲────────┘                   │
+│                                │                             │
+└────────────────────────────────┼─────────────────────────────┘
+                                 │
+                         (Continuous Sync)
+                                 │
+                    ┌────────────▼─────────────┐
+                    │   Test TiDB Instance     │
+                    │  (Remote Environment)    │
+                    └──────────────────────────┘
 ```
 
 **Components:**
 - **TiDB (Standalone Mode)**: Runs with embedded storage (unistore), no separate PD/TiKV needed
+- **DM Master**: Manages data migration tasks
+- **DM Worker**: Executes the actual data synchronization from test to local
+- **DataGrip/MySQL Clients**: Connect directly to TiDB on port 4000
 
 ## Quick Reference
 
@@ -24,15 +45,24 @@ A minimal TiDB instance for local development.
 | Service | Host | Port | User | Password |
 |---------|------|------|------|----------|
 | TiDB | `127.0.0.1` | `4000` | `root` | _(empty)_ |
+| DM Master | `127.0.0.1` | `8261` | - | - |
 
 ### Useful Commands
-``bash
+```bash
-# Start environment
+# Start environment (auto-starts sync)
 ./start.sh
 
 # Test connection
 ./test-connection.sh
 
+# Check sync status
+./status.sh
+# or use the sync control script:
+./sync-control.sh status
+
+# Control sync task
+./sync-control.sh [start|stop|pause|resume|restart|reinit]
+
 # Connect with MySQL client
 mysql -h 127.0.0.1 -P 4000 -u root
 
@@ -49,6 +79,7 @@ For DataGrip/DBeaver setup, see [DATAGRIP_SETUP.md](DATAGRIP_SETUP.md)
 
 - macOS with OrbStack (or Docker Desktop)
 - Docker Compose v2 (command: `docker compose`, not `docker-compose`)
+- Access to test TiDB instance
 
 **Check your setup:**
 ```bash
@@ -59,17 +90,55 @@ For DataGrip/DBeaver setup, see [DATAGRIP_SETUP.md](DATAGRIP_SETUP.md)
 
 ## Configuration
 
-No configuration needed for the basic setup.
+1. **Copy and edit `.env` file**:
+```bash
+cp .env.example .env
+# Edit .env with your test database credentials
+```
+
+Required variables:
+- `TEST_DB_HOST`: Your test TiDB host
+- `TEST_DB_PORT`: Test TiDB port (default: 4000)
+- `TEST_DB_USER`: Test database username
+- `TEST_DB_PASSWORD`: Test database password
+- `DATABASE_NAME`: Database to sync
+- `TABLES`: Comma-separated list of tables (e.g., "table1,table2,table3")
 
 ## Usage
 
+### How the Sync Works
+
+The data synchronization is **automatically configured and started** when you run `./start.sh`:
+
+1. **Automatic Setup** (recommended):
+   - The `dm-init` container runs [`scripts/init-dm.sh`](scripts/init-dm.sh)
+   - It generates the actual DM task config from your `.env` variables
+   - The sync task is automatically started
+   - No manual intervention needed!
+
+2. **Manual Control** (optional):
+   - Use [`./sync-control.sh`](sync-control.sh) for easy management
+   - Or use `dmctl` commands directly (see Manual DM Operations below)
+
+**Note:** [`configs/task.yaml`](configs/task.yaml) is just a template. The real task config is generated dynamically at runtime.
+
+**For detailed sync operations, see [SYNC_GUIDE.md](SYNC_GUIDE.md)**
+
 ### Start the environment
-``bash
+```bash
 docker compose up -d
 ```
 
 This will:
 1. Start TiDB in standalone mode
+2. Start DM master and worker
+3. Automatically configure the data source and sync task
+4. Begin syncing data from test to local
+
+### Check sync status
+```bash
+docker exec dm-master /dmctl --master-addr=dm-master:8261 query-status test-to-local
+```
 
 ### Connect to local TiDB
 
@@ -87,16 +156,17 @@ mysql -h 127.0.0.1 -P 4000 -u root
 See [DATAGRIP_SETUP.md](DATAGRIP_SETUP.md) for detailed client setup instructions.
 
 ### View logs
-```
+```bash
 # All services
 docker compose logs -f
 
 # Specific service
 docker compose logs -f tidb
+docker compose logs -f dm-worker
 ```
 
 ### Stop the environment
-``bash
+```bash
 docker compose down
 ```
 
@@ -105,16 +175,76 @@ docker compose down
 docker compose down -v
 ```
 
+## Manual DM Operations
+
+### Check source configuration
+```bash
+docker exec dm-master /dmctl --master-addr=dm-master:8261 operate-source show
+```
+
+### Stop sync task
+```bash
+docker exec dm-master /dmctl --master-addr=dm-master:8261 stop-task test-to-local
+```
+
+### Start sync task
+```bash
+docker exec dm-master /dmctl --master-addr=dm-master:8261 start-task /configs/task.yaml
+```
+
+### Pause sync task
+```bash
+docker exec dm-master /dmctl --master-addr=dm-master:8261 pause-task test-to-local
+```
+
+### Resume sync task
+```bash
+docker exec dm-master /dmctl --master-addr=dm-master:8261 resume-task test-to-local
+```
+
 ## Troubleshooting
 
+### Common Issues
+
+For detailed troubleshooting, see [TROUBLESHOOTING.md](TROUBLESHOOTING.md)
+
+### Quick Checks
+
+#### TiDB health check failing
+```bash
+# Check if TiDB is healthy
+docker ps | grep tidb
+
+# Should show: (healthy)
+# If not, check logs:
+docker logs tidb
+```
+
+#### DM task fails to start
+- Check if test database is accessible from container
+- Verify credentials in `.env`
+- Check logs: `docker-compose logs dm-worker`
+
+### Tables not syncing
+- Ensure tables exist in source database
+- Verify table names in `TABLES` variable
+- Check task status for specific errors
+
 ### TiDB connection issues
 - Verify TiDB is running: `docker ps | grep tidb`
 - Check health: `docker exec tidb mysql -h 127.0.0.1 -P 4000 -u root -e "SELECT 1"`
 
+### Re-initialize DM configuration
+```bash
+docker compose up -d dm-init
+```
+
 ## Resource Usage
 
 Default resource limits (suitable for local development):
 - TiDB: 2 CPU, 2GB RAM
+- DM Worker: 1 CPU, 1GB RAM
+- DM Master: 0.5 CPU, 512MB RAM
 
 Adjust in `docker-compose.yml` if needed.
 
@@ -123,3 +253,5 @@ Adjust in `docker-compose.yml` if needed.
 - **Docker Compose v2**: This project uses `docker compose` (v2 syntax). If you have v1, either upgrade or create an alias: `alias docker-compose='docker compose'`
 - **Standalone Mode**: TiDB runs without distributed storage, suitable for development only
 - **Data Persistence**: Data is stored in Docker volumes, persists across restarts
+- **Sync Mode**: Configured for full + incremental sync ("all" mode)
+- **OrbStack DNS**: Uses `.orb.local` hostnames for container networking

SYNC_GUIDE.md

@@ -0,0 +1,354 @@
+# Data Sync Guide
+
+## How Sync Works
+
+Your TiDB Data Migration (DM) setup continuously syncs data from your test environment to the local TiDB instance.
+
+```
+Test TiDB  ──────┐
+                 │
+        (DM reads changes)
+                 │
+                 ▼
+             DM Worker
+                 │
+        (Applies to local)
+                 │
+                 ▼
+           Local TiDB
+```
+
+## Automatic Sync Setup
+
+When you run `./start.sh`, the sync is **automatically configured and started**:
+
+1. ✅ Reads your `.env` file for credentials and table list
+2. ✅ Generates the DM task configuration
+3. ✅ Configures the source connection (test TiDB)
+4. ✅ Starts the sync task
+5. ✅ Begins syncing data (full + incremental)
+
+**You don't need to do anything manually!**
+
+## Sync Modes
+
+The sync is configured with `task-mode: "all"`:
+- **Full sync**: Initial copy of all existing data
+- **Incremental sync**: Continuous replication of changes (INSERT, UPDATE, DELETE)
+
+## Managing Sync
+
+### Easy Way (Recommended)
+
+Use the [`sync-control.sh`](sync-control.sh) script:
+
+```bash
+# Check sync status
+./sync-control.sh status
+
+# Stop sync
+./sync-control.sh stop
+
+# Start sync
+./sync-control.sh start
+
+# Pause sync (temporarily)
+./sync-control.sh pause
+
+# Resume sync
+./sync-control.sh resume
+
+# Restart sync (stop + start)
+./sync-control.sh restart
+
+# Re-initialize configuration
+./sync-control.sh reinit
+```
+
+### Advanced Way (dmctl)
+
+Use `dmctl` directly:
+
+```bash
+# Check status
+docker exec dm-master /dmctl --master-addr=dm-master:8261 query-status test-to-local
+
+# Stop task
+docker exec dm-master /dmctl --master-addr=dm-master:8261 stop-task test-to-local
+
+# Start task
+docker exec dm-master /dmctl --master-addr=dm-master:8261 start-task test-to-local
+
+# Pause task
+docker exec dm-master /dmctl --master-addr=dm-master:8261 pause-task test-to-local
+
+# Resume task
+docker exec dm-master /dmctl --master-addr=dm-master:8261 resume-task test-to-local
+```
+
+## Checking Sync Status
+
+### Quick Check
+
+```bash
+./status.sh
+```
+
+This shows:
+- Source configuration
+- Task status (running, paused, stopped)
+- Current sync position
+- Error messages (if any)
+- Local databases
+
+### Detailed Status
+
+```bash
+./sync-control.sh status
+```
+
+### Verify Data Sync
+
+Connect to local TiDB and verify:
+
+```sql
+-- Connect
+mysql -h 127.0.0.1 -P 4000 -u root
+
+-- Check databases
+SHOW DATABASES;
+
+-- Switch to your database
+USE your_database;
+
+-- Check tables
+SHOW TABLES;
+
+-- Verify row count
+SELECT COUNT(*) FROM table1;
+
+-- Compare with source (if you have access)
+-- Run the same query on test environment
+```
+
+## Configuration Files
+
+### Environment Variables (`.env`)
+
+This is where you configure what to sync:
+
+```bash
+# Source database
+TEST_DB_HOST=your-test-tidb-host
+TEST_DB_PORT=4000
+TEST_DB_USER=root
+TEST_DB_PASSWORD=your-password
+
+# What to sync
+DATABASE_NAME=your_database
+TABLES="table1,table2,table3"
+```
+
+### Task Template (`configs/task.yaml`)
+
+This is just a **template for reference**. The actual task config is generated by [`scripts/init-dm.sh`](scripts/init-dm.sh).
+
+### Source Config (`configs/source.yaml`)
+
+Template for source database connection. Also generated dynamically.
+
+## Common Scenarios
+
+### Adding/Removing Tables
+
+1. Edit `.env` and update `TABLES` variable:
+   ```bash
+   TABLES="table1,table2,table3,new_table4"
+   ```
+
+2. Re-initialize:
+   ```bash
+   ./sync-control.sh reinit
+   ```
+
+### Changing Source Database
+
+1. Edit `.env` with new credentials
+2. Restart everything:
+   ```bash
+   docker compose down
+   ./start.sh
+   ```
+
+### Resetting Sync (Fresh Start)
+
+```bash
+# Stop and remove everything
+docker compose down -v
+
+# Start fresh
+./start.sh
+```
+
+### Pausing Sync Temporarily
+
+```bash
+# Pause (without stopping containers)
+./sync-control.sh pause
+
+# Resume when ready
+./sync-control.sh resume
+```
+
+## Monitoring
+
+### View Logs
+
+```bash
+# All services
+docker compose logs -f
+
+# DM Worker only
+docker compose logs -f dm-worker
+
+# DM Master only
+docker compose logs -f dm-master
+
+# Init script logs
+docker logs dm-init
+```
+
+### Check DM Master Status
+
+```bash
+docker exec dm-master /dmctl --master-addr=dm-master:8261 operate-source show
+```
+
+### Check DM Worker Status
+
+```bash
+docker ps | grep dm-worker
+```
+
+## Troubleshooting
+
+### Sync Not Starting
+
+**Check init logs:**
+```bash
+docker logs dm-init
+```
+
+**Common issues:**
+- Wrong credentials in `.env`
+- Test database not accessible
+- Tables don't exist in source
+
+**Solution:**
+```bash
+# Fix .env, then:
+./sync-control.sh reinit
+```
+
+### Sync Stopped with Errors
+
+**Check error message:**
+```bash
+./sync-control.sh status
+```
+
+**Common errors:**
+- Network connectivity issues
+- Permission denied on source
+- Table schema mismatch
+
+**Solution:**
+```bash
+# Fix the underlying issue, then:
+./sync-control.sh restart
+```
+
+### Data Not Syncing
+
+**Verify task is running:**
+```bash
+./sync-control.sh status
+```
+
+**Check if tables exist:**
+```bash
+# On source
+mysql -h $TEST_DB_HOST -P 4000 -u root -p -e "SHOW TABLES FROM your_database;"
+
+# On local
+mysql -h 127.0.0.1 -P 4000 -u root -e "SHOW TABLES FROM your_database;"
+```
+
+**Compare row counts:**
+```bash
+# Create a verification script
+mysql -h 127.0.0.1 -P 4000 -u root -e "SELECT COUNT(*) FROM your_database.table1;"
+```
+
+## Performance Tuning
+
+### Adjust Sync Speed
+
+Edit `docker-compose.yml`:
+
+```yaml
+dm-worker:
+  deploy:
+    resources:
+      limits:
+        cpus: '2'      # Increase CPU
+        memory: 2G     # Increase memory
+```
+
+Then restart:
+```bash
+docker compose down
+docker compose up -d
+```
+
+### Monitor Resource Usage
+
+```bash
+docker stats
+```
+
+## Best Practices
+
+1. **Always check status** after starting: `./status.sh`
+2. **Monitor logs** during initial sync: `docker compose logs -f dm-worker`
+3. **Verify data** in local TiDB after sync completes
+4. **Use pause/resume** instead of stop/start for temporary halts
+5. **Keep `.env` secure** - it contains credentials
+6. **Test connectivity** before sync: `./test-connection.sh`
+
+## FAQ
+
+**Q: Is the sync real-time?**  
+A: Near real-time. Changes are replicated with minimal delay (usually seconds).
+
+**Q: What happens if my laptop sleeps?**  
+A: Sync will resume automatically when containers restart.
+
+**Q: Can I sync from multiple sources?**  
+A: Yes, but requires manual DM configuration. This setup is for single source.
+
+**Q: Does it sync schema changes?**  
+A: Yes, DDL statements are replicated (CREATE, ALTER, DROP).
+
+**Q: Can I sync to a different database name locally?**  
+A: Requires custom task configuration. Default syncs to same database name.
+
+**Q: How do I exclude certain tables?**  
+A: Remove them from `TABLES` in `.env` and run `./sync-control.sh reinit`.
+
+## See Also
+
+- [README.md](README.md) - Main documentation
+- [DATAGRIP_SETUP.md](DATAGRIP_SETUP.md) - Connect with GUI clients
+- [scripts/init-dm.sh](scripts/init-dm.sh) - Initialization script
+- [TiDB DM Documentation](https://docs.pingcap.com/tidb-data-migration/stable)

TROUBLESHOOTING.md

@@ -0,0 +1,419 @@
+# Troubleshooting Guide
+
+## Common Issues and Solutions
+
+### 1. TiDB Health Check Failing
+
+#### Symptom
+```
+dependency failed to start: container tidb is unhealthy
+```
+
+Even though you can connect to TiDB from your host machine, Docker health check fails.
+
+#### Root Cause
+The original health check tried to use `mysql` command inside the TiDB container:
+```yaml
+healthcheck:
+  test: ["CMD", "mysql", "-h", "127.0.0.1", "-P", "4000", "-u", "root", "-e", "SELECT 1"]
+```
+
+The TiDB Docker image doesn't include the MySQL client binary, so this check always failed.
+
+#### Solution ✅
+Use TiDB's built-in HTTP status endpoint instead:
+```yaml
+healthcheck:
+  test: ["CMD", "wget", "-q", "-O-", "http://127.0.0.1:10080/status"]
+  interval: 10s
+  timeout: 5s
+  retries: 3
+  start_period: 10s
+```
+
+**Why this works:**
+- TiDB exposes a status endpoint on port 10080
+- `wget` is available in the container
+- Returns HTTP 200 when TiDB is ready
+- `start_period` gives TiDB time to initialize before health checks begin
+
+### 2. Docker Compose Version Warning
+
+#### Symptom
+```
+WARN[0000] version is obsolete, it will be ignored
+```
+
+#### Solution ✅
+Remove the `version` field from `docker-compose.yml`. Modern Docker Compose (v2) doesn't need it.
+
+**Before:**
+```yaml
+version: '3.8'
+services:
+  ...
+```
+
+**After:**
+```yaml
+services:
+  ...
+```
+
+### 3. Service Dependencies Not Starting in Order
+
+#### Symptom
+Services fail because dependencies aren't ready yet.
+
+#### Solution ✅
+Use proper health checks and dependency conditions:
+
+```yaml
+dm-worker:
+  depends_on:
+    tidb:
+      condition: service_healthy
+    dm-master:
+      condition: service_healthy
+```
+
+**Important:**
+- Each dependency must have a working health check
+- `start_period` prevents false negatives during startup
+
+### 4. dm-init Fails to Start
+
+#### Symptom
+```
+Error: dm-init exits immediately
+```
+
+#### Check:
+```bash
+docker logs dm-init
+```
+
+#### Common Causes:
+
+**a) .env not configured:**
+```bash
+# Check if .env exists and has real values
+cat .env
+```
+
+**Solution:**
+```bash
+# Copy template and edit
+cp .env.example .env
+vim .env
+```
+
+**b) Test database not reachable:**
+```bash
+# Test from dm-init container
+docker run --rm --network tidb-network pingcap/dm:latest \
+  sh -c "wget -q -O- http://tidb:10080/status"
+```
+
+**c) Script syntax error:**
+```bash
+# Check init script
+sh -n scripts/init-dm.sh
+```
+
+### 5. Containers Keep Restarting
+
+#### Check Status:
+```bash
+docker ps -a
+docker logs <container_name>
+```
+
+#### Common Issues:
+
+**a) Port already in use:**
+```
+Error: bind: address already in use
+```
+
+**Solution:** Change ports in `docker-compose.yml`:
+```yaml
+ports:
+  - "14000:4000"  # Changed from 4000:4000
+```
+
+**b) Out of memory:**
+```
+Error: OOM killed
+```
+
+**Solution:** Increase memory limits or free up system resources.
+
+**c) Permission issues:**
+```
+Error: permission denied
+```
+
+**Solution:** Check volume permissions or run:
+```bash
+docker compose down -v  # Remove volumes
+docker compose up -d    # Recreate
+```
+
+### 6. Sync Task Not Running
+
+#### Check Status:
+```bash
+./status.sh
+# or
+./sync-control.sh status
+```
+
+#### Common Issues:
+
+**a) Task not created:**
+```bash
+# Check if source is configured
+docker exec dm-master /dmctl --master-addr=dm-master:8261 operate-source show
+```
+
+**Solution:**
+```bash
+./sync-control.sh reinit
+```
+
+**b) Wrong credentials:**
+Check logs:
+```bash
+docker logs dm-worker
+```
+
+Fix `.env` and reinit:
+```bash
+vim .env
+./sync-control.sh reinit
+```
+
+**c) Table doesn't exist:**
+Verify tables exist on source database:
+```bash
+# Connect to test DB and check
+SHOW TABLES FROM your_database;
+```
+
+### 7. Connection Refused to TiDB
+
+#### Symptom
+```
+ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (61)
+```
+
+#### Checks:
+
+**a) Is TiDB running?**
+```bash
+docker ps | grep tidb
+```
+
+**b) Is it healthy?**
+```bash
+docker ps
+# Look for "(healthy)" status
+```
+
+**c) Is port exposed?**
+```bash
+docker port tidb
+# Should show: 4000/tcp -> 0.0.0.0:4000
+```
+
+**d) Test from inside container:**
+```bash
+docker exec tidb wget -q -O- http://127.0.0.1:10080/status
+```
+
+#### Solutions:
+
+**If container not running:**
+```bash
+docker compose up -d tidb
+docker logs tidb
+```
+
+**If unhealthy:**
+```bash
+# Wait for health check
+sleep 15
+docker ps
+
+# If still unhealthy, check logs
+docker logs tidb
+```
+
+**If port not exposed:**
+```bash
+# Recreate container
+docker compose down
+docker compose up -d
+```
+
+### 8. Data Not Syncing
+
+#### Verify sync is running:
+```bash
+./sync-control.sh status
+```
+
+#### Check sync lag:
+Look for "syncer" section in status output.
+
+#### Common Issues:
+
+**a) Sync paused:**
+```bash
+./sync-control.sh resume
+```
+
+**b) Sync stopped with error:**
+```bash
+# Check error in status output
+./sync-control.sh status
+
+# Fix the issue, then restart
+./sync-control.sh restart
+```
+
+**c) Network issues:**
+```bash
+# Test connectivity from dm-worker to source
+docker exec dm-worker ping -c 3 your-test-db-host
+```
+
+**d) Binlog not enabled on source:**
+Source database must have binlog enabled for incremental sync.
+
+### 9. Slow Sync Performance
+
+#### Check resource usage:
+```bash
+docker stats
+```
+
+#### Solutions:
+
+**a) Increase worker resources:**
+Edit `docker-compose.yml`:
+```yaml
+dm-worker:
+  deploy:
+    resources:
+      limits:
+        cpus: '2'
+        memory: 2G
+```
+
+**b) Optimize batch size:**
+See [TiDB DM Documentation](https://docs.pingcap.com/tidb-data-migration/stable/tune-configuration) for advanced tuning.
+
+### 10. Docker Compose v1 vs v2 Issues
+
+#### Symptom
+```
+docker: 'compose' is not a docker command
+```
+
+#### Solution
+See [DOCKER_COMPOSE_V2.md](DOCKER_COMPOSE_V2.md) for:
+- Upgrading to v2
+- Creating an alias
+- Compatibility mode
+
+## Diagnostic Commands
+
+### Check everything at once:
+```bash
+# Service status
+docker compose ps
+
+# Health checks
+docker ps
+
+# Logs (all services)
+docker compose logs --tail=50
+
+# Logs (specific service)
+docker compose logs --tail=50 tidb
+
+# Resource usage
+docker stats --no-stream
+
+# Network connectivity
+docker network inspect tidb-network
+```
+
+### Test connectivity:
+```bash
+# From host to TiDB
+mysql -h 127.0.0.1 -P 4000 -u root -e "SELECT 1"
+
+# From host (HTTP)
+curl http://127.0.0.1:10080/status
+
+# From container to TiDB
+docker run --rm --network tidb-network pingcap/dm:latest \
+  wget -q -O- http://tidb:10080/status
+```
+
+### Reset everything:
+```bash
+# Stop and remove everything (including data)
+docker compose down -v
+
+# Start fresh
+./start.sh
+```
+
+## Getting Help
+
+### Collect diagnostic information:
+```bash
+# Create a diagnostic report
+echo "=== Docker Version ===" > diagnostic.txt
+docker --version >> diagnostic.txt
+docker compose version >> diagnostic.txt
+
+echo -e "\n=== Container Status ===" >> diagnostic.txt
+docker ps -a >> diagnostic.txt
+
+echo -e "\n=== TiDB Logs ===" >> diagnostic.txt
+docker logs tidb --tail=50 >> diagnostic.txt 2>&1
+
+echo -e "\n=== DM Worker Logs ===" >> diagnostic.txt
+docker logs dm-worker --tail=50 >> diagnostic.txt 2>&1
+
+echo -e "\n=== DM Init Logs ===" >> diagnostic.txt
+docker logs dm-init >> diagnostic.txt 2>&1
+
+echo -e "\n=== Network Info ===" >> diagnostic.txt
+docker network inspect tidb-network >> diagnostic.txt
+
+echo "Report saved to diagnostic.txt"
+```
+
+### Useful resources:
+- [TiDB Documentation](https://docs.pingcap.com/tidb/stable)
+- [TiDB DM Documentation](https://docs.pingcap.com/tidb-data-migration/stable)
+- Project documentation:
+  - [README.md](README.md)
+  - [SYNC_GUIDE.md](SYNC_GUIDE.md)
+  - [DATAGRIP_SETUP.md](DATAGRIP_SETUP.md)
+  - [DOCKER_COMPOSE_V2.md](DOCKER_COMPOSE_V2.md)
+
+## Still Having Issues?
+
+If none of these solutions work:
+
+1. Check logs: `docker compose logs`
+2. Create diagnostic report (see above)
+3. Check if it's a known issue in TiDB/DM GitHub issues
+4. Verify your environment meets prerequisites (see [README.md](README.md))

check-docker.sh

@@ -42,4 +42,4 @@ if command -v orb &> /dev/null; then
 fi
 
 echo ""
-echo "Ready to start! Run: ./start.sh"
+echo "Ready to start! Run: ./start.sh"

configs/source.yaml

@@ -0,0 +1,12 @@
+source-id: "test-tidb"
+enable-gtid: false
+enable-relay: false
+from:
+  host: "${TEST_DB_HOST}"
+  port: ${TEST_DB_PORT}
+  user: "${TEST_DB_USER}"
+  password: "${TEST_DB_PASSWORD}"
+  security:
+    ssl-ca: ""
+    ssl-cert: ""
+    ssl-key: ""

configs/task.yaml

@@ -0,0 +1,52 @@
+# ==============================================================================
+# DM Task Configuration Template
+# ==============================================================================
+#
+# This file is a TEMPLATE and is NOT used directly by DM.
+#
+# The actual task configuration is dynamically generated by:
+#   scripts/init-dm.sh
+#
+# The script reads environment variables from .env and creates the real task.yaml
+# with your specific database name and table list.
+#
+# HOW TO RUN THE SYNC:
+# --------------------
+# 1. Configure .env file with your credentials
+# 2. Run: ./start.sh (auto-generates and starts sync)
+# 3. Check status: ./status.sh or ./sync-control.sh status
+#
+# For detailed guide, see: SYNC_GUIDE.md
+#
+# MANUAL CONTROL:
+# ---------------
+# - Start:   ./sync-control.sh start
+# - Stop:    ./sync-control.sh stop
+# - Pause:   ./sync-control.sh pause
+# - Resume:  ./sync-control.sh resume
+# - Restart: ./sync-control.sh restart
+# - Reinit:  ./sync-control.sh reinit
+#
+# ==============================================================================
+
+# Template structure (for reference):
+# name: "test-to-local"
+# task-mode: "all"  # full + incremental sync
+# target-database:
+#   host: "tidb"
+#   port: 4000
+#   user: "root"
+#   password: ""
+#
+# mysql-instances:
+#   - source-id: "test-tidb"
+#     block-allow-list: "sync-tables"
+#
+# block-allow-list:
+#   sync-tables:
+#     do-dbs: ["${DATABASE_NAME}"]
+#     do-tables:
+#       - db-name: "${DATABASE_NAME}"
+#         tbl-name: "table1"
+#       - db-name: "${DATABASE_NAME}"
+#         tbl-name: "table2"

docker-compose.yml

@@ -1,6 +1,15 @@
-# Minimal TiDB setup - Standalone mode with unistore
+# OrbStack optimized settings
+x-common-settings: &common
+  restart: unless-stopped
+  logging:
+    driver: "json-file"
+    options:
+      max-size: "10m"
+      max-file: "3"
+
 services:
   tidb:
+    <<: *common
     image: pingcap/tidb:latest
     container_name: tidb
     hostname: tidb.orb.local  # OrbStack DNS
@@ -30,11 +39,89 @@ services:
         reservations:
           cpus: '1'
           memory: 1G
-    restart: unless-stopped
+
+  dm-master:
+    <<: *common
+    image: pingcap/dm:latest
+    container_name: dm-master
+    hostname: dm-master.orb.local
+    command:
+      - /dm-master
+      - --master-addr=:8261
+      - --advertise-addr=dm-master:8261
+    ports:
+      - "8261:8261"
+    volumes:
+      - dm_master_data:/data
+      - ./configs:/configs:ro
+    healthcheck:
+      test: ["CMD", "wget", "-q", "-O-", "http://127.0.0.1:8261/status"]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+      start_period: 5s
+    deploy:
+      resources:
+        limits:
+          cpus: '0.5'
+          memory: 512M
+
+  dm-worker:
+    <<: *common
+    image: pingcap/dm:latest
+    container_name: dm-worker
+    hostname: dm-worker.orb.local
+    command:
+      - /dm-worker
+      - --worker-addr=:8262
+      - --advertise-addr=dm-worker:8262
+      - --join=dm-master:8261
+    volumes:
+      - dm_worker_data:/data
+      - ./configs:/configs:ro
+    depends_on:
+      tidb:
+        condition: service_healthy
+      dm-master:
+        condition: service_healthy
+    healthcheck:
+      test: ["CMD", "wget", "-q", "-O-", "http://127.0.0.1:8262/status"]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+      start_period: 5s
+    deploy:
+      resources:
+        limits:
+          cpus: '1'
+          memory: 1G
+
+  dm-init:
+    image: pingcap/dm:latest
+    container_name: dm-init
+    volumes:
+      - ./configs:/configs:ro
+      - ./scripts:/scripts:ro
+    environment:
+      - TEST_DB_HOST=${TEST_DB_HOST}
+      - TEST_DB_PORT=${TEST_DB_PORT}
+      - TEST_DB_USER=${TEST_DB_USER}
+      - TEST_DB_PASSWORD=${TEST_DB_PASSWORD}
+      - DATABASE_NAME=${DATABASE_NAME}
+      - TABLES=${TABLES}
+    depends_on:
+      dm-worker:
+        condition: service_healthy
+    command: ["/bin/sh", "/scripts/init-dm.sh"]
+    restart: "no"
 
 volumes:
   tidb_data:
     driver: local
+  dm_master_data:
+    driver: local
+  dm_worker_data:
+    driver: local
 
 networks:
   default:

scripts/init-dm.sh

@@ -0,0 +1,75 @@
+#!/bin/sh
+set -e
+
+echo "Waiting for DM master to be ready..."
+sleep 5
+
+# Check if it's TiDB Cloud (requires TLS)
+if echo "$TEST_DB_HOST" | grep -q "tidbcloud.com"; then
+  echo "Detected TiDB Cloud - downloading CA certificate for TLS..."
+  wget -q -O /tmp/isrgrootx1.pem https://letsencrypt.org/certs/isrgrootx1.pem
+  
+  # Generate source.yaml with TLS for TiDB Cloud
+  cat > /tmp/source.yaml <<EOF
+source-id: "test-tidb"
+enable-gtid: false
+enable-relay: false
+server-id: 101
+from:
+  host: "$TEST_DB_HOST"
+  port: $TEST_DB_PORT
+  user: "$TEST_DB_USER"
+  password: "$TEST_DB_PASSWORD"
+  security:
+    ssl-ca: "/tmp/isrgrootx1.pem"
+EOF
+else
+  # Generate source.yaml without TLS for regular TiDB
+  cat > /tmp/source.yaml <<EOF
+source-id: "test-tidb"
+enable-gtid: false
+enable-relay: false
+from:
+  host: "$TEST_DB_HOST"
+  port: $TEST_DB_PORT
+  user: "$TEST_DB_USER"
+  password: "$TEST_DB_PASSWORD"
+EOF
+fi
+
+echo "Creating DM source configuration..."
+/dmctl --master-addr=dm-master:8261 operate-source create /tmp/source.yaml || true
+
+# Generate task.yaml with multiple tables
+echo "name: \"test-to-local\"
+task-mode: \"all\"
+target-database:
+  host: \"tidb\"
+  port: 4000
+  user: \"root\"
+  password: \"\"
+
+mysql-instances:
+  - source-id: \"test-tidb\"
+    block-allow-list: \"sync-tables\"
+
+block-allow-list:
+  sync-tables:
+    do-dbs: [\"$DATABASE_NAME\"]
+    do-tables:" > /tmp/task.yaml
+
+# Add each table from TABLES variable
+IFS=',' read -ra TABLE_ARRAY <<< "$TABLES"
+for table in "${TABLE_ARRAY[@]}"; do
+  table=$(echo "$table" | xargs)  # trim whitespace
+  echo "      - db-name: \"$DATABASE_NAME\"
+        tbl-name: \"$table\"" >> /tmp/task.yaml
+done
+
+echo "Starting DM sync task..."
+/dmctl --master-addr=dm-master:8261 start-task /tmp/task.yaml || true
+
+echo "Checking task status..."
+/dmctl --master-addr=dm-master:8261 query-status test-to-local
+
+echo "DM initialization complete!"

start.sh

@@ -1,7 +1,26 @@
 #!/bin/bash
 set -e
 
-echo "🚀 Starting Minimal TiDB Environment..."
+echo "🚀 Starting TiDB Local Environment..."
+
+# Check if .env exists
+if [ ! -f .env ]; then
+    echo "⚠️  .env file not found!"
+    echo "📝 Creating .env from template..."
+    cp .env.example .env
+    echo "✏️  Please edit .env file with your test database credentials"
+    echo "   Then run this script again."
+    exit 1
+fi
+
+# Source .env to check if configured
+source .env
+
+if [ "$TEST_DB_HOST" = "your-test-tidb-host" ]; then
+    echo "⚠️  .env file needs configuration!"
+    echo "✏️  Please edit .env file with your test database credentials"
+    exit 1
+fi
 
 echo "🐳 Starting Docker containers..."
 docker compose up -d
@@ -16,9 +35,12 @@ echo ""
 echo "📊 Connection Info:"
 echo "   TiDB:       mysql -h 127.0.0.1 -P 4000 -u root"
 echo "   DataGrip:   Host: 127.0.0.1, Port: 4000, User: root, Password: (empty)"
+echo "   DM Master:  http://localhost:8261"
 echo ""
 echo "🔍 Useful commands:"
 echo "   Test connection:    ./test-connection.sh"
+echo "   Check sync status:  ./status.sh"
+echo "   Control sync:       ./sync-control.sh [start|stop|pause|resume|restart]"
 echo "   View logs:          docker compose logs -f"
 echo "   Stop environment:   docker compose down"
 echo ""

status.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+
+echo "🔍 Checking DM Sync Status..."
+echo ""
+
+# Check if containers are running
+if ! docker ps | grep -q dm-master; then
+    echo "❌ DM Master is not running. Start the environment first:"
+    echo "   ./start.sh"
+    exit 1
+fi
+
+echo "📡 Source Configuration:"
+docker exec dm-master /dmctl --master-addr=dm-master:8261 operate-source show
+
+echo ""
+echo "📊 Task Status:"
+docker exec dm-master /dmctl --master-addr=dm-master:8261 query-status test-to-local
+
+echo ""
+echo "💾 Local TiDB Databases:"
+docker exec tidb mysql -h 127.0.0.1 -P 4000 -u root -e "SHOW DATABASES;"

sync-control.sh

@@ -0,0 +1,74 @@
+#!/bin/bash
+
+TASK_NAME="test-to-local"
+DMCTL="docker exec dm-master /dmctl --master-addr=dm-master:8261"
+
+show_usage() {
+    echo "Usage: ./sync-control.sh [command]"
+    echo ""
+    echo "Commands:"
+    echo "  status    - Show sync task status"
+    echo "  start     - Start the sync task"
+    echo "  stop      - Stop the sync task"
+    echo "  pause     - Pause the sync task"
+    echo "  resume    - Resume the sync task"
+    echo "  restart   - Restart the sync task (stop + start)"
+    echo "  reinit    - Re-initialize DM configuration"
+    echo ""
+}
+
+check_dm() {
+    if ! docker ps | grep -q dm-master; then
+        echo "❌ DM Master is not running"
+        echo "   Start with: ./start.sh"
+        exit 1
+    fi
+}
+
+case "$1" in
+    status)
+        check_dm
+        echo "📊 Checking sync status for task: $TASK_NAME"
+        echo ""
+        $DMCTL query-status $TASK_NAME
+        ;;
+    start)
+        check_dm
+        echo "▶️  Starting sync task: $TASK_NAME"
+        $DMCTL start-task $TASK_NAME
+        ;;
+    stop)
+        check_dm
+        echo "⏹️  Stopping sync task: $TASK_NAME"
+        $DMCTL stop-task $TASK_NAME
+        ;;
+    pause)
+        check_dm
+        echo "⏸️  Pausing sync task: $TASK_NAME"
+        $DMCTL pause-task $TASK_NAME
+        ;;
+    resume)
+        check_dm
+        echo "▶️  Resuming sync task: $TASK_NAME"
+        $DMCTL resume-task $TASK_NAME
+        ;;
+    restart)
+        check_dm
+        echo "🔄 Restarting sync task: $TASK_NAME"
+        $DMCTL stop-task $TASK_NAME
+        sleep 2
+        $DMCTL start-task $TASK_NAME
+        ;;
+    reinit)
+        echo "🔄 Re-initializing DM configuration..."
+        docker compose up -d dm-init
+        echo ""
+        echo "⏳ Waiting for initialization..."
+        sleep 5
+        docker logs dm-init
+        ;;
+    *)
+        show_usage
+        exit 1
+        ;;
+esac

test-connection.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-echo "🔌 Testing TiDB Connection..."
+echo "🔌 Testing TiDB Connection for DataGrip..."
 echo ""
 
 # Check if TiDB container is running
@@ -22,7 +22,7 @@ if docker exec tidb mysql -h 127.0.0.1 -P 4000 -u root -e "SELECT VERSION();" 2>
     echo "📊 Available databases:"
     docker exec tidb mysql -h 127.0.0.1 -P 4000 -u root -e "SHOW DATABASES;" 2>/dev/null
     echo ""
-    echo "🎯 Connection Settings:"
+    echo "🎯 DataGrip Connection Settings:"
     echo "   Host:     127.0.0.1"
     echo "   Port:     4000"
     echo "   User:     root"
@@ -33,4 +33,4 @@ else
     echo ""
     echo "❌ Connection failed"
     echo "   Check TiDB logs: docker logs tidb"
-fi
+fi