tidbstandalone/README.md

TiDB Local Development Environment

A minimal TiDB instance with Data Migration (DM) for syncing data from test environments.

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Your macOS (OrbStack)                    │
│                                                             │
│  ┌──────────────┐      ┌──────────────┐                   │
│  │   DataGrip   │──────▶│     TiDB     │                   │
│  │  (port 4000) │      │  (Standalone) │                   │
│  └──────────────┘      └───────▲───────┘                   │
│                                │                             │
│                        ┌───────┴────────┐                   │
│                        │   DM Worker    │                   │
│                        │  (Sync Engine) │                   │
│                        └───────▲────────┘                   │
│                                │                             │
│                        ┌───────┴────────┐                   │
│                        │   DM Master    │                   │
│                        │ (Orchestrator) │                   │
│                        └───────▲────────┘                   │
│                                │                             │
└────────────────────────────────┼─────────────────────────────┘
                                 │
                         (Continuous Sync)
                                 │
                    ┌────────────▼─────────────┐
                    │   Test TiDB Instance     │
                    │  (Remote Environment)    │
                    └──────────────────────────┘

Components:

• TiDB (Standalone Mode): Runs with embedded storage (unistore), no separate PD/TiKV needed
• DM Master: Manages data migration tasks
• DM Worker: Executes the actual data synchronization from test to local
• DataGrip/MySQL Clients: Connect directly to TiDB on port 4000

Quick Reference

Connection Info

Service	Host	Port	User	Password
TiDB	127.0.0.1	4000	root	(empty)
DM Master	127.0.0.1	8261	-	-

Useful Commands

# Start environment (TiDB only, no working sync)
./start.sh

# Test connection
./test-connection.sh

# NEW: Sync data from TiDB Cloud to local
./sync-data.sh

# Check sync status (deprecated - DM doesn't work with TiDB Cloud)
# ./status.sh
# or use the sync control script:
# ./sync-control.sh status

# Connect with MySQL client
mysql -h 127.0.0.1 -P 4000 -u root

# View logs
docker compose logs -f

# Stop environment
docker compose down

For DataGrip/DBeaver setup, see DATAGRIP_SETUP.md

Prerequisites

• macOS with OrbStack (or Docker Desktop)
• Docker Compose v2 (command: docker compose, not docker-compose)
• Access to test TiDB instance

Check your setup:

./check-docker.sh

Note: If you have Docker Compose v1 only, see DOCKER_COMPOSE_V2.md for migration options.

Configuration

1. Copy and edit .env file:

cp .env.example .env
# Edit .env with your test database credentials

Required variables:

• TEST_DB_HOST: Your test TiDB host
• TEST_DB_PORT: Test TiDB port (default: 4000)
• TEST_DB_USER: Test database username
• TEST_DB_PASSWORD: Test database password
• DATABASE_NAME: Database to sync
• TABLES: Comma-separated list of tables (e.g., "table1,table2,table3")

Usage

How the Sync Works

Important Note: The original DM-based sync approach doesn't work with TiDB Cloud Serverless because TiDB Cloud doesn't support the MySQL replication features that DM requires.

Officially Recommended Approaches

See TIDB_CLOUD_MIGRATION.md for officially supported migration methods:

1. Console Export + SQL Import (simplest for development)
2. Dumpling + TiDB Lightning (for larger datasets)
3. Periodic Sync Scripts (created in this project)
4. Application-Level Sync (for real-time needs)

For detailed sync operations, see SYNC_GUIDE.md

Start the environment

docker compose up -d

This will:

1. Start TiDB in standalone mode
2. Start DM master and worker
3. Automatically configure the data source and sync task
4. Begin syncing data from test to local

Check sync status

docker exec dm-master /dmctl --master-addr=dm-master:8261 query-status test-to-local

Connect to local TiDB

Command Line:

mysql -h 127.0.0.1 -P 4000 -u root

DataGrip / MySQL Workbench / DBeaver:

• Host: 127.0.0.1
• Port: 4000
• User: root
• Password: (leave empty)

See DATAGRIP_SETUP.md for detailed client setup instructions.

View logs

# All services
docker compose logs -f

# Specific service
docker compose logs -f tidb
docker compose logs -f dm-worker

Stop the environment

docker compose down

Reset everything (including data)

docker compose down -v

Manual DM Operations

Check source configuration

docker exec dm-master /dmctl --master-addr=dm-master:8261 operate-source show

Stop sync task

docker exec dm-master /dmctl --master-addr=dm-master:8261 stop-task test-to-local

Start sync task

docker exec dm-master /dmctl --master-addr=dm-master:8261 start-task /configs/task.yaml

Pause sync task

docker exec dm-master /dmctl --master-addr=dm-master:8261 pause-task test-to-local

Resume sync task

docker exec dm-master /dmctl --master-addr=dm-master:8261 resume-task test-to-local

Troubleshooting

Common Issues

For detailed troubleshooting, see TROUBLESHOOTING.md

Quick Checks

TiDB health check failing

# Check if TiDB is healthy
docker ps | grep tidb

# Should show: (healthy)
# If not, check logs:
docker logs tidb

DM task fails to start

• Check if test database is accessible from container
• Verify credentials in .env
• Check logs: docker-compose logs dm-worker

Tables not syncing

• Ensure tables exist in source database
• Verify table names in TABLES variable
• Check task status for specific errors

TiDB connection issues

• Verify TiDB is running: docker ps | grep tidb
• Check health: docker exec tidb mysql -h 127.0.0.1 -P 4000 -u root -e "SELECT 1"

Re-initialize DM configuration

docker compose up -d dm-init

Resource Usage

Default resource limits (suitable for local development):

• TiDB: 2 CPU, 2GB RAM
• DM Worker: 1 CPU, 1GB RAM
• DM Master: 0.5 CPU, 512MB RAM

Adjust in docker-compose.yml if needed.

Notes

• Docker Compose v2: This project uses docker compose (v2 syntax). If you have v1, either upgrade or create an alias: alias docker-compose='docker compose'
• Standalone Mode: TiDB runs without distributed storage, suitable for development only
• Data Persistence: Data is stored in Docker volumes, persists across restarts
• Sync Mode: Configured for full + incremental sync ("all" mode)
• OrbStack DNS: Uses .orb.local hostnames for container networking