tidbstandalone/README.md

# TiDB Local Development Environment

A minimal TiDB instance with Data Migration (DM) for syncing data from test environments.

## Architecture

```
┌─────────────────────────────────────────────────────────────┐
│                    Your macOS (OrbStack)                    │
│                                                             │
│  ┌──────────────┐      ┌──────────────┐                   │
│  │   DataGrip   │──────▶│     TiDB     │                   │
│  │  (port 4000) │      │  (Standalone) │                   │
│  └──────────────┘      └───────▲───────┘                   │
│                                │                             │
│                        ┌───────┴────────┐                   │
│                        │   DM Worker    │                   │
│                        │  (Sync Engine) │                   │
│                        └───────▲────────┘                   │
│                                │                             │
│                        ┌───────┴────────┐                   │
│                        │   DM Master    │                   │
│                        │ (Orchestrator) │                   │
│                        └───────▲────────┘                   │
│                                │                             │
└────────────────────────────────┼─────────────────────────────┘
                                 │
                         (Continuous Sync)
                                 │
                    ┌────────────▼─────────────┐
                    │   Test TiDB Instance     │
                    │  (Remote Environment)    │
                    └──────────────────────────┘
```

**Components:**
- **TiDB (Standalone Mode)**: Runs with embedded storage (unistore), no separate PD/TiKV needed
- **DM Master**: Manages data migration tasks
- **DM Worker**: Executes the actual data synchronization from test to local
- **DataGrip/MySQL Clients**: Connect directly to TiDB on port 4000

## Quick Reference

### Connection Info
| Service | Host | Port | User | Password |
|---------|------|------|------|----------|
| TiDB | `127.0.0.1` | `4000` | `root` | _(empty)_ |
| DM Master | `127.0.0.1` | `8261` | - | - |

### Useful Commands
```bash
# Start environment (TiDB only, no working sync)
./start.sh

# Test connection
./test-connection.sh

# NEW: Sync data from TiDB Cloud to local
./sync-data.sh

# Check sync status (deprecated - DM doesn't work with TiDB Cloud)
# ./status.sh
# or use the sync control script:
# ./sync-control.sh status

# Connect with MySQL client
mysql -h 127.0.0.1 -P 4000 -u root

# View logs
docker compose logs -f

# Stop environment
docker compose down
```

For DataGrip/DBeaver setup, see [DATAGRIP_SETUP.md](DATAGRIP_SETUP.md)

## Prerequisites

- macOS with OrbStack (or Docker Desktop)
- Docker Compose v2 (command: `docker compose`, not `docker-compose`)
- Access to test TiDB instance

**Check your setup:**
```bash
./check-docker.sh
```

**Note:** If you have Docker Compose v1 only, see [DOCKER_COMPOSE_V2.md](DOCKER_COMPOSE_V2.md) for migration options.

## Configuration

1. **Copy and edit `.env` file**:
```bash
cp .env.example .env
# Edit .env with your test database credentials
```

Required variables:
- `TEST_DB_HOST`: Your test TiDB host
- `TEST_DB_PORT`: Test TiDB port (default: 4000)
- `TEST_DB_USER`: Test database username
- `TEST_DB_PASSWORD`: Test database password
- `DATABASE_NAME`: Database to sync
- `TABLES`: Comma-separated list of tables (e.g., "table1,table2,table3")

## Usage

### How the Sync Works

**Important Note**: The original DM-based sync approach doesn't work with TiDB Cloud Serverless because TiDB Cloud doesn't support the MySQL replication features that DM requires.

### Officially Recommended Approaches

See [TIDB_CLOUD_MIGRATION.md](TIDB_CLOUD_MIGRATION.md) for officially supported migration methods:

1. **Console Export + SQL Import** (simplest for development)
2. **Dumpling + TiDB Lightning** (for larger datasets)
3. **Periodic Sync Scripts** (created in this project)
4. **Application-Level Sync** (for real-time needs)

**For detailed sync operations, see [SYNC_GUIDE.md](SYNC_GUIDE.md)**

### Start the environment
```bash
docker compose up -d
```

This will:
1. Start TiDB in standalone mode
2. Start DM master and worker
3. Automatically configure the data source and sync task
4. Begin syncing data from test to local

### Check sync status
```bash
docker exec dm-master /dmctl --master-addr=dm-master:8261 query-status test-to-local
```

### Connect to local TiDB

**Command Line:**
```bash
mysql -h 127.0.0.1 -P 4000 -u root
```

**DataGrip / MySQL Workbench / DBeaver:**
- Host: `127.0.0.1`
- Port: `4000`
- User: `root`
- Password: _(leave empty)_

See [DATAGRIP_SETUP.md](DATAGRIP_SETUP.md) for detailed client setup instructions.

### View logs
```bash
# All services
docker compose logs -f

# Specific service
docker compose logs -f tidb
docker compose logs -f dm-worker
```

### Stop the environment
```bash
docker compose down
```

### Reset everything (including data)
```bash
docker compose down -v
```

## Manual DM Operations

### Check source configuration
```bash
docker exec dm-master /dmctl --master-addr=dm-master:8261 operate-source show
```

### Stop sync task
```bash
docker exec dm-master /dmctl --master-addr=dm-master:8261 stop-task test-to-local
```

### Start sync task
```bash
docker exec dm-master /dmctl --master-addr=dm-master:8261 start-task /configs/task.yaml
```

### Pause sync task
```bash
docker exec dm-master /dmctl --master-addr=dm-master:8261 pause-task test-to-local
```

### Resume sync task
```bash
docker exec dm-master /dmctl --master-addr=dm-master:8261 resume-task test-to-local
```

## Troubleshooting

### Common Issues

For detailed troubleshooting, see [TROUBLESHOOTING.md](TROUBLESHOOTING.md)

### Quick Checks

#### TiDB health check failing
```bash
# Check if TiDB is healthy
docker ps | grep tidb

# Should show: (healthy)
# If not, check logs:
docker logs tidb
```

#### DM task fails to start
- Check if test database is accessible from container
- Verify credentials in `.env`
- Check logs: `docker-compose logs dm-worker`

### Tables not syncing
- Ensure tables exist in source database
- Verify table names in `TABLES` variable
- Check task status for specific errors

### TiDB connection issues
- Verify TiDB is running: `docker ps | grep tidb`
- Check health: `docker exec tidb mysql -h 127.0.0.1 -P 4000 -u root -e "SELECT 1"`

### Re-initialize DM configuration
```bash
docker compose up -d dm-init
```

## Resource Usage

Default resource limits (suitable for local development):
- TiDB: 2 CPU, 2GB RAM
- DM Worker: 1 CPU, 1GB RAM
- DM Master: 0.5 CPU, 512MB RAM

Adjust in `docker-compose.yml` if needed.

## Notes

- **Docker Compose v2**: This project uses `docker compose` (v2 syntax). If you have v1, either upgrade or create an alias: `alias docker-compose='docker compose'`
- **Standalone Mode**: TiDB runs without distributed storage, suitable for development only
- **Data Persistence**: Data is stored in Docker volumes, persists across restarts
- **Sync Mode**: Configured for full + incremental sync ("all" mode)
- **OrbStack DNS**: Uses `.orb.local` hostnames for container networking