tidbstandalone/TROUBLESHOOTING.md

Troubleshooting Guide

Common Issues and Solutions

1. TiDB Health Check Failing

Symptom

dependency failed to start: container tidb is unhealthy

Even though you can connect to TiDB from your host machine, Docker health check fails.

Root Cause

The original health check tried to use mysql command inside the TiDB container:

healthcheck:
  test: ["CMD", "mysql", "-h", "127.0.0.1", "-P", "4000", "-u", "root", "-e", "SELECT 1"]

The TiDB Docker image doesn't include the MySQL client binary, so this check always failed.

Solution ✅

Use TiDB's built-in HTTP status endpoint instead:

healthcheck:
  test: ["CMD", "wget", "-q", "-O-", "http://127.0.0.1:10080/status"]
  interval: 10s
  timeout: 5s
  retries: 3
  start_period: 10s

Why this works:

• TiDB exposes a status endpoint on port 10080
• wget is available in the container
• Returns HTTP 200 when TiDB is ready
• start_period gives TiDB time to initialize before health checks begin

2. Docker Compose Version Warning

Symptom

WARN[0000] version is obsolete, it will be ignored

Solution ✅

Remove the version field from docker-compose.yml. Modern Docker Compose (v2) doesn't need it.

Before:

version: '3.8'
services:
  ...

After:

services:
  ...

3. Service Dependencies Not Starting in Order

Symptom

Services fail because dependencies aren't ready yet.

Solution ✅

Use proper health checks and dependency conditions:

dm-worker:
  depends_on:
    tidb:
      condition: service_healthy
    dm-master:
      condition: service_healthy

Important:

• Each dependency must have a working health check
• start_period prevents false negatives during startup

4. dm-init Fails to Start

Symptom

Error: dm-init exits immediately

Check:

docker logs dm-init

Common Causes:

a) .env not configured:

# Check if .env exists and has real values
cat .env

Solution:

# Copy template and edit
cp .env.example .env
vim .env

b) Test database not reachable:

# Test from dm-init container
docker run --rm --network tidb-network pingcap/dm:latest \
  sh -c "wget -q -O- http://tidb:10080/status"

c) Script syntax error:

# Check init script
sh -n scripts/init-dm.sh

5. Containers Keep Restarting

Check Status:

docker ps -a
docker logs <container_name>

Common Issues:

a) Port already in use:

Error: bind: address already in use

Solution: Change ports in docker-compose.yml:

ports:
  - "14000:4000"  # Changed from 4000:4000

b) Out of memory:

Error: OOM killed

Solution: Increase memory limits or free up system resources.

c) Permission issues:

Error: permission denied

Solution: Check volume permissions or run:

docker compose down -v  # Remove volumes
docker compose up -d    # Recreate

6. Sync Task Not Running

Check Status:

./status.sh
# or
./sync-control.sh status

Common Issues:

a) Task not created:

# Check if source is configured
docker exec dm-master /dmctl --master-addr=dm-master:8261 operate-source show

Solution:

./sync-control.sh reinit

b) Wrong credentials: Check logs:

docker logs dm-worker

Fix .env and reinit:

vim .env
./sync-control.sh reinit

c) Table doesn't exist: Verify tables exist on source database:

# Connect to test DB and check
SHOW TABLES FROM your_database;

7. Connection Refused to TiDB

Symptom

ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (61)

Checks:

a) Is TiDB running?

docker ps | grep tidb

b) Is it healthy?

docker ps
# Look for "(healthy)" status

c) Is port exposed?

docker port tidb
# Should show: 4000/tcp -> 0.0.0.0:4000

d) Test from inside container:

docker exec tidb wget -q -O- http://127.0.0.1:10080/status

Solutions:

If container not running:

docker compose up -d tidb
docker logs tidb

If unhealthy:

# Wait for health check
sleep 15
docker ps

# If still unhealthy, check logs
docker logs tidb

If port not exposed:

# Recreate container
docker compose down
docker compose up -d

8. Data Not Syncing

Verify sync is running:

./sync-control.sh status

Check sync lag:

Look for "syncer" section in status output.

Common Issues:

a) Sync paused:

./sync-control.sh resume

b) Sync stopped with error:

# Check error in status output
./sync-control.sh status

# Fix the issue, then restart
./sync-control.sh restart

c) Network issues:

# Test connectivity from dm-worker to source
docker exec dm-worker ping -c 3 your-test-db-host

d) Binlog not enabled on source: Source database must have binlog enabled for incremental sync.

9. Slow Sync Performance

Check resource usage:

docker stats

Solutions:

a) Increase worker resources: Edit docker-compose.yml:

dm-worker:
  deploy:
    resources:
      limits:
        cpus: '2'
        memory: 2G

b) Optimize batch size: See TiDB DM Documentation for advanced tuning.

10. Docker Compose v1 vs v2 Issues

Symptom

docker: 'compose' is not a docker command

Solution

See DOCKER_COMPOSE_V2.md for:

• Upgrading to v2
• Creating an alias
• Compatibility mode

Diagnostic Commands

Check everything at once:

# Service status
docker compose ps

# Health checks
docker ps

# Logs (all services)
docker compose logs --tail=50

# Logs (specific service)
docker compose logs --tail=50 tidb

# Resource usage
docker stats --no-stream

# Network connectivity
docker network inspect tidb-network

Test connectivity:

# From host to TiDB
mysql -h 127.0.0.1 -P 4000 -u root -e "SELECT 1"

# From host (HTTP)
curl http://127.0.0.1:10080/status

# From container to TiDB
docker run --rm --network tidb-network pingcap/dm:latest \
  wget -q -O- http://tidb:10080/status

Reset everything:

# Stop and remove everything (including data)
docker compose down -v

# Start fresh
./start.sh

Getting Help

Collect diagnostic information:

# Create a diagnostic report
echo "=== Docker Version ===" > diagnostic.txt
docker --version >> diagnostic.txt
docker compose version >> diagnostic.txt

echo -e "\n=== Container Status ===" >> diagnostic.txt
docker ps -a >> diagnostic.txt

echo -e "\n=== TiDB Logs ===" >> diagnostic.txt
docker logs tidb --tail=50 >> diagnostic.txt 2>&1

echo -e "\n=== DM Worker Logs ===" >> diagnostic.txt
docker logs dm-worker --tail=50 >> diagnostic.txt 2>&1

echo -e "\n=== DM Init Logs ===" >> diagnostic.txt
docker logs dm-init >> diagnostic.txt 2>&1

echo -e "\n=== Network Info ===" >> diagnostic.txt
docker network inspect tidb-network >> diagnostic.txt

echo "Report saved to diagnostic.txt"

Useful resources:

• TiDB Documentation
• TiDB DM Documentation
• Project documentation:
  • README.md
  • SYNC_GUIDE.md
  • DATAGRIP_SETUP.md
  • DOCKER_COMPOSE_V2.md

Still Having Issues?

If none of these solutions work:

1. Check logs: docker compose logs
2. Create diagnostic report (see above)
3. Check if it's a known issue in TiDB/DM GitHub issues
4. Verify your environment meets prerequisites (see README.md)