diff --git a/DOCKER_MIGRATION_GUIDE.md b/DOCKER_MIGRATION_GUIDE.md new file mode 100644 index 0000000..9d7d2de --- /dev/null +++ b/DOCKER_MIGRATION_GUIDE.md @@ -0,0 +1,399 @@ +# Docker Image Migration Guide + +This guide explains how to export your built Docker images, transfer them to another environment, and run them without rebuilding. + +## Overview + +The migration process involves: +1. **Export**: Save built images to tar files +2. **Transfer**: Copy tar files to target environment +3. **Import**: Load images on target environment +4. **Run**: Start services with imported images + +## Prerequisites + +### Source Environment (where images are built) +- Docker installed and running +- All services built and working +- Sufficient disk space for image export + +### Target Environment (where images will run) +- Docker installed and running +- Sufficient disk space for image import +- Network access to source environment (or USB drive) + +## Step 1: Export Docker Images + +### 1.1 List Current Images + +First, check what images you have: + +```bash +docker images --format "table {{.Repository}}\t{{.Tag}}\t{{.ID}}\t{{.Size}}" +``` + +You should see images like: +- `legal-doc-masker-backend-api` +- `legal-doc-masker-frontend` +- `legal-doc-masker-mineru-api` +- `redis:alpine` + +### 1.2 Export Individual Images + +Create a directory for exports: + +```bash +mkdir -p docker-images-export +cd docker-images-export +``` + +Export each image: + +```bash +# Export backend image +docker save legal-doc-masker-backend-api:latest -o backend-api.tar + +# Export frontend image +docker save legal-doc-masker-frontend:latest -o frontend.tar + +# Export mineru image +docker save legal-doc-masker-mineru-api:latest -o mineru-api.tar + +# Export redis image (if not using official) +docker save redis:alpine -o redis.tar +``` + +### 1.3 Export All Images at Once (Alternative) + +If you want to export all images in one command: + +```bash +# Export all project images +docker save \ + legal-doc-masker-backend-api:latest \ + legal-doc-masker-frontend:latest \ + legal-doc-masker-mineru-api:latest \ + redis:alpine \ + -o legal-doc-masker-all.tar +``` + +### 1.4 Verify Export Files + +Check the exported files: + +```bash +ls -lh *.tar +``` + +You should see files like: +- `backend-api.tar` (~200-500MB) +- `frontend.tar` (~100-300MB) +- `mineru-api.tar` (~1-3GB) +- `redis.tar` (~30-50MB) + +## Step 2: Transfer Images + +### 2.1 Transfer via Network (SCP/RSYNC) + +```bash +# Transfer to remote server +scp *.tar user@remote-server:/path/to/destination/ + +# Or using rsync (more efficient for large files) +rsync -avz --progress *.tar user@remote-server:/path/to/destination/ +``` + +### 2.2 Transfer via USB Drive + +```bash +# Copy to USB drive +cp *.tar /Volumes/USB_DRIVE/docker-images/ + +# Or create a compressed archive +tar -czf legal-doc-masker-images.tar.gz *.tar +cp legal-doc-masker-images.tar.gz /Volumes/USB_DRIVE/ +``` + +### 2.3 Transfer via Cloud Storage + +```bash +# Upload to cloud storage (example with AWS S3) +aws s3 cp *.tar s3://your-bucket/docker-images/ + +# Or using Google Cloud Storage +gsutil cp *.tar gs://your-bucket/docker-images/ +``` + +## Step 3: Import Images on Target Environment + +### 3.1 Prepare Target Environment + +```bash +# Create directory for images +mkdir -p docker-images-import +cd docker-images-import + +# Copy images from transfer method +# (SCP, USB, or download from cloud storage) +``` + +### 3.2 Import Individual Images + +```bash +# Import backend image +docker load -i backend-api.tar + +# Import frontend image +docker load -i frontend.tar + +# Import mineru image +docker load -i mineru-api.tar + +# Import redis image +docker load -i redis.tar +``` + +### 3.3 Import All Images at Once (if exported together) + +```bash +docker load -i legal-doc-masker-all.tar +``` + +### 3.4 Verify Imported Images + +```bash +docker images --format "table {{.Repository}}\t{{.Tag}}\t{{.ID}}\t{{.Size}}" +``` + +## Step 4: Prepare Target Environment + +### 4.1 Copy Project Files + +Transfer the following files to target environment: + +```bash +# Essential files to copy +docker-compose.yml +DOCKER_COMPOSE_README.md +setup-unified-docker.sh + +# Environment files (if they exist) +backend/.env +frontend/.env +mineru/.env + +# Storage directories (if you want to preserve data) +backend/storage/ +mineru/storage/ +backend/legal_doc_masker.db +``` + +### 4.2 Create Directory Structure + +```bash +# Create necessary directories +mkdir -p backend/storage +mkdir -p mineru/storage/uploads +mkdir -p mineru/storage/processed +``` + +## Step 5: Run Services + +### 5.1 Start All Services + +```bash +# Start all services using imported images +docker-compose up -d +``` + +### 5.2 Verify Services + +```bash +# Check service status +docker-compose ps + +# Check service logs +docker-compose logs -f +``` + +### 5.3 Test Endpoints + +```bash +# Test frontend +curl -I http://localhost:3000 + +# Test backend API +curl -I http://localhost:8000/api/v1 + +# Test mineru API +curl -I http://localhost:8001/health +``` + +## Automation Scripts + +### Export Script + +Create `export-images.sh`: + +```bash +#!/bin/bash + +set -e + +echo "🚀 Exporting Docker Images" + +# Create export directory +mkdir -p docker-images-export +cd docker-images-export + +# Export images +echo "📦 Exporting backend-api image..." +docker save legal-doc-masker-backend-api:latest -o backend-api.tar + +echo "📦 Exporting frontend image..." +docker save legal-doc-masker-frontend:latest -o frontend.tar + +echo "📦 Exporting mineru-api image..." +docker save legal-doc-masker-mineru-api:latest -o mineru-api.tar + +echo "📦 Exporting redis image..." +docker save redis:alpine -o redis.tar + +# Show file sizes +echo "📊 Export complete. File sizes:" +ls -lh *.tar + +echo "✅ Images exported successfully!" +``` + +### Import Script + +Create `import-images.sh`: + +```bash +#!/bin/bash + +set -e + +echo "🚀 Importing Docker Images" + +# Check if tar files exist +if [ ! -f "backend-api.tar" ]; then + echo "❌ backend-api.tar not found" + exit 1 +fi + +# Import images +echo "📦 Importing backend-api image..." +docker load -i backend-api.tar + +echo "📦 Importing frontend image..." +docker load -i frontend.tar + +echo "📦 Importing mineru-api image..." +docker load -i mineru-api.tar + +echo "📦 Importing redis image..." +docker load -i redis.tar + +# Verify imports +echo "📊 Imported images:" +docker images --format "table {{.Repository}}\t{{.Tag}}\t{{.Size}}" | grep legal-doc-masker + +echo "✅ Images imported successfully!" +``` + +## Troubleshooting + +### Common Issues + +1. **Image not found during import** + ```bash + # Check if image exists + docker images | grep image-name + + # Re-export if needed + docker save image-name:tag -o image-name.tar + ``` + +2. **Port conflicts on target environment** + ```bash + # Check what's using the ports + lsof -i :8000 + lsof -i :8001 + lsof -i :3000 + + # Modify docker-compose.yml if needed + ports: + - "8002:8000" # Change external port + ``` + +3. **Permission issues** + ```bash + # Fix file permissions + chmod +x setup-unified-docker.sh + chmod +x export-images.sh + chmod +x import-images.sh + ``` + +4. **Storage directory issues** + ```bash + # Create directories with proper permissions + sudo mkdir -p backend/storage + sudo mkdir -p mineru/storage/uploads + sudo mkdir -p mineru/storage/processed + sudo chown -R $USER:$USER backend/storage mineru/storage + ``` + +### Performance Optimization + +1. **Compress images for transfer** + ```bash + # Compress before transfer + gzip *.tar + + # Decompress on target + gunzip *.tar.gz + ``` + +2. **Use parallel transfer** + ```bash + # Transfer multiple files in parallel + parallel scp {} user@server:/path/ ::: *.tar + ``` + +3. **Use Docker registry (alternative)** + ```bash + # Push to registry + docker tag legal-doc-masker-backend-api:latest your-registry/backend-api:latest + docker push your-registry/backend-api:latest + + # Pull on target + docker pull your-registry/backend-api:latest + ``` + +## Complete Migration Checklist + +- [ ] Export all Docker images +- [ ] Transfer image files to target environment +- [ ] Transfer project configuration files +- [ ] Import images on target environment +- [ ] Create necessary directories +- [ ] Start services +- [ ] Verify all services are running +- [ ] Test all endpoints +- [ ] Update any environment-specific configurations + +## Security Considerations + +1. **Secure transfer**: Use encrypted transfer methods (SCP, SFTP) +2. **Image verification**: Verify image integrity after transfer +3. **Environment isolation**: Ensure target environment is properly secured +4. **Access control**: Limit access to Docker daemon on target environment + +## Cost Optimization + +1. **Image size**: Remove unnecessary layers before export +2. **Compression**: Use compression for large images +3. **Selective transfer**: Only transfer images you need +4. **Cleanup**: Remove old images after successful migration \ No newline at end of file diff --git a/MIGRATION_QUICK_REFERENCE.md b/MIGRATION_QUICK_REFERENCE.md new file mode 100644 index 0000000..1257053 --- /dev/null +++ b/MIGRATION_QUICK_REFERENCE.md @@ -0,0 +1,159 @@ +# Docker Migration Quick Reference + +## 🚀 Quick Migration Process + +### Source Environment (Export) + +```bash +# 1. Build images first (if not already built) +docker-compose build + +# 2. Export all images +./export-images.sh + +# 3. Transfer files to target environment +# Option A: SCP +scp -r docker-images-export-*/ user@target-server:/path/to/destination/ + +# Option B: USB Drive +cp -r docker-images-export-*/ /Volumes/USB_DRIVE/ + +# Option C: Compressed archive +scp legal-doc-masker-images-*.tar.gz user@target-server:/path/to/destination/ +``` + +### Target Environment (Import) + +```bash +# 1. Copy project files +scp docker-compose.yml user@target-server:/path/to/destination/ +scp DOCKER_COMPOSE_README.md user@target-server:/path/to/destination/ + +# 2. Import images +./import-images.sh + +# 3. Start services +docker-compose up -d + +# 4. Verify +docker-compose ps +``` + +## 📋 Essential Files to Transfer + +### Required Files +- `docker-compose.yml` - Unified compose configuration +- `DOCKER_COMPOSE_README.md` - Documentation +- `backend/.env` - Backend environment variables +- `frontend/.env` - Frontend environment variables +- `mineru/.env` - Mineru environment variables (if exists) + +### Optional Files (for data preservation) +- `backend/storage/` - Backend storage directory +- `mineru/storage/` - Mineru storage directory +- `backend/legal_doc_masker.db` - Database file + +## 🔧 Common Commands + +### Export Commands +```bash +# Manual export +docker save legal-doc-masker-backend-api:latest -o backend-api.tar +docker save legal-doc-masker-frontend:latest -o frontend.tar +docker save legal-doc-masker-mineru-api:latest -o mineru-api.tar +docker save redis:alpine -o redis.tar + +# Compress for transfer +tar -czf legal-doc-masker-images.tar.gz *.tar +``` + +### Import Commands +```bash +# Manual import +docker load -i backend-api.tar +docker load -i frontend.tar +docker load -i mineru-api.tar +docker load -i redis.tar + +# Extract compressed archive +tar -xzf legal-doc-masker-images.tar.gz +``` + +### Service Management +```bash +# Start all services +docker-compose up -d + +# Stop all services +docker-compose down + +# View logs +docker-compose logs -f [service-name] + +# Check status +docker-compose ps +``` + +## 🌐 Service URLs + +After successful migration: +- **Frontend**: http://localhost:3000 +- **Backend API**: http://localhost:8000 +- **Mineru API**: http://localhost:8001 + +## ⚠️ Troubleshooting + +### Port Conflicts +```bash +# Check what's using ports +lsof -i :8000 +lsof -i :8001 +lsof -i :3000 + +# Modify docker-compose.yml if needed +ports: + - "8002:8000" # Change external port +``` + +### Permission Issues +```bash +# Fix script permissions +chmod +x export-images.sh +chmod +x import-images.sh +chmod +x setup-unified-docker.sh + +# Fix directory permissions +sudo chown -R $USER:$USER backend/storage mineru/storage +``` + +### Disk Space Issues +```bash +# Check available space +df -h + +# Clean up Docker +docker system prune -a +``` + +## 📊 Expected File Sizes + +- `backend-api.tar`: ~200-500MB +- `frontend.tar`: ~100-300MB +- `mineru-api.tar`: ~1-3GB +- `redis.tar`: ~30-50MB +- `legal-doc-masker-images.tar.gz`: ~1-2GB (compressed) + +## 🔒 Security Notes + +1. Use encrypted transfer (SCP, SFTP) for sensitive environments +2. Verify image integrity after transfer +3. Update environment variables for target environment +4. Ensure proper network security on target environment + +## 📞 Support + +If you encounter issues: +1. Check the full `DOCKER_MIGRATION_GUIDE.md` +2. Verify all required files are present +3. Check Docker logs: `docker-compose logs -f` +4. Ensure sufficient disk space and permissions \ No newline at end of file diff --git a/export-images.sh b/export-images.sh new file mode 100644 index 0000000..221cb66 --- /dev/null +++ b/export-images.sh @@ -0,0 +1,168 @@ +#!/bin/bash + +# Docker Image Export Script +# Exports all project Docker images for migration to another environment + +set -e + +echo "🚀 Legal Document Masker - Docker Image Export" +echo "==============================================" + +# Function to check if Docker is running +check_docker() { + if ! docker info > /dev/null 2>&1; then + echo "❌ Docker is not running. Please start Docker and try again." + exit 1 + fi + echo "✅ Docker is running" +} + +# Function to check if images exist +check_images() { + echo "🔍 Checking for required images..." + + local missing_images=() + + if ! docker images | grep -q "legal-doc-masker-backend-api"; then + missing_images+=("legal-doc-masker-backend-api") + fi + + if ! docker images | grep -q "legal-doc-masker-frontend"; then + missing_images+=("legal-doc-masker-frontend") + fi + + if ! docker images | grep -q "legal-doc-masker-mineru-api"; then + missing_images+=("legal-doc-masker-mineru-api") + fi + + if ! docker images | grep -q "redis:alpine"; then + missing_images+=("redis:alpine") + fi + + if [ ${#missing_images[@]} -ne 0 ]; then + echo "❌ Missing images: ${missing_images[*]}" + echo "Please build the images first using: docker-compose build" + exit 1 + fi + + echo "✅ All required images found" +} + +# Function to create export directory +create_export_dir() { + local export_dir="docker-images-export-$(date +%Y%m%d-%H%M%S)" + mkdir -p "$export_dir" + cd "$export_dir" + echo "📁 Created export directory: $export_dir" + echo "$export_dir" +} + +# Function to export images +export_images() { + local export_dir="$1" + + echo "📦 Exporting Docker images..." + + # Export backend image + echo " 📦 Exporting backend-api image..." + docker save legal-doc-masker-backend-api:latest -o backend-api.tar + + # Export frontend image + echo " 📦 Exporting frontend image..." + docker save legal-doc-masker-frontend:latest -o frontend.tar + + # Export mineru image + echo " 📦 Exporting mineru-api image..." + docker save legal-doc-masker-mineru-api:latest -o mineru-api.tar + + # Export redis image + echo " 📦 Exporting redis image..." + docker save redis:alpine -o redis.tar + + echo "✅ All images exported successfully!" +} + +# Function to show export summary +show_summary() { + echo "" + echo "📊 Export Summary:" + echo "==================" + ls -lh *.tar + + echo "" + echo "📋 Files to transfer:" + echo "====================" + for file in *.tar; do + echo " - $file" + done + + echo "" + echo "💾 Total size: $(du -sh . | cut -f1)" +} + +# Function to create compressed archive +create_archive() { + echo "" + echo "🗜️ Creating compressed archive..." + + local archive_name="legal-doc-masker-images-$(date +%Y%m%d-%H%M%S).tar.gz" + tar -czf "$archive_name" *.tar + + echo "✅ Created archive: $archive_name" + echo "📊 Archive size: $(du -sh "$archive_name" | cut -f1)" + + echo "" + echo "📋 Transfer options:" + echo "===================" + echo "1. Transfer individual .tar files" + echo "2. Transfer compressed archive: $archive_name" +} + +# Function to show transfer instructions +show_transfer_instructions() { + echo "" + echo "📤 Transfer Instructions:" + echo "========================" + echo "" + echo "Option 1: Transfer individual files" + echo "-----------------------------------" + echo "scp *.tar user@target-server:/path/to/destination/" + echo "" + echo "Option 2: Transfer compressed archive" + echo "-------------------------------------" + echo "scp legal-doc-masker-images-*.tar.gz user@target-server:/path/to/destination/" + echo "" + echo "Option 3: USB Drive" + echo "-------------------" + echo "cp *.tar /Volumes/USB_DRIVE/docker-images/" + echo "cp legal-doc-masker-images-*.tar.gz /Volumes/USB_DRIVE/" + echo "" + echo "Option 4: Cloud Storage" + echo "----------------------" + echo "aws s3 cp *.tar s3://your-bucket/docker-images/" + echo "aws s3 cp legal-doc-masker-images-*.tar.gz s3://your-bucket/docker-images/" +} + +# Main execution +main() { + check_docker + check_images + + local export_dir=$(create_export_dir) + export_images "$export_dir" + show_summary + create_archive + show_transfer_instructions + + echo "" + echo "🎉 Export completed successfully!" + echo "📁 Export location: $(pwd)" + echo "" + echo "Next steps:" + echo "1. Transfer the files to your target environment" + echo "2. Use import-images.sh on the target environment" + echo "3. Copy docker-compose.yml and other config files" +} + +# Run main function +main "$@" \ No newline at end of file diff --git a/import-images.sh b/import-images.sh new file mode 100644 index 0000000..d3f445d --- /dev/null +++ b/import-images.sh @@ -0,0 +1,232 @@ +#!/bin/bash + +# Docker Image Import Script +# Imports Docker images on target environment for migration + +set -e + +echo "🚀 Legal Document Masker - Docker Image Import" +echo "==============================================" + +# Function to check if Docker is running +check_docker() { + if ! docker info > /dev/null 2>&1; then + echo "❌ Docker is not running. Please start Docker and try again." + exit 1 + fi + echo "✅ Docker is running" +} + +# Function to check for tar files +check_tar_files() { + echo "🔍 Checking for Docker image files..." + + local missing_files=() + + if [ ! -f "backend-api.tar" ]; then + missing_files+=("backend-api.tar") + fi + + if [ ! -f "frontend.tar" ]; then + missing_files+=("frontend.tar") + fi + + if [ ! -f "mineru-api.tar" ]; then + missing_files+=("mineru-api.tar") + fi + + if [ ! -f "redis.tar" ]; then + missing_files+=("redis.tar") + fi + + if [ ${#missing_files[@]} -ne 0 ]; then + echo "❌ Missing files: ${missing_files[*]}" + echo "" + echo "Please ensure all .tar files are in the current directory." + echo "If you have a compressed archive, extract it first:" + echo " tar -xzf legal-doc-masker-images-*.tar.gz" + exit 1 + fi + + echo "✅ All required files found" +} + +# Function to check available disk space +check_disk_space() { + echo "💾 Checking available disk space..." + + local required_space=0 + for file in *.tar; do + local file_size=$(stat -f%z "$file" 2>/dev/null || stat -c%s "$file" 2>/dev/null || echo 0) + required_space=$((required_space + file_size)) + done + + local available_space=$(df . | awk 'NR==2 {print $4}') + available_space=$((available_space * 1024)) # Convert to bytes + + if [ $required_space -gt $available_space ]; then + echo "❌ Insufficient disk space" + echo "Required: $(numfmt --to=iec $required_space)" + echo "Available: $(numfmt --to=iec $available_space)" + exit 1 + fi + + echo "✅ Sufficient disk space available" +} + +# Function to import images +import_images() { + echo "📦 Importing Docker images..." + + # Import backend image + echo " 📦 Importing backend-api image..." + docker load -i backend-api.tar + + # Import frontend image + echo " 📦 Importing frontend image..." + docker load -i frontend.tar + + # Import mineru image + echo " 📦 Importing mineru-api image..." + docker load -i mineru-api.tar + + # Import redis image + echo " 📦 Importing redis image..." + docker load -i redis.tar + + echo "✅ All images imported successfully!" +} + +# Function to verify imported images +verify_images() { + echo "🔍 Verifying imported images..." + + local missing_images=() + + if ! docker images | grep -q "legal-doc-masker-backend-api"; then + missing_images+=("legal-doc-masker-backend-api") + fi + + if ! docker images | grep -q "legal-doc-masker-frontend"; then + missing_images+=("legal-doc-masker-frontend") + fi + + if ! docker images | grep -q "legal-doc-masker-mineru-api"; then + missing_images+=("legal-doc-masker-mineru-api") + fi + + if ! docker images | grep -q "redis:alpine"; then + missing_images+=("redis:alpine") + fi + + if [ ${#missing_images[@]} -ne 0 ]; then + echo "❌ Missing imported images: ${missing_images[*]}" + exit 1 + fi + + echo "✅ All images verified successfully!" +} + +# Function to show imported images +show_imported_images() { + echo "" + echo "📊 Imported Images:" + echo "===================" + docker images --format "table {{.Repository}}\t{{.Tag}}\t{{.Size}}" | grep legal-doc-masker + docker images --format "table {{.Repository}}\t{{.Tag}}\t{{.Size}}" | grep redis +} + +# Function to create necessary directories +create_directories() { + echo "" + echo "📁 Creating necessary directories..." + + mkdir -p backend/storage + mkdir -p mineru/storage/uploads + mkdir -p mineru/storage/processed + + echo "✅ Directories created" +} + +# Function to check for required files +check_required_files() { + echo "" + echo "🔍 Checking for required configuration files..." + + local missing_files=() + + if [ ! -f "docker-compose.yml" ]; then + missing_files+=("docker-compose.yml") + fi + + if [ ! -f "DOCKER_COMPOSE_README.md" ]; then + missing_files+=("DOCKER_COMPOSE_README.md") + fi + + if [ ${#missing_files[@]} -ne 0 ]; then + echo "⚠️ Missing files: ${missing_files[*]}" + echo "Please copy these files from the source environment:" + echo " - docker-compose.yml" + echo " - DOCKER_COMPOSE_README.md" + echo " - backend/.env (if exists)" + echo " - frontend/.env (if exists)" + echo " - mineru/.env (if exists)" + else + echo "✅ All required configuration files found" + fi +} + +# Function to show next steps +show_next_steps() { + echo "" + echo "🎉 Import completed successfully!" + echo "" + echo "📋 Next Steps:" + echo "==============" + echo "" + echo "1. Copy configuration files (if not already present):" + echo " - docker-compose.yml" + echo " - backend/.env" + echo " - frontend/.env" + echo " - mineru/.env" + echo "" + echo "2. Start the services:" + echo " docker-compose up -d" + echo "" + echo "3. Verify services are running:" + echo " docker-compose ps" + echo "" + echo "4. Test the endpoints:" + echo " - Frontend: http://localhost:3000" + echo " - Backend API: http://localhost:8000" + echo " - Mineru API: http://localhost:8001" + echo "" + echo "5. View logs if needed:" + echo " docker-compose logs -f [service-name]" +} + +# Function to handle compressed archive +handle_compressed_archive() { + if ls legal-doc-masker-images-*.tar.gz 1> /dev/null 2>&1; then + echo "🗜️ Found compressed archive, extracting..." + tar -xzf legal-doc-masker-images-*.tar.gz + echo "✅ Archive extracted" + fi +} + +# Main execution +main() { + check_docker + handle_compressed_archive + check_tar_files + check_disk_space + import_images + verify_images + show_imported_images + create_directories + check_required_files + show_next_steps +} + +# Run main function +main "$@" \ No newline at end of file