xorbitlab
/
tubewatch
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

docs(build): add comprehensive build guide for playlist monitor 

- Provide detailed system requirements and software dependencies
- Describe multiple build options including uv, pip/venv, and Poetry
- Include steps for development setup, database initialization, and production build
- Document UI integration options: standalone web UI, Angular UI extension, and simple dashboard
- Add Docker build instructions for development, production, and multi-stage builds
- Provide troubleshooting tips for build and runtime issues
- Offer advanced build options such as custom database backend, Redis caching, HTTPS, and load balancing
- Add a build checklist and next step recommendations for users

docs(getting-started): add complete getting started guide for playlist monitor

- Summarize implemented backend features and note missing UI
- Present quick start paths: API-only, simple UI, and full React app
- Detail API usage with examples and testing instructions
- Explain Docker deployment instructions
- List UI implementation priorities and recommended next steps
- Supply common setup issues and resolution tips
- Provide community and debugging resources
- Define success metrics and final usage recommendations

docs(quick-start): add quick start guide for rapid setup of playlist monitor

- Outline prerequisites including Python and MeTube requirements
- Provide recommended Docker usage instructions
- Detail manual installation steps and command examples
- Explain how to access API documentation and MeTube service
- Include example commands to add and monitor playlists via API
- Offer troubleshooting advice for service startup, connectivity, and database issues
- Present common operational commands for playlist
This commit is contained in:
tigeren 2025-11-21 17:17:23 +00:00
parent 02260fd139
commit b01ef238e2
5 changed files with 2954 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

522
playlist-monitor/BUILD_GUIDE.md Normal file
View File

@ -0,0 +1,522 @@
# Playlist Monitor Service - Comprehensive Build Guide
## 📋 Table of Contents
1. [Prerequisites](#prerequisites)
2. [Build Options](#build-options)
3. [Development Setup](#development-setup)
4. [Production Build](#production-build)
5. [UI Integration Options](#ui-integration-options)
6. [Docker Build](#docker-build)
7. [Troubleshooting](#troubleshooting)
8. [Advanced Build Options](#advanced-build-options)
## 🔧 Prerequisites
### System Requirements
- **Python**: 3.13+ (3.14 recommended)
- **Operating System**: Linux/macOS/Windows WSL2
- **Memory**: Minimum 512MB RAM, 1GB recommended
- **Storage**: 100MB for application + space for downloads
- **Network**: Access to YouTube and MeTube instance
### Software Dependencies
```bash
# Ubuntu/Debian
sudo apt update
sudo apt install python3.13 python3.13-dev python3.13-venv build-essential curl git
# macOS (using Homebrew)
brew install python@3.13 curl git
# CentOS/RHEL/Fedora
sudo dnf install python3.13 python3.13-devel gcc curl git
```
### MeTube Requirements
- MeTube instance running (locally or remotely)
- MeTube version that supports playlist downloads
- Network connectivity between services
## 🏗️ Build Options
### Option 1: Modern Python with uv (Recommended)
**Fastest, most reliable method**
```bash
# Install uv package manager
curl -LsSf https://astral.sh/uv/install.sh | sh
source ~/.cargo/env
# Clone and setup
cd /root/workspace/tubewatch/playlist-monitor
uv sync
# Run
uv run python -m app.main
```
### Option 2: Traditional pip/venv
**Standard Python approach**
```bash
# Create virtual environment
python3.13 -m venv venv
source venv/bin/activate # Linux/macOS
# or
venv\Scripts\activate # Windows
# Install dependencies
pip install --upgrade pip
pip install -e .
# Run
python -m app.main
```
### Option 3: Poetry (Alternative)
**For Poetry users**
```bash
# Install Poetry
curl -sSL https://install.python-poetry.org | python3 -
# Setup project
poetry install
poetry run python -m app.main
```
## 🛠️ Development Setup
### 1. Environment Configuration
```bash
# Copy environment template
cp .env.example .env
# Edit configuration
nano .env
```
**Essential settings:**
```env
# MeTube Integration
METUBE_URL=http://localhost:8081
# Database
DATABASE_URL=sqlite:///data/playlists.db
# Server
HOST=0.0.0.0
PORT=8082
# Scheduler
DEFAULT_CHECK_INTERVAL=60
MAX_CONCURRENT_DOWNLOADS=3
# Logging
LOG_LEVEL=INFO
```
### 2. Development Dependencies
```bash
# Install development dependencies
uv sync --extra dev
# Or with pip
pip install -e ".[dev]"
```
### 3. Development Tools Setup
```bash
# Code formatting
uv run black app/ tests/
uv run isort app/ tests/
# Type checking
uv run mypy app/
# Linting
uv run flake8 app/ tests/
```
### 4. Database Setup
```bash
# Create data directory
mkdir -p data logs
# Initialize database (automatic on first run)
uv run python -c "from app.core.database import engine, Base; Base.metadata.create_all(bind=engine)"
```
## 🚀 Production Build
### Step 1: Production Dependencies
```bash
# Install only production dependencies
uv sync --no-dev
# Or with pip
pip install -r requirements.txt
```
### Step 2: Production Configuration
```bash
# Create production environment
cp .env.example .env.production
# Edit production settings
nano .env.production
```
**Production recommendations:**
```env
# Security
DEBUG=false
LOG_LEVEL=WARNING
CORS_ORIGINS=["https://yourdomain.com"]
# Performance
MAX_CONCURRENT_DOWNLOADS=5
DEFAULT_CHECK_INTERVAL=30
# Database (PostgreSQL recommended)
DATABASE_URL=postgresql://user:pass@localhost/metube_playlists
```
### Step 3: Production Server
```bash
# Use production ASGI server
uv run gunicorn app.main:app \
--workers 4 \
--worker-class uvicorn.workers.UvicornWorker \
--bind 0.0.0.0:8082 \
--log-level info
# Or with uvicorn directly
uv run uvicorn app.main:app \
--host 0.0.0.0 \
--port 8082 \
--workers 4 \
--log-level info
```
## 🎨 UI Integration Options
### Option 1: Standalone Web UI (Recommended)
Create a separate frontend application that communicates with the API.
**Technologies:**
- **React** with TypeScript
- **Vue.js 3** with Composition API
- **SvelteKit** for modern approach
**Example React setup:**
```bash
# Create React app
cd /root/workspace/tubewatch
npx create-react-app playlist-monitor-ui --template typescript
cd playlist-monitor-ui
# Install UI libraries
npm install @mui/material @emotion/react @emotion/styled axios react-query
npm install @mui/icons-material @mui/x-data-grid date-fns
# Start development
npm start
```
### Option 2: Extend MeTube Angular UI
Modify the existing MeTube Angular frontend to include playlist monitoring.
**Approach:**
1. Add new Angular components to MeTube's UI
2. Integrate with existing MeTube routing
3. Use MeTube's existing styling and components
**Integration points:**
- Add playlist tab to main navigation
- Extend existing download interface
- Reuse MeTube's component library
### Option 3: Simple HTML Dashboard
For minimal UI requirements.
```bash
# Create simple dashboard
mkdir ui
cd ui
# Create index.html with vanilla JS
cat > index.html << 'EOF'
<!DOCTYPE html>
<html>
<head>
<title>Playlist Monitor</title>
<link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/css/bootstrap.min.css" rel="stylesheet">
</head>
<body>
<div class="container mt-4">
<h1>Playlist Monitor Dashboard</h1>
<div id="playlists"></div>
</div>
<script src="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/js/bootstrap.bundle.min.js"></script>
<script src="app.js"></script>
</body>
</html>
EOF
# Simple JavaScript API client
cat > app.js << 'EOF'
const API_BASE = 'http://localhost:8082/api';
async function loadPlaylists() {
try {
const response = await fetch(`${API_BASE}/playlists`);
const data = await response.json();
displayPlaylists(data);
} catch (error) {
console.error('Error loading playlists:', error);
}
}
function displayPlaylists(playlists) {
const container = document.getElementById('playlists');
container.innerHTML = playlists.map(playlist => `
<div class="card mb-3">
<div class="card-body">
<h5 class="card-title">${playlist.title || 'Untitled'}</h5>
<p class="card-text">${playlist.url}</p>
<p>Status: ${playlist.enabled ? 'Enabled' : 'Disabled'}</p>
</div>
</div>
`).join('');
}
loadPlaylists();
EOF
```
## 🐳 Docker Build
### Development Docker Build
```bash
# Build development image
docker build -t playlist-monitor:dev .
# Run development container
docker run -d \
--name playlist-monitor-dev \
-p 8082:8082 \
-e DEBUG=true \
-v $(pwd)/data:/app/data \
-v $(pwd)/logs:/app/logs \
playlist-monitor:dev
```
### Production Docker Build
```bash
# Build production image
docker build -t playlist-monitor:latest .
# Run production container
docker run -d \
--name playlist-monitor \
-p 8082:8082 \
-e METUBE_URL=http://metube:8081 \
-v playlist-data:/app/data \
-v playlist-logs:/app/logs \
--restart unless-stopped \
--health-cmd="curl -f http://localhost:8082/health || exit 1" \
--health-interval=30s \
playlist-monitor:latest
```
### Multi-stage Docker Build (Optimized)
```dockerfile
# Build stage
FROM python:3.13-slim as builder
WORKDIR /build
COPY pyproject.toml ./
RUN pip install uv
RUN uv sync --no-dev
# Runtime stage
FROM python:3.13-slim
RUN apt-get update && apt-get install -y --no-install-recommends \
curl && \
rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY --from=builder /build/.venv ./.venv
COPY app/ ./app/
ENV PATH="/app/.venv/bin:$PATH"
USER 1000:1000
EXPOSE 8082
CMD ["python", "-m", "app.main"]
```
## 🔍 Troubleshooting
### Build Issues
#### Python Version Mismatch
```bash
# Check Python version
python3 --version
# Install correct version
pyenv install 3.13.0
pyenv local 3.13.0
```
#### Dependency Conflicts
```bash
# Clear cache
uv cache clean
rm -rf .venv
# Reinstall
uv sync
```
#### Compilation Errors
```bash
# Install build dependencies
sudo apt install build-essential python3-dev
# Or use pre-built wheels
pip install --only-binary=all -r requirements.txt
```
### Runtime Issues
#### Port Already in Use
```bash
# Find process using port 8082
lsof -i :8082
# Kill process
kill -9 <PID>
# Or use different port
export PORT=8083
```
#### Database Locked
```bash
# Check for existing processes
ps aux | grep playlist-monitor
# Remove lock file
rm data/*.db-journal
```
#### MeTube Connection Failed
```bash
# Test MeTube connectivity
curl http://localhost:8081/info
# Check MeTube logs
docker logs metube
```
## 🔧 Advanced Build Options
### Custom Database Backend
```bash
# PostgreSQL setup
docker run -d \
--name postgres-playlist \
-e POSTGRES_DB=playlists \
-e POSTGRES_USER=playlist_user \
-e POSTGRES_PASSWORD=secure_password \
-p 5432:5432 \
postgres:15
# Update .env
DATABASE_URL=postgresql://playlist_user:secure_password@localhost:5432/playlists
```
### Redis for Caching
```bash
# Redis setup
docker run -d \
--name redis-playlist \
-p 6379:6379 \
redis:7-alpine
# Install Redis dependencies
uv add redis
```
### HTTPS/SSL Configuration
```bash
# Generate SSL certificates
openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes
# Update configuration
HTTPS_ENABLED=true
SSL_CERTFILE=cert.pem
SSL_KEYFILE=key.pem
```
### Load Balancing with Nginx
```nginx
upstream playlist_monitor {
server localhost:8082;
server localhost:8083;
server localhost:8084;
}
server {
listen 80;
server_name your-domain.com;
location / {
proxy_pass http://playlist_monitor;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
}
```
## 📋 Build Checklist
### Pre-build
- [ ] Python 3.13+ installed
- [ ] MeTube accessible
- [ ] Required ports available (8082)
- [ ] Sufficient disk space
### Build Process
- [ ] Dependencies installed successfully
- [ ] Configuration file created
- [ ] Database initialized
- [ ] Tests pass
- [ ] Service starts without errors
### Post-build
- [ ] API accessible at http://localhost:8082
- [ ] MeTube connection established
- [ ] Database operations working
- [ ] Scheduler running
- [ ] Logs rotating properly
## 🎯 Next Steps
After successful build:
1. **Test the API** using the Quick Start Guide
2. **Set up monitoring** with health checks
3. **Configure automatic startup** with systemd or Docker
4. **Implement UI** (see UI_INTEGRATION.md)
5. **Set up logging** and monitoring
## 📚 Related Documentation
- [QUICK_START_GUIDE.md](QUICK_START_GUIDE.md) - Getting started quickly
- [TESTING_GUIDE.md](TESTING_GUIDE.md) - Comprehensive testing
- [UI_INTEGRATION.md](UI_INTEGRATION.md) - UI implementation options
- [ADVANCED_CONFIG.md](ADVANCED_CONFIG.md) - Advanced configuration options

218
playlist-monitor/GETTING_STARTED.md Normal file
View File

@ -0,0 +1,218 @@
# Playlist Monitor Service - Complete Getting Started Guide
## 🎯 What You Have Now
### ✅ **Fully Implemented Backend**
- **Complete REST API** with 20+ endpoints
- **Database models** with SQLite/PostgreSQL support
- **MeTube integration** via HTTP API + WebSocket
- **Automated playlist monitoring** with configurable intervals
- **Video tracking** with status management
- **Docker support** with multi-stage builds
- **Comprehensive testing** with pytest
### ❌ **Missing: User Interface**
The service currently only provides a **REST API** with Swagger documentation at `http://localhost:8082/docs`. There is **no web UI** for visual management.
## 🚀 Quick Start (Choose Your Path)
### Path 1: API-Only (Immediate Use)
```bash
# Start the service
cd /root/workspace/tubewatch/playlist-monitor
uv run python -m app.main
# Access API documentation
open http://localhost:8082/docs
# Test with curl
curl -X POST http://localhost:8082/api/playlists \
-H "Content-Type: application/json" \
-d '{
"url": "https://www.youtube.com/playlist?list=PLrAXtmErZgOeiKm4sgNOknGvNjby9efdf",
"check_interval": 60,
"quality": "best",
"format": "mp4",
"folder": "kurzgesagt"
}'
```
### Path 2: Add Simple UI (30 minutes)
```bash
# Create basic HTML dashboard
cd /root/workspace/tubewatch/playlist-monitor
mkdir ui
cp templates/simple-dashboard.html ui/index.html
# Modify backend to serve UI
echo "app.mount('/ui', StaticFiles(directory='ui', html=True), name='ui')" >> app/main.py
# Restart and access UI
uv run python -m app.main
open http://localhost:8082/ui
```
### Path 3: Full React App (2-4 hours)
```bash
# Create React application
cd /root/workspace/tubewatch
npx create-react-app playlist-monitor-ui --template typescript
cd playlist-monitor-ui
# Install UI libraries
npm install @mui/material @emotion/react @emotion/styled axios react-query
# Follow UI_INTEGRATION.md for complete setup
```
## 📚 Documentation You Have
### Essential Guides
- **[QUICK_START_GUIDE.md](QUICK_START_GUIDE.md)** - Get started in 5 minutes
- **[BUILD_GUIDE.md](BUILD_GUIDE.md)** - Comprehensive build instructions
- **[TESTING_GUIDE.md](TESTING_GUIDE.md)** - Complete testing procedures
- **[UI_INTEGRATION.md](UI_INTEGRATION.md)** - UI implementation options
### Reference Documents
- **[IMPLEMENTATION_STATUS.md](IMPLEMENTATION_STATUS.md)** - What was built
- **[PLAYLIST_MONITOR_ARCHITECTURE.md](PLAYLIST_MONITOR_ARCHITECTURE.md)** - Original architecture
## 🎯 What You Can Do Right Now
### 1. Start Using the API (No UI Needed)
```bash
# Check system status
curl http://localhost:8082/api/status
# Add a playlist
curl -X POST http://localhost:8082/api/playlists \
-H "Content-Type: application/json" \
-d '{"url": "https://www.youtube.com/playlist?list=TEST123"}'
# Monitor downloads
curl http://localhost:8082/api/playlists/{id}/videos
```
### 2. Test the Service
```bash
# Run all tests
uv run pytest tests/ -v
# Run specific test
uv run pytest tests/unit/test_models.py -v
# Check coverage
uv run pytest --cov=app --cov-report=html
```
### 3. Deploy with Docker
```bash
# From main tubewatch directory
docker-compose -f docker-compose-with-monitor.yml up -d
# Check status
docker-compose -f docker-compose-with-monitor.yml ps
```
## 🎨 UI Implementation Priority
Since you mentioned the UI is missing, here are your options ranked by effort:
### 🟢 **15 Minutes: Basic HTML Dashboard**
- Simple Bootstrap-based interface
- List playlists, show status
- Add/delete operations
- **Good for**: Quick setup, basic needs
### 🟡 **2-4 Hours: React App**
- Modern Material-UI interface
- Real-time updates, charts
- Full CRUD operations
- **Good for**: Production use, modern UX
### 🔴 **8+ Hours: Extend MeTube Angular**
- Integrate with existing MeTube UI
- Consistent styling and navigation
- Most complex but seamless
- **Good for**: Existing MeTube deployments
## 🚀 Recommended Next Steps
### Immediate (Next 30 minutes)
1. **Start the service**: `uv run python -m app.main`
2. **Test the API**: Visit http://localhost:8082/docs
3. **Add a playlist**: Use the interactive API docs
4. **Monitor progress**: Check logs and status endpoints
### Short Term (Next few hours)
1. **Choose UI option** based on your needs
2. **Implement basic interface** following UI_INTEGRATION.md
3. **Test end-to-end workflow** with real playlists
4. **Set up monitoring** and health checks
### Long Term (Next few days)
1. **Deploy to production** with Docker
2. **Set up automated monitoring** and alerts
3. **Add authentication** if needed
4. **Optimize performance** based on usage
## 🔧 Common First-Time Setup Issues
### "MeTube connection failed"
- Ensure MeTube is running on port 8081
- Check MeTube logs: `docker logs metube`
- Verify network connectivity
### "Port 8082 already in use"
- Find process: `lsof -i :8082`
- Kill process or change port in .env
### "Database locked"
- Remove lock file: `rm data/*.db-journal`
- Check for zombie processes
### "Python version too old"
- Install Python 3.13+ or use Docker
- Use pyenv for version management
## 📞 Getting Help
### Documentation
- **API Reference**: http://localhost:8082/docs (when running)
- **Architecture**: See PLAYLIST_MONITOR_ARCHITECTURE.md
- **Build Issues**: See BUILD_GUIDE.md troubleshooting section
### Debugging
- Check logs: `tail -f logs/playlist-monitor.log`
- Health check: `curl http://localhost:8082/health`
- System status: `curl http://localhost:8082/api/status`
### Community
- Check existing issues in the repository
- Create detailed bug reports with logs
- Include system information and reproduction steps
## 🎉 Success Metrics
You'll know everything is working when you can:
1. ✅ **Start the service** without errors
2. ✅ **Access API docs** at http://localhost:8082/docs
3. ✅ **Add a playlist** via API or UI
4. ✅ **See videos detected** from the playlist
5. ✅ **Monitor download progress** in real-time
6. ✅ **Manage video status** (skip, reset, etc.)
## 🎯 Final Recommendation
**Start with the API-only approach** to verify everything works, then add a UI based on your comfort level:
- **API-only**: Great for automation, scripts, or integration with other tools
- **Simple HTML**: Perfect for personal use or quick setup
- **React App**: Best for production deployments and user-friendly experience
The backend is **production-ready** and thoroughly tested. Focus your effort on the UI implementation based on your specific needs! 🚀
---
**Next Step**: Choose your UI path and follow the relevant guide in [UI_INTEGRATION.md](UI_INTEGRATION.md)!

208
playlist-monitor/QUICK_START_GUIDE.md Normal file
View File

@ -0,0 +1,208 @@
# Playlist Monitor Service - Quick Start Guide
## 🚀 Quick Start (5 minutes)
### Prerequisites
- Python 3.13+ installed
- MeTube running (or available to start)
- Basic command line knowledge
### Option 1: Docker (Recommended)
```bash
# From the main tubewatch directory
cd /root/workspace/tubewatch
# Start both MeTube and Playlist Monitor
docker-compose -f docker-compose-with-monitor.yml up -d
# Check if services are running
docker-compose -f docker-compose-with-monitor.yml ps
# View logs
docker-compose -f docker-compose-with-monitor.yml logs -f playlist-monitor
```
### Option 2: Manual Installation
```bash
# Navigate to playlist monitor directory
cd /root/workspace/tubewatch/playlist-monitor
# Install uv (if not already installed)
curl -LsSf https://astral.sh/uv/install.sh | sh
source ~/.cargo/env
# Install dependencies
uv sync
# Copy environment file
cp .env.example .env
# Start the service
uv run python -m app.main
```
### Access the Services
- **Playlist Monitor API**: http://localhost:8082/docs
- **MeTube**: http://localhost:8081
- **API Documentation**: http://localhost:8082/docs
## 🎯 Your First Playlist
### Using the API (Interactive)
1. Open http://localhost:8082/docs
2. Click on `POST /api/playlists`
3. Click "Try it out"
4. Enter this data:
```json
{
"url": "https://www.youtube.com/playlist?list=PLrAXtmErZgOeiKm4sgNOknGvNjby9efdf",
"check_interval": 60,
"quality": "best",
"format": "mp4",
"folder": "kurzgesagt",
"enabled": true
}
```
5. Click "Execute"
### Using curl
```bash
# Add a playlist
curl -X POST http://localhost:8082/api/playlists \
-H "Content-Type: application/json" \
-d '{
"url": "https://www.youtube.com/playlist?list=PLrAXtmErZgOeiKm4sgNOknGvNjby9efdf",
"check_interval": 60,
"quality": "best",
"format": "mp4",
"folder": "kurzgesagt"
}'
# List playlists
curl http://localhost:8082/api/playlists
# Check system status
curl http://localhost:8082/api/status
```
## 📊 Monitor Your Downloads
### Check Status
```bash
# System status
curl http://localhost:8082/api/status
# List playlists with stats
curl http://localhost:8082/api/playlists
# Get specific playlist details
curl http://localhost:8082/api/playlists/{playlist_id}
```
### View Videos
```bash
# List videos for a playlist
curl http://localhost:8082/api/playlists/{playlist_id}/videos
# Get specific video details
curl http://localhost:8082/api/videos/{video_id}
```
## 🛠️ Troubleshooting
### Service Won't Start
```bash
# Check if port 8082 is available
netstat -tulpn | grep 8082
# Check logs
tail -f logs/playlist-monitor.log
# Test configuration
uv run python -c "from app.core.config import settings; print(settings.METUBE_URL)"
```
### MeTube Connection Issues
```bash
# Test MeTube connectivity
curl http://localhost:8081/info
# Check MeTube logs
docker logs metube
# Verify MeTube URL in .env file
grep METUBE_URL .env
```
### Database Issues
```bash
# Check database file
ls -la data/playlists.db
# Reset database (WARNING: deletes all data)
rm data/playlists.db
# Then restart the service
```
## 🎮 Common Operations
### Add Multiple Playlists
```bash
# YouTube channel uploads playlist
curl -X POST http://localhost:8082/api/playlists \
-H "Content-Type: application/json" \
-d '{
"url": "https://www.youtube.com/playlist?list=UUX6OQ3DkcsbYNE6H8uQQuVA",
"check_interval": 30,
"folder": "tech-channels"
}'
# Specific playlist
curl -X POST http://localhost:8082/api/playlists \
-H "Content-Type: application/json" \
-d '{
"url": "https://www.youtube.com/playlist?list=PLQVvvaa0QuDfpEcGUM6ogsbrlWtqpS5-1",
"check_interval": 120,
"folder": "python-tutorials"
}'
```
### Manual Playlist Check
```bash
# Force check a playlist
curl -X POST http://localhost:8082/api/playlists/{playlist_id}/check?force=true
```
### Manage Videos
```bash
# Mark video as moved
curl -X POST http://localhost:8082/api/videos/{video_id}/file-moved \
-H "Content-Type: application/json" \
-d '{"location_note": "Moved to /mnt/nas/videos/"}'
# Reset failed video
curl -X POST http://localhost:8082/api/videos/{video_id}/reset
```
## 📈 Next Steps
1. **Set up a web UI** (see BUILD_GUIDE.md for UI options)
2. **Configure automatic startup** with systemd or Docker
3. **Set up monitoring** with health checks
4. **Customize settings** for your needs
5. **Add authentication** if needed (see ADVANCED_CONFIG.md)
## 🆘 Need Help?
- Check the full documentation: [README.md](README.md)
- View API docs: http://localhost:8082/docs
- Check logs: `tail -f logs/playlist-monitor.log`
- Run health check: `curl http://localhost:8082/health`
## 🚀 Going Further
See these guides for advanced usage:
- [BUILD_GUIDE.md](BUILD_GUIDE.md) - Detailed build instructions
- [TESTING_GUIDE.md](TESTING_GUIDE.md) - Comprehensive testing
- [UI_INTEGRATION.md](UI_INTEGRATION.md) - UI implementation options
- [ADVANCED_CONFIG.md](ADVANCED_CONFIG.md) - Advanced configuration

921
playlist-monitor/TESTING_GUIDE.md Normal file
View File

@ -0,0 +1,921 @@
# Playlist Monitor Service - Comprehensive Testing Guide
## 📋 Table of Contents
1. [Testing Overview](#testing-overview)
2. [Test Environment Setup](#test-environment-setup)
3. [Unit Tests](#unit-tests)
4. [Integration Tests](#integration-tests)
5. [API Tests](#api-tests)
6. [End-to-End Tests](#end-to-end-tests)
7. [Performance Tests](#performance-tests)
8. [Manual Testing](#manual-testing)
9. [CI/CD Integration](#cicd-integration)
10. [Test Data Management](#test-data-management)
## 🔍 Testing Overview
### Test Pyramid
```
🎯 E2E Tests (Few)
↑
🔌 Integration Tests (Some)
↑
⚡ Unit Tests (Many)
```
### Test Categories
- **Unit Tests**: Individual components (models, services, utils)
- **Integration Tests**: Component interactions (database, API, external services)
- **API Tests**: REST endpoint validation
- **E2E Tests**: Full user workflows
- **Performance Tests**: Load and stress testing
## 🛠️ Test Environment Setup
### 1. Test Dependencies
```bash
# Install test dependencies
uv sync --extra dev
# Or install manually
pip install pytest pytest-asyncio pytest-cov pytest-mock httpx
```
### 2. Test Configuration
```bash
# Create test environment
cp .env.example .env.test
# Update test settings
nano .env.test
```
**Test-specific settings:**
```env
# Test database (in-memory SQLite)
DATABASE_URL=sqlite:///:memory:
# Test logging
LOG_LEVEL=DEBUG
LOG_FILE=logs/test.log
# Test MeTube (mock or test instance)
METUBE_URL=http://localhost:8081
# Test mode
TESTING=true
```
### 3. Test Directory Structure
```
tests/
├── unit/ # Unit tests
│ ├── test_config.py
│ ├── test_models.py
│ └── test_services.py
├── integration/ # Integration tests
│ ├── test_database.py
│ ├── test_api.py
│ └── test_metube_client.py
├── e2e/ # End-to-end tests
│ ├── test_workflows.py
│ └── test_scenarios.py
├── fixtures/ # Test data
│ ├── playlists.json
│ └── videos.json
└── conftest.py # Pytest configuration
```
## ⚡ Unit Tests
### Running Unit Tests
```bash
# Run all unit tests
uv run pytest tests/unit/ -v
# Run specific test file
uv run pytest tests/unit/test_models.py -v
# Run with coverage
uv run pytest tests/unit/ --cov=app --cov-report=html
# Run specific test
uv run pytest tests/unit/test_models.py::test_playlist_creation -v
```
### Model Tests (tests/unit/test_models.py)
```python
import pytest
from datetime import datetime, timedelta
from app.models.playlist import PlaylistSubscription
from app.models.video import VideoRecord, VideoStatus
class TestPlaylistModel:
def test_playlist_creation(self, test_db):
"""Test playlist creation and validation"""
playlist = PlaylistSubscription(
url="https://www.youtube.com/playlist?list=TEST123",
title="Test Playlist",
check_interval=60
)
assert playlist.url == "https://www.youtube.com/playlist?list=TEST123"
assert playlist.title == "Test Playlist"
assert playlist.check_interval == 60
assert playlist.enabled is True
def test_playlist_should_check_logic(self):
"""Test playlist check logic"""
playlist = PlaylistSubscription(check_interval=60)
# Should check if never checked
assert playlist.should_check() is True
# Should not check if recently checked
playlist.last_checked = datetime.utcnow() - timedelta(minutes=30)
assert playlist.should_check() is False
# Should check if interval passed
playlist.last_checked = datetime.utcnow() - timedelta(minutes=61)
assert playlist.should_check() is True
class TestVideoModel:
def test_video_status_transitions(self):
"""Test video status management"""
video = VideoRecord(status=VideoStatus.PENDING)
# Test downloading
video.mark_as_downloading("metube_123")
assert video.status == VideoStatus.DOWNLOADING
assert video.metube_download_id == "metube_123"
# Test completion
video.mark_as_completed("video.mp4")
assert video.status == VideoStatus.COMPLETED
assert video.original_filename == "video.mp4"
```
### Service Tests (tests/unit/test_services.py)
```python
import pytest
from unittest.mock import Mock, AsyncMock
from app.services.playlist_service import PlaylistService
from app.services.metube_client import MeTubeClient
class TestPlaylistService:
@pytest.fixture
def service(self, test_db):
return PlaylistService(test_db)
@pytest.fixture
def mock_yt_dlp(self, monkeypatch):
"""Mock yt-dlp for testing"""
mock = Mock()
mock.extract_info.return_value = {
"title": "Test Playlist",
"entries": [
{"id": "video1", "title": "Video 1", "playlist_index": 1},
{"id": "video2", "title": "Video 2", "playlist_index": 2}
]
}
monkeypatch.setattr("yt_dlp.YoutubeDL", lambda x: mock)
return mock
async def test_add_playlist_success(self, service, mock_yt_dlp):
"""Test successful playlist addition"""
playlist = await service.add_playlist(
url="https://www.youtube.com/playlist?list=TEST123",
title="Test Playlist"
)
assert playlist.title == "Test Playlist"
assert playlist.check_interval == 60
assert len(playlist.videos) == 2
```
### Configuration Tests (tests/unit/test_config.py)
```python
import pytest
from app.core.config import Settings
class TestConfiguration:
def test_default_values(self):
"""Test default configuration values"""
settings = Settings()
assert settings.HOST == "0.0.0.0"
assert settings.PORT == 8082
assert settings.METUBE_URL == "http://localhost:8081"
def test_validation(self):
"""Test configuration validation"""
with pytest.raises(ValueError):
Settings(DEFAULT_CHECK_INTERVAL=0)
with pytest.raises(ValueError):
Settings(MAX_CONCURRENT_DOWNLOADS=15)
```
## 🔌 Integration Tests
### Database Integration (tests/integration/test_database.py)
```python
import pytest
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from app.core.database import Base, get_db
from app.models.playlist import PlaylistSubscription
@pytest.fixture
def test_engine():
"""Create test database engine"""
engine = create_engine("sqlite:///:memory:", echo=False)
Base.metadata.create_all(bind=engine)
return engine
class TestDatabaseIntegration:
def test_playlist_crud_operations(self, test_engine):
"""Test database CRUD operations"""
SessionLocal = sessionmaker(bind=test_engine)
db = SessionLocal()
# Create
playlist = PlaylistSubscription(
url="https://www.youtube.com/playlist?list=TEST123",
title="Test Playlist"
)
db.add(playlist)
db.commit()
# Read
retrieved = db.query(PlaylistSubscription).first()
assert retrieved.title == "Test Playlist"
# Update
retrieved.title = "Updated Playlist"
db.commit()
assert db.query(PlaylistSubscription).first().title == "Updated Playlist"
# Delete
db.delete(retrieved)
db.commit()
assert db.query(PlaylistSubscription).count() == 0
db.close()
```
### MeTube Client Integration (tests/integration/test_metube_client.py)
```python
import pytest
import respx
from httpx import Response
from app.services.metube_client import MeTubeClient
@pytest.mark.asyncio
class TestMeTubeClientIntegration:
@respx.mock
async def test_add_download_success(self):
"""Test successful download addition"""
# Mock MeTube API
route = respx.post("http://localhost:8081/add").mock(
return_value=Response(200, json={"id": "download_123", "status": "pending"})
)
client = MeTubeClient("http://localhost:8081")
await client.connect()
result = await client.add_download(
url="https://www.youtube.com/watch?v=TEST123",
quality="best",
format="mp4"
)
assert result["id"] == "download_123"
assert result["status"] == "pending"
assert route.called
await client.disconnect()
```
## 🌐 API Tests
### API Integration (tests/integration/test_api.py)
```python
import pytest
from fastapi.testclient import TestClient
from app.main import app
class TestAPIIntegration:
@pytest.fixture
def client(self):
return TestClient(app)
def test_health_endpoint(self, client):
"""Test health check endpoint"""
response = client.get("/health")
assert response.status_code == 200
data = response.json()
assert data["status"] == "healthy"
def test_playlist_crud(self, client):
"""Test playlist CRUD operations"""
# Create playlist
response = client.post("/api/playlists", json={
"url": "https://www.youtube.com/playlist?list=TEST123",
"title": "Test Playlist",
"check_interval": 60
})
assert response.status_code == 201
playlist_id = response.json()["id"]
# Read playlist
response = client.get(f"/api/playlists/{playlist_id}")
assert response.status_code == 200
assert response.json()["title"] == "Test Playlist"
# Update playlist
response = client.put(f"/api/playlists/{playlist_id}", json={
"title": "Updated Playlist"
})
assert response.status_code == 200
assert response.json()["title"] == "Updated Playlist"
# Delete playlist
response = client.delete(f"/api/playlists/{playlist_id}")
assert response.status_code == 204
def test_invalid_playlist_url(self, client):
"""Test validation of invalid playlist URLs"""
response = client.post("/api/playlists", json={
"url": "https://invalid-url.com",
"title": "Invalid Playlist"
})
assert response.status_code == 400
assert "Invalid YouTube playlist URL" in response.json()["detail"]
```
### Async API Tests (tests/integration/test_api_async.py)
```python
import pytest
import asyncio
from httpx import AsyncClient
from app.main import app
@pytest.mark.asyncio
class TestAsyncAPI:
async def test_websocket_connection(self):
"""Test WebSocket connection and events"""
# This would require a mock WebSocket server
pass
async def test_async_playlist_check(self):
"""Test async playlist checking"""
async with AsyncClient(app=app, base_url="http://test") as client:
# Create playlist
response = await client.post("/api/playlists", json={
"url": "https://www.youtube.com/playlist?list=TEST123",
"title": "Test Playlist"
})
playlist_id = response.json()["id"]
# Trigger check
response = await client.post(f"/api/playlists/{playlist_id}/check")
assert response.status_code == 200
assert response.json()["status"] == "ok"
```
## 🎯 End-to-End Tests
### Workflow Tests (tests/e2e/test_workflows.py)
```python
import pytest
import asyncio
from httpx import AsyncClient
from app.main import app
@pytest.mark.asyncio
class TestE2EWorkflows:
async def test_complete_playlist_workflow(self):
"""Test complete playlist lifecycle"""
async with AsyncClient(app=app, base_url="http://test") as client:
# 1. Create playlist
response = await client.post("/api/playlists", json={
"url": "https://www.youtube.com/playlist?list=PLrAXtmErZgOeiKm4sgNOknGvNjby9efdf",
"title": "Kurzgesagt Playlist",
"check_interval": 60,
"folder": "kurzgesagt"
})
assert response.status_code == 201
playlist_id = response.json()["id"]
# 2. Verify playlist was created
response = await client.get(f"/api/playlists/{playlist_id}")
assert response.status_code == 200
playlist = response.json()
assert playlist["title"] == "Kurzgesagt Playlist"
# 3. Check playlist (manual trigger)
response = await client.post(f"/api/playlists/{playlist_id}/check")
assert response.status_code == 200
# 4. Verify videos were created
response = await client.get(f"/api/playlists/{playlist_id}/videos")
assert response.status_code == 200
videos = response.json()
assert len(videos) > 0
# 5. Test video operations
video_id = videos[0]["id"]
# Skip a video
response = await client.post(f"/api/videos/{video_id}/skip")
assert response.status_code == 200
assert response.json()["video"]["status"] == "SKIPPED"
# Reset video
response = await client.post(f"/api/videos/{video_id}/reset")
assert response.status_code == 200
assert response.json()["video"]["status"] == "PENDING"
# 6. Check system status
response = await client.get("/api/status")
assert response.status_code == 200
status = response.json()
assert status["total_playlists"] >= 1
assert status["total_videos"] >= 1
# 7. Delete playlist
response = await client.delete(f"/api/playlists/{playlist_id}")
assert response.status_code == 204
async def test_error_handling_workflow(self):
"""Test error handling scenarios"""
async with AsyncClient(app=app, base_url="http://test") as client:
# Test invalid playlist URL
response = await client.post("/api/playlists", json={
"url": "https://invalid-url.com",
"title": "Invalid Playlist"
})
assert response.status_code == 400
# Test non-existent playlist
response = await client.get("/api/playlists/non-existent-id")
assert response.status_code == 404
# Test invalid video operations
response = await client.post("/api/videos/non-existent/download")
assert response.status_code == 404
```
## ⚡ Performance Tests
### Load Testing (tests/performance/test_load.py)
```python
import asyncio
import time
import aiohttp
import pytest
class TestLoadPerformance:
@pytest.mark.performance
async def test_api_response_time(self):
"""Test API response times"""
async with aiohttp.ClientSession() as session:
start_time = time.time()
for i in range(100):
async with session.get("http://localhost:8082/api/status") as response:
assert response.status == 200
elapsed = time.time() - start_time
avg_response_time = elapsed / 100
# Assert average response time is under 100ms
assert avg_response_time < 0.1
@pytest.mark.performance
async def test_database_performance(self):
"""Test database query performance"""
from app.core.database import SessionLocal
from app.models.playlist import PlaylistSubscription
db = SessionLocal()
# Create test data
for i in range(1000):
playlist = PlaylistSubscription(
url=f"https://www.youtube.com/playlist?list=TEST{i}",
title=f"Test Playlist {i}"
)
db.add(playlist)
db.commit()
# Measure query performance
start_time = time.time()
playlists = db.query(PlaylistSubscription).all()
elapsed = time.time() - start_time
assert len(playlists) == 1000
assert elapsed < 1.0 # Should complete in under 1 second
db.close()
```
### Stress Testing (tests/performance/test_stress.py)
```python
import asyncio
import pytest
from httpx import AsyncClient
from app.main import app
@pytest.mark.stress
class TestStressScenarios:
async def test_concurrent_playlist_creation(self):
"""Test concurrent playlist creation"""
async with AsyncClient(app=app, base_url="http://test") as client:
# Create 50 playlists concurrently
tasks = []
for i in range(50):
task = client.post("/api/playlists", json={
"url": f"https://www.youtube.com/playlist?list=TEST{i}",
"title": f"Stress Test Playlist {i}"
})
tasks.append(task)
responses = await asyncio.gather(*tasks)
# All should succeed
for response in responses:
assert response.status_code == 201
# Verify all were created
response = await client.get("/api/playlists")
assert response.status_code == 200
assert len(response.json()) >= 50
```
## 🔧 Manual Testing
### Manual Test Checklist
#### Basic Functionality
- [ ] Service starts without errors
- [ ] Health endpoint returns 200
- [ ] API documentation accessible at /docs
- [ ] Database connection successful
#### Playlist Operations
- [ ] Create playlist with valid YouTube URL
- [ ] Create playlist with invalid URL (should fail)
- [ ] Update playlist settings
- [ ] Delete playlist
- [ ] List all playlists
- [ ] Get playlist with videos
#### Video Operations
- [ ] Videos automatically created from playlist
- [ ] Manual playlist check triggers new video detection
- [ ] Video status transitions work correctly
- [ ] Skip/reset operations work
#### Integration
- [ ] MeTube connection established
- [ ] Downloads triggered successfully
- [ ] WebSocket events received
- [ ] Status synchronization works
#### Error Handling
- [ ] Invalid URLs handled gracefully
- [ ] Database errors handled
- [ ] MeTube connection failures handled
- [ ] Proper error messages returned
### Manual Testing Commands
#### Service Health
```bash
# Health check
curl http://localhost:8082/health
# System status
curl http://localhost:8082/api/status
# Scheduler status
curl http://localhost:8082/api/scheduler/status
```
#### Playlist Testing
```bash
# Create test playlist
curl -X POST http://localhost:8082/api/playlists \
-H "Content-Type: application/json" \
-d '{
"url": "https://www.youtube.com/playlist?list=PLrAXtmErZgOeiKm4sgNOknGvNjby9efdf",
"title": "Manual Test Playlist",
"check_interval": 30
}'
# Trigger manual check
curl -X POST http://localhost:8082/api/playlists/{id}/check
# Monitor logs
tail -f logs/playlist-monitor.log
```
## 🔄 CI/CD Integration
### GitHub Actions (.github/workflows/tests.yml)
```yaml
name: Tests
on:
push:
branches: [ main, develop ]
pull_request:
branches: [ main ]
jobs:
test:
runs-on: ubuntu-latest
services:
postgres:
image: postgres:15
env:
POSTGRES_PASSWORD: postgres
POSTGRES_DB: test_playlists
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
- 5432:5432
strategy:
matrix:
python-version: [3.13, 3.14]
steps:
- uses: actions/checkout@v3
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Install uv
run: curl -LsSf https://astral.sh/uv/install.sh | sh
- name: Install dependencies
run: uv sync --extra dev
- name: Run unit tests
run: uv run pytest tests/unit/ -v --cov=app --cov-report=xml
- name: Run integration tests
run: uv run pytest tests/integration/ -v
- name: Run performance tests
run: uv run pytest tests/performance/ -v --tb=short
- name: Upload coverage reports
uses: codecov/codecov-action@v3
with:
file: ./coverage.xml
```
### Pre-commit Hooks (.pre-commit-config.yaml)
```yaml
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.4.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-yaml
- id: check-added-large-files
- repo: https://github.com/psf/black
rev: 23.3.0
hooks:
- id: black
language_version: python3.13
- repo: https://github.com/pycqa/isort
rev: 5.12.0
hooks:
- id: isort
- repo: https://github.com/pycqa/flake8
rev: 6.0.0
hooks:
- id: flake8
- repo: https://github.com/pre-commit/mirrors-mypy
rev: v1.3.0
hooks:
- id: mypy
additional_dependencies: [types-all]
```
## 📊 Test Data Management
### Test Fixtures (tests/fixtures/)
```python
# tests/fixtures/playlists.py
TEST_PLAYLISTS = [
{
"url": "https://www.youtube.com/playlist?list=PLrAXtmErZgOeiKm4sgNOknGvNjby9efdf",
"title": "Kurzgesagt – In a Nutshell",
"check_interval": 60,
"quality": "best",
"format": "mp4",
"folder": "kurzgesagt"
},
{
"url": "https://www.youtube.com/playlist?list=UUX6OQ3DkcsbYNE6H8uQQuVA",
"title": "Tech Channel Uploads",
"check_interval": 30,
"quality": "1080p",
"format": "mp4",
"folder": "tech-channels"
}
]
# tests/fixtures/videos.py
TEST_VIDEOS = [
{
"video_id": "dQw4w9WgXcQ",
"title": "Never Gonna Give You Up",
"playlist_index": 1,
"status": "COMPLETED"
},
{
"video_id": "9bZkp7q19f0",
"title": "Gangnam Style",
"playlist_index": 2,
"status": "PENDING"
}
]
```
### Mock Data Generation
```python
# tests/utils/mock_data.py
def generate_mock_playlists(count=10):
"""Generate mock playlist data"""
playlists = []
for i in range(count):
playlists.append({
"url": f"https://www.youtube.com/playlist?list=TEST{i:03d}",
"title": f"Test Playlist {i}",
"check_interval": 60,
"quality": "best",
"format": "mp4",
"folder": f"test-folder-{i}"
})
return playlists
def generate_mock_videos(playlist_id, count=20):
"""Generate mock video data"""
videos = []
for i in range(count):
videos.append({
"playlist_id": playlist_id,
"video_url": f"https://www.youtube.com/watch?v=VIDEO{i:03d}",
"video_id": f"VIDEO{i:03d}",
"title": f"Test Video {i}",
"playlist_index": i + 1,
"status": "PENDING"
})
return videos
```
## 📈 Test Coverage
### Coverage Report Generation
```bash
# Generate coverage report
uv run pytest --cov=app --cov-report=html --cov-report=term
# View HTML report
open htmlcov/index.html
# Generate XML report for CI
uv run pytest --cov=app --cov-report=xml
```
### Coverage Goals
- **Unit Tests**: >90% coverage
- **Integration Tests**: >80% coverage
- **API Tests**: >95% coverage
- **Overall**: >85% coverage
### Coverage Configuration (pyproject.toml)
```toml
[tool.coverage.run]
source = ["app"]
omit = [
"*/tests/*",
"*/venv/*",
"*/__pycache__/*",
"app/main.py", # Entry point
]
[tool.coverage.report]
exclude_lines = [
"pragma: no cover",
"def __repr__",
"raise AssertionError",
"raise NotImplementedError",
]
[tool.coverage.html]
directory = "htmlcov"
```
## 🎯 Test Execution Strategies
### Parallel Testing
```bash
# Run tests in parallel
uv run pytest -n auto
# Run specific test categories in parallel
uv run pytest tests/unit/ -n 4 &
uv run pytest tests/integration/ -n 2 &
wait
```
### Test Selection
```bash
# Run tests by marker
uv run pytest -m "not performance"
uv run pytest -m "performance"
# Run tests by name pattern
uv run pytest -k "test_playlist"
uv run pytest -k "not test_performance"
# Run specific test file
uv run pytest tests/unit/test_models.py
```
### Continuous Testing
```bash
# Watch mode for development
uv run pytest-watch tests/unit/
# Run on file changes
uv run ptw -- tests/unit/
```
## 📚 Testing Best Practices
### 1. Test Naming
```python
def test_playlist_creation_with_valid_url(self):
"""Should create playlist when valid YouTube URL provided"""
# Test implementation
def test_playlist_creation_fails_with_invalid_url(self):
"""Should reject playlist creation with invalid URL"""
# Test implementation
```
### 2. Test Independence
```python
@pytest.fixture
def clean_database():
"""Provide clean database for each test"""
# Setup clean state
yield
# Cleanup after test
```
### 3. Mock External Services
```python
@pytest.fixture
def mock_metube():
"""Mock MeTube service for testing"""
with respx.mock:
respx.get("http://localhost:8081/info").mock(
return_value=Response(200, json={"status": "ok"})
)
yield
```
### 4. Test Data Management
```python
@pytest.fixture
def sample_playlist():
"""Provide sample playlist data"""
return {
"url": "https://www.youtube.com/playlist?list=TEST123",
"title": "Test Playlist",
"check_interval": 60
}
```
This comprehensive testing guide ensures robust, reliable testing of the Playlist Monitor Service! 🧪✅

1085
playlist-monitor/UI_INTEGRATION.md Normal file
View File

File diff suppressed because it is too large Load Diff