tubewatch/playlist-monitor/TESTING_GUIDE.md

Playlist Monitor Service - Comprehensive Testing Guide

📋 Table of Contents

1. Testing Overview
2. Test Environment Setup
3. Unit Tests
4. Integration Tests
5. API Tests
6. End-to-End Tests
7. Performance Tests
8. Manual Testing
9. CI/CD Integration
10. Test Data Management

🔍 Testing Overview

Test Pyramid

    🎯 E2E Tests (Few)
         ↑
    🔌 Integration Tests (Some)
         ↑
    ⚡ Unit Tests (Many)

Test Categories

• Unit Tests: Individual components (models, services, utils)
• Integration Tests: Component interactions (database, API, external services)
• API Tests: REST endpoint validation
• E2E Tests: Full user workflows
• Performance Tests: Load and stress testing

🛠️ Test Environment Setup

1. Test Dependencies

# Install test dependencies
uv sync --extra dev

# Or install manually
pip install pytest pytest-asyncio pytest-cov pytest-mock httpx

2. Test Configuration

# Create test environment
cp .env.example .env.test

# Update test settings
nano .env.test

Test-specific settings:

# Test database (in-memory SQLite)
DATABASE_URL=sqlite:///:memory:

# Test logging
LOG_LEVEL=DEBUG
LOG_FILE=logs/test.log

# Test MeTube (mock or test instance)
METUBE_URL=http://localhost:8081

# Test mode
TESTING=true

3. Test Directory Structure

tests/
├── unit/                    # Unit tests
│   ├── test_config.py
│   ├── test_models.py
│   └── test_services.py
├── integration/             # Integration tests
│   ├── test_database.py
│   ├── test_api.py
│   └── test_metube_client.py
├── e2e/                     # End-to-end tests
│   ├── test_workflows.py
│   └── test_scenarios.py
├── fixtures/                # Test data
│   ├── playlists.json
│   └── videos.json
└── conftest.py             # Pytest configuration

⚡ Unit Tests

Running Unit Tests

# Run all unit tests
uv run pytest tests/unit/ -v

# Run specific test file
uv run pytest tests/unit/test_models.py -v

# Run with coverage
uv run pytest tests/unit/ --cov=app --cov-report=html

# Run specific test
uv run pytest tests/unit/test_models.py::test_playlist_creation -v

Model Tests (tests/unit/test_models.py)

import pytest
from datetime import datetime, timedelta
from app.models.playlist import PlaylistSubscription
from app.models.video import VideoRecord, VideoStatus

class TestPlaylistModel:
    def test_playlist_creation(self, test_db):
        """Test playlist creation and validation"""
        playlist = PlaylistSubscription(
            url="https://www.youtube.com/playlist?list=TEST123",
            title="Test Playlist",
            check_interval=60
        )
        
        assert playlist.url == "https://www.youtube.com/playlist?list=TEST123"
        assert playlist.title == "Test Playlist"
        assert playlist.check_interval == 60
        assert playlist.enabled is True
    
    def test_playlist_should_check_logic(self):
        """Test playlist check logic"""
        playlist = PlaylistSubscription(check_interval=60)
        
        # Should check if never checked
        assert playlist.should_check() is True
        
        # Should not check if recently checked
        playlist.last_checked = datetime.utcnow() - timedelta(minutes=30)
        assert playlist.should_check() is False
        
        # Should check if interval passed
        playlist.last_checked = datetime.utcnow() - timedelta(minutes=61)
        assert playlist.should_check() is True

class TestVideoModel:
    def test_video_status_transitions(self):
        """Test video status management"""
        video = VideoRecord(status=VideoStatus.PENDING)
        
        # Test downloading
        video.mark_as_downloading("metube_123")
        assert video.status == VideoStatus.DOWNLOADING
        assert video.metube_download_id == "metube_123"
        
        # Test completion
        video.mark_as_completed("video.mp4")
        assert video.status == VideoStatus.COMPLETED
        assert video.original_filename == "video.mp4"

Service Tests (tests/unit/test_services.py)

import pytest
from unittest.mock import Mock, AsyncMock
from app.services.playlist_service import PlaylistService
from app.services.metube_client import MeTubeClient

class TestPlaylistService:
    @pytest.fixture
    def service(self, test_db):
        return PlaylistService(test_db)
    
    @pytest.fixture
    def mock_yt_dlp(self, monkeypatch):
        """Mock yt-dlp for testing"""
        mock = Mock()
        mock.extract_info.return_value = {
            "title": "Test Playlist",
            "entries": [
                {"id": "video1", "title": "Video 1", "playlist_index": 1},
                {"id": "video2", "title": "Video 2", "playlist_index": 2}
            ]
        }
        monkeypatch.setattr("yt_dlp.YoutubeDL", lambda x: mock)
        return mock
    
    async def test_add_playlist_success(self, service, mock_yt_dlp):
        """Test successful playlist addition"""
        playlist = await service.add_playlist(
            url="https://www.youtube.com/playlist?list=TEST123",
            title="Test Playlist"
        )
        
        assert playlist.title == "Test Playlist"
        assert playlist.check_interval == 60
        assert len(playlist.videos) == 2

Configuration Tests (tests/unit/test_config.py)

import pytest
from app.core.config import Settings

class TestConfiguration:
    def test_default_values(self):
        """Test default configuration values"""
        settings = Settings()
        assert settings.HOST == "0.0.0.0"
        assert settings.PORT == 8082
        assert settings.METUBE_URL == "http://localhost:8081"
    
    def test_validation(self):
        """Test configuration validation"""
        with pytest.raises(ValueError):
            Settings(DEFAULT_CHECK_INTERVAL=0)
        
        with pytest.raises(ValueError):
            Settings(MAX_CONCURRENT_DOWNLOADS=15)

🔌 Integration Tests

Database Integration (tests/integration/test_database.py)

import pytest
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from app.core.database import Base, get_db
from app.models.playlist import PlaylistSubscription

@pytest.fixture
def test_engine():
    """Create test database engine"""
    engine = create_engine("sqlite:///:memory:", echo=False)
    Base.metadata.create_all(bind=engine)
    return engine

class TestDatabaseIntegration:
    def test_playlist_crud_operations(self, test_engine):
        """Test database CRUD operations"""
        SessionLocal = sessionmaker(bind=test_engine)
        db = SessionLocal()
        
        # Create
        playlist = PlaylistSubscription(
            url="https://www.youtube.com/playlist?list=TEST123",
            title="Test Playlist"
        )
        db.add(playlist)
        db.commit()
        
        # Read
        retrieved = db.query(PlaylistSubscription).first()
        assert retrieved.title == "Test Playlist"
        
        # Update
        retrieved.title = "Updated Playlist"
        db.commit()
        assert db.query(PlaylistSubscription).first().title == "Updated Playlist"
        
        # Delete
        db.delete(retrieved)
        db.commit()
        assert db.query(PlaylistSubscription).count() == 0
        
        db.close()

MeTube Client Integration (tests/integration/test_metube_client.py)

import pytest
import respx
from httpx import Response
from app.services.metube_client import MeTubeClient

@pytest.mark.asyncio
class TestMeTubeClientIntegration:
    @respx.mock
    async def test_add_download_success(self):
        """Test successful download addition"""
        # Mock MeTube API
        route = respx.post("http://localhost:8081/add").mock(
            return_value=Response(200, json={"id": "download_123", "status": "pending"})
        )
        
        client = MeTubeClient("http://localhost:8081")
        await client.connect()
        
        result = await client.add_download(
            url="https://www.youtube.com/watch?v=TEST123",
            quality="best",
            format="mp4"
        )
        
        assert result["id"] == "download_123"
        assert result["status"] == "pending"
        assert route.called
        
        await client.disconnect()

🌐 API Tests

API Integration (tests/integration/test_api.py)

import pytest
from fastapi.testclient import TestClient
from app.main import app

class TestAPIIntegration:
    @pytest.fixture
    def client(self):
        return TestClient(app)
    
    def test_health_endpoint(self, client):
        """Test health check endpoint"""
        response = client.get("/health")
        assert response.status_code == 200
        data = response.json()
        assert data["status"] == "healthy"
    
    def test_playlist_crud(self, client):
        """Test playlist CRUD operations"""
        # Create playlist
        response = client.post("/api/playlists", json={
            "url": "https://www.youtube.com/playlist?list=TEST123",
            "title": "Test Playlist",
            "check_interval": 60
        })
        assert response.status_code == 201
        playlist_id = response.json()["id"]
        
        # Read playlist
        response = client.get(f"/api/playlists/{playlist_id}")
        assert response.status_code == 200
        assert response.json()["title"] == "Test Playlist"
        
        # Update playlist
        response = client.put(f"/api/playlists/{playlist_id}", json={
            "title": "Updated Playlist"
        })
        assert response.status_code == 200
        assert response.json()["title"] == "Updated Playlist"
        
        # Delete playlist
        response = client.delete(f"/api/playlists/{playlist_id}")
        assert response.status_code == 204
    
    def test_invalid_playlist_url(self, client):
        """Test validation of invalid playlist URLs"""
        response = client.post("/api/playlists", json={
            "url": "https://invalid-url.com",
            "title": "Invalid Playlist"
        })
        assert response.status_code == 400
        assert "Invalid YouTube playlist URL" in response.json()["detail"]

Async API Tests (tests/integration/test_api_async.py)

import pytest
import asyncio
from httpx import AsyncClient
from app.main import app

@pytest.mark.asyncio
class TestAsyncAPI:
    async def test_websocket_connection(self):
        """Test WebSocket connection and events"""
        # This would require a mock WebSocket server
        pass
    
    async def test_async_playlist_check(self):
        """Test async playlist checking"""
        async with AsyncClient(app=app, base_url="http://test") as client:
            # Create playlist
            response = await client.post("/api/playlists", json={
                "url": "https://www.youtube.com/playlist?list=TEST123",
                "title": "Test Playlist"
            })
            playlist_id = response.json()["id"]
            
            # Trigger check
            response = await client.post(f"/api/playlists/{playlist_id}/check")
            assert response.status_code == 200
            assert response.json()["status"] == "ok"

🎯 End-to-End Tests

Workflow Tests (tests/e2e/test_workflows.py)

import pytest
import asyncio
from httpx import AsyncClient
from app.main import app

@pytest.mark.asyncio
class TestE2EWorkflows:
    async def test_complete_playlist_workflow(self):
        """Test complete playlist lifecycle"""
        async with AsyncClient(app=app, base_url="http://test") as client:
            # 1. Create playlist
            response = await client.post("/api/playlists", json={
                "url": "https://www.youtube.com/playlist?list=PLrAXtmErZgOeiKm4sgNOknGvNjby9efdf",
                "title": "Kurzgesagt Playlist",
                "check_interval": 60,
                "folder": "kurzgesagt"
            })
            assert response.status_code == 201
            playlist_id = response.json()["id"]
            
            # 2. Verify playlist was created
            response = await client.get(f"/api/playlists/{playlist_id}")
            assert response.status_code == 200
            playlist = response.json()
            assert playlist["title"] == "Kurzgesagt Playlist"
            
            # 3. Check playlist (manual trigger)
            response = await client.post(f"/api/playlists/{playlist_id}/check")
            assert response.status_code == 200
            
            # 4. Verify videos were created
            response = await client.get(f"/api/playlists/{playlist_id}/videos")
            assert response.status_code == 200
            videos = response.json()
            assert len(videos) > 0
            
            # 5. Test video operations
            video_id = videos[0]["id"]
            
            # Skip a video
            response = await client.post(f"/api/videos/{video_id}/skip")
            assert response.status_code == 200
            assert response.json()["video"]["status"] == "SKIPPED"
            
            # Reset video
            response = await client.post(f"/api/videos/{video_id}/reset")
            assert response.status_code == 200
            assert response.json()["video"]["status"] == "PENDING"
            
            # 6. Check system status
            response = await client.get("/api/status")
            assert response.status_code == 200
            status = response.json()
            assert status["total_playlists"] >= 1
            assert status["total_videos"] >= 1
            
            # 7. Delete playlist
            response = await client.delete(f"/api/playlists/{playlist_id}")
            assert response.status_code == 204
    
    async def test_error_handling_workflow(self):
        """Test error handling scenarios"""
        async with AsyncClient(app=app, base_url="http://test") as client:
            # Test invalid playlist URL
            response = await client.post("/api/playlists", json={
                "url": "https://invalid-url.com",
                "title": "Invalid Playlist"
            })
            assert response.status_code == 400
            
            # Test non-existent playlist
            response = await client.get("/api/playlists/non-existent-id")
            assert response.status_code == 404
            
            # Test invalid video operations
            response = await client.post("/api/videos/non-existent/download")
            assert response.status_code == 404

⚡ Performance Tests

Load Testing (tests/performance/test_load.py)

import asyncio
import time
import aiohttp
import pytest

class TestLoadPerformance:
    @pytest.mark.performance
    async def test_api_response_time(self):
        """Test API response times"""
        async with aiohttp.ClientSession() as session:
            start_time = time.time()
            
            for i in range(100):
                async with session.get("http://localhost:8082/api/status") as response:
                    assert response.status == 200
            
            elapsed = time.time() - start_time
            avg_response_time = elapsed / 100
            
            # Assert average response time is under 100ms
            assert avg_response_time < 0.1
    
    @pytest.mark.performance
    async def test_database_performance(self):
        """Test database query performance"""
        from app.core.database import SessionLocal
        from app.models.playlist import PlaylistSubscription
        
        db = SessionLocal()
        
        # Create test data
        for i in range(1000):
            playlist = PlaylistSubscription(
                url=f"https://www.youtube.com/playlist?list=TEST{i}",
                title=f"Test Playlist {i}"
            )
            db.add(playlist)
        
        db.commit()
        
        # Measure query performance
        start_time = time.time()
        playlists = db.query(PlaylistSubscription).all()
        elapsed = time.time() - start_time
        
        assert len(playlists) == 1000
        assert elapsed < 1.0  # Should complete in under 1 second
        
        db.close()

Stress Testing (tests/performance/test_stress.py)

import asyncio
import pytest
from httpx import AsyncClient
from app.main import app

@pytest.mark.stress
class TestStressScenarios:
    async def test_concurrent_playlist_creation(self):
        """Test concurrent playlist creation"""
        async with AsyncClient(app=app, base_url="http://test") as client:
            # Create 50 playlists concurrently
            tasks = []
            for i in range(50):
                task = client.post("/api/playlists", json={
                    "url": f"https://www.youtube.com/playlist?list=TEST{i}",
                    "title": f"Stress Test Playlist {i}"
                })
                tasks.append(task)
            
            responses = await asyncio.gather(*tasks)
            
            # All should succeed
            for response in responses:
                assert response.status_code == 201
            
            # Verify all were created
            response = await client.get("/api/playlists")
            assert response.status_code == 200
            assert len(response.json()) >= 50

🔧 Manual Testing

Manual Test Checklist

Basic Functionality

• Service starts without errors
• Health endpoint returns 200
• API documentation accessible at /docs
• Database connection successful

Playlist Operations

• Create playlist with valid YouTube URL
• Create playlist with invalid URL (should fail)
• Update playlist settings
• Delete playlist
• List all playlists
• Get playlist with videos

Video Operations

• Videos automatically created from playlist
• Manual playlist check triggers new video detection
• Video status transitions work correctly
• Skip/reset operations work

Integration

• MeTube connection established
• Downloads triggered successfully
• WebSocket events received
• Status synchronization works

Error Handling

• Invalid URLs handled gracefully
• Database errors handled
• MeTube connection failures handled
• Proper error messages returned

Manual Testing Commands

Service Health

# Health check
curl http://localhost:8082/health

# System status
curl http://localhost:8082/api/status

# Scheduler status
curl http://localhost:8082/api/scheduler/status

Playlist Testing

# Create test playlist
curl -X POST http://localhost:8082/api/playlists \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.youtube.com/playlist?list=PLrAXtmErZgOeiKm4sgNOknGvNjby9efdf",
    "title": "Manual Test Playlist",
    "check_interval": 30
  }'

# Trigger manual check
curl -X POST http://localhost:8082/api/playlists/{id}/check

# Monitor logs
tail -f logs/playlist-monitor.log

🔄 CI/CD Integration

GitHub Actions (.github/workflows/tests.yml)

name: Tests

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]

jobs:
  test:
    runs-on: ubuntu-latest
    
    services:
      postgres:
        image: postgres:15
        env:
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: test_playlists
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5          
        ports:
          - 5432:5432
    
    strategy:
      matrix:
        python-version: [3.13, 3.14]
    
    steps:
    - uses: actions/checkout@v3
    
    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v4
      with:
        python-version: ${{ matrix.python-version }}
    
    - name: Install uv
      run: curl -LsSf https://astral.sh/uv/install.sh | sh
    
    - name: Install dependencies
      run: uv sync --extra dev
    
    - name: Run unit tests
      run: uv run pytest tests/unit/ -v --cov=app --cov-report=xml
    
    - name: Run integration tests
      run: uv run pytest tests/integration/ -v
    
    - name: Run performance tests
      run: uv run pytest tests/performance/ -v --tb=short
    
    - name: Upload coverage reports
      uses: codecov/codecov-action@v3
      with:
        file: ./coverage.xml

Pre-commit Hooks (.pre-commit-config.yaml)

repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.4.0
    hooks:
      - id: trailing-whitespace
      - id: end-of-file-fixer
      - id: check-yaml
      - id: check-added-large-files
  
  - repo: https://github.com/psf/black
    rev: 23.3.0
    hooks:
      - id: black
        language_version: python3.13
  
  - repo: https://github.com/pycqa/isort
    rev: 5.12.0
    hooks:
      - id: isort
  
  - repo: https://github.com/pycqa/flake8
    rev: 6.0.0
    hooks:
      - id: flake8
  
  - repo: https://github.com/pre-commit/mirrors-mypy
    rev: v1.3.0
    hooks:
      - id: mypy
        additional_dependencies: [types-all]

📊 Test Data Management

Test Fixtures (tests/fixtures/)

# tests/fixtures/playlists.py
TEST_PLAYLISTS = [
    {
        "url": "https://www.youtube.com/playlist?list=PLrAXtmErZgOeiKm4sgNOknGvNjby9efdf",
        "title": "Kurzgesagt – In a Nutshell",
        "check_interval": 60,
        "quality": "best",
        "format": "mp4",
        "folder": "kurzgesagt"
    },
    {
        "url": "https://www.youtube.com/playlist?list=UUX6OQ3DkcsbYNE6H8uQQuVA",
        "title": "Tech Channel Uploads",
        "check_interval": 30,
        "quality": "1080p",
        "format": "mp4",
        "folder": "tech-channels"
    }
]

# tests/fixtures/videos.py
TEST_VIDEOS = [
    {
        "video_id": "dQw4w9WgXcQ",
        "title": "Never Gonna Give You Up",
        "playlist_index": 1,
        "status": "COMPLETED"
    },
    {
        "video_id": "9bZkp7q19f0",
        "title": "Gangnam Style",
        "playlist_index": 2,
        "status": "PENDING"
    }
]

Mock Data Generation

# tests/utils/mock_data.py
def generate_mock_playlists(count=10):
    """Generate mock playlist data"""
    playlists = []
    for i in range(count):
        playlists.append({
            "url": f"https://www.youtube.com/playlist?list=TEST{i:03d}",
            "title": f"Test Playlist {i}",
            "check_interval": 60,
            "quality": "best",
            "format": "mp4",
            "folder": f"test-folder-{i}"
        })
    return playlists

def generate_mock_videos(playlist_id, count=20):
    """Generate mock video data"""
    videos = []
    for i in range(count):
        videos.append({
            "playlist_id": playlist_id,
            "video_url": f"https://www.youtube.com/watch?v=VIDEO{i:03d}",
            "video_id": f"VIDEO{i:03d}",
            "title": f"Test Video {i}",
            "playlist_index": i + 1,
            "status": "PENDING"
        })
    return videos

📈 Test Coverage

Coverage Report Generation

# Generate coverage report
uv run pytest --cov=app --cov-report=html --cov-report=term

# View HTML report
open htmlcov/index.html

# Generate XML report for CI
uv run pytest --cov=app --cov-report=xml

Coverage Goals

• Unit Tests: >90% coverage
• Integration Tests: >80% coverage
• API Tests: >95% coverage
• Overall: >85% coverage

Coverage Configuration (pyproject.toml)

[tool.coverage.run]
source = ["app"]
omit = [
    "*/tests/*",
    "*/venv/*",
    "*/__pycache__/*",
    "app/main.py",  # Entry point
]

[tool.coverage.report]
exclude_lines = [
    "pragma: no cover",
    "def __repr__",
    "raise AssertionError",
    "raise NotImplementedError",
]

[tool.coverage.html]
directory = "htmlcov"

🎯 Test Execution Strategies

Parallel Testing

# Run tests in parallel
uv run pytest -n auto

# Run specific test categories in parallel
uv run pytest tests/unit/ -n 4 &
uv run pytest tests/integration/ -n 2 &
wait

Test Selection

# Run tests by marker
uv run pytest -m "not performance"
uv run pytest -m "performance"

# Run tests by name pattern
uv run pytest -k "test_playlist"
uv run pytest -k "not test_performance"

# Run specific test file
uv run pytest tests/unit/test_models.py

Continuous Testing

# Watch mode for development
uv run pytest-watch tests/unit/

# Run on file changes
uv run ptw -- tests/unit/

📚 Testing Best Practices

1. Test Naming

def test_playlist_creation_with_valid_url(self):
    """Should create playlist when valid YouTube URL provided"""
    # Test implementation

def test_playlist_creation_fails_with_invalid_url(self):
    """Should reject playlist creation with invalid URL"""
    # Test implementation

2. Test Independence

@pytest.fixture
def clean_database():
    """Provide clean database for each test"""
    # Setup clean state
    yield
    # Cleanup after test

3. Mock External Services

@pytest.fixture
def mock_metube():
    """Mock MeTube service for testing"""
    with respx.mock:
        respx.get("http://localhost:8081/info").mock(
            return_value=Response(200, json={"status": "ok"})
        )
        yield

4. Test Data Management

@pytest.fixture
def sample_playlist():
    """Provide sample playlist data"""
    return {
        "url": "https://www.youtube.com/playlist?list=TEST123",
        "title": "Test Playlist",
        "check_interval": 60
    }

This comprehensive testing guide ensures robust, reliable testing of the Playlist Monitor Service! 🧪✅