24 KiB

Raw Permalink Blame History

Playlist Monitor Service - Architecture & Design Document

Executive Summary

This document outlines the architecture for a Playlist Monitor Service that extends MeTube's capabilities by adding automated playlist monitoring, periodic checking, and intelligent download management. The service will monitor YouTube playlists, track which videos have been downloaded, and automatically download new videos using MeTube as the download engine.

1. Current MeTube Capabilities (Base Service)

1.1 Core Features

Video Download Engine: Uses yt-dlp to download videos from YouTube and 100+ sites
REST API:
- POST /add - Add download with parameters (url, quality, format, folder, etc.)
- POST /delete - Cancel/clear downloads
- POST /start - Start pending downloads
- GET /history - Get download history
WebSocket Events: Real-time updates (added, updated, completed, canceled, cleared)
Queue Management:
- Sequential, concurrent, or limited concurrent download modes
- Persistent queue storage using shelve
- Pending, active, and completed download tracking
Cookie Support: Authentication via cookie files stored in STATE_DIR/cookies/
Playlist Support: Can download entire playlists with item limits
Custom Output Templates: Flexible file naming and directory structure

1.2 Key Technical Components

Backend: Python 3.13, aiohttp, socketio, yt-dlp
Frontend: Angular, TypeScript
Storage: Shelve-based persistent queues

Download Info Structure:

DownloadInfo(id, title, url, quality, format, folder, 
             custom_name_prefix, error, entry, playlist_item_limit)

1.3 API Integration Points

# Add download to MeTube
POST /add
{
  "url": "https://youtube.com/watch?v=...",
  "quality": "best",
  "format": "mp4",
  "folder": "playlist_name",
  "auto_start": true
}

2. New Playlist Monitor Service - Architecture

2.1 Service Overview

The Playlist Monitor Service is a separate microservice that:

Manages playlist subscriptions
Periodically checks playlists for new videos
Tracks download status of each video
Delegates actual downloads to MeTube
Maintains persistent state across restarts

2.2 High-Level Architecture

┌─────────────────────────────────────────────────────────────┐
│                    User Interface (Web)                      │
│  - Add/Remove Playlists                                     │
│  - Configure Check Intervals                                │
│  - Set Start Points                                         │
│  - View Video Status                                        │
│  - Manual Re-download                                       │
└────────────────────┬────────────────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────────────────────┐
│           Playlist Monitor Service (REST API)                │
│  ┌──────────────────┐  ┌─────────────────┐                 │
│  │ Playlist Manager │  │ Video Tracker   │                 │
│  └──────────────────┘  └─────────────────┘                 │
│  ┌──────────────────┐  ┌─────────────────┐                 │
│  │ Scheduler Engine │  │ State Manager   │                 │
│  └──────────────────┘  └─────────────────┘                 │
└────────────────────┬────────────────────┬───────────────────┘
                     │                    │
                     ▼                    ▼
         ┌───────────────────┐  ┌──────────────────┐
         │  MeTube Service   │  │ Database/Storage │
         │  (Download Engine)│  │ (SQLite/Shelve)  │
         └───────────────────┘  └──────────────────┘
                     │
                     ▼
         ┌───────────────────┐
         │  Download Files   │
         │  (Filesystem)     │
         └───────────────────┘

2.3 Component Breakdown

2.3.1 Playlist Manager

Responsibilities:
- Add/remove/update playlist subscriptions
- Configure playlist-specific settings (check interval, quality, format, folder)
- Fetch playlist metadata using yt-dlp
- Extract video list from playlists

Data Model:

class PlaylistSubscription:
    id: str                    # Unique playlist ID
    url: str                   # Playlist URL
    title: str                 # Playlist title
    check_interval: int        # Minutes between checks
    last_checked: datetime     # Last check timestamp
    start_point: str           # Video ID/index to start from
    quality: str               # Download quality (best, 1080, etc.)
    format: str                # Download format (mp4, any, etc.)
    folder: str                # Download folder
    enabled: bool              # Active/paused
    created_at: datetime
    updated_at: datetime

2.3.2 Video Tracker

Responsibilities:
- Track status of each video in each playlist
- Persist video metadata and download status
- Handle file movement tracking (detached from filesystem)
- Prevent re-downloads unless explicitly requested

Data Model:

class VideoRecord:
    id: str                    # Unique video ID
    playlist_id: str           # Foreign key to playlist
    video_url: str             # Direct video URL
    video_id: str              # YouTube video ID
    title: str                 # Video title
    playlist_index: int        # Position in playlist
    upload_date: datetime      # Video upload date

    # Download tracking
    status: VideoStatus        # PENDING, DOWNLOADING, COMPLETED, FAILED, SKIPPED
    download_requested_at: datetime
    download_completed_at: datetime
    metube_download_id: str    # Reference to MeTube download

    # File tracking (decoupled from actual file)
    original_filename: str     # Filename when downloaded
    file_moved: bool           # Whether user moved the file
    file_location_note: str    # Optional note about file location

    # Error handling
    error_message: str
    retry_count: int
    last_error_at: datetime

    created_at: datetime
    updated_at: datetime

enum VideoStatus:
    PENDING       # Not yet downloaded
    DOWNLOADING   # Currently being downloaded
    COMPLETED     # Successfully downloaded
    FAILED        # Download failed
    SKIPPED       # Before start_point or manually skipped

2.3.3 Scheduler Engine

Responsibilities:
- Periodic task execution (check playlists)
- Background job queue management
- Rate limiting and retry logic
- Health monitoring
Implementation Options:
- Option A: APScheduler (Python async scheduler)
- Option B: Celery + Redis (production-grade)
- Recommended: APScheduler for simplicity

Tasks:

# Scheduled Tasks
1. check_playlist(playlist_id)     # Check single playlist for new videos
2. check_all_playlists()           # Check all enabled playlists
3. retry_failed_downloads()        # Retry failed downloads
4. cleanup_old_records()           # Archive old data
5. sync_metube_status()            # Sync status from MeTube

2.3.4 State Manager

Responsibilities:
- Persistent storage of playlists and videos
- Database migrations
- Data backup and recovery
- Import/export functionality
Storage Options:
- Option A: SQLite (recommended for simplicity)
- Option B: PostgreSQL (for scalability)
- Option C: Shelve (consistency with MeTube)
Recommended: SQLite with SQLAlchemy ORM

2.3.5 MeTube Client

Responsibilities:
- HTTP client to communicate with MeTube API
- WebSocket client to receive real-time updates
- Status synchronization
- Error handling and retry logic

Implementation:

class MeTubeClient:
    def __init__(self, base_url: str):
        self.base_url = base_url
        self.session = aiohttp.ClientSession()

    async def add_download(self, url, quality, format, folder):
        response = await self.session.post(
            f"{self.base_url}/add",
            json={
                "url": url,
                "quality": quality,
                "format": format,
                "folder": folder,
                "auto_start": True
            }
        )
        return await response.json()

    async def get_history(self):
        response = await self.session.get(f"{self.base_url}/history")
        return await response.json()

    async def listen_to_events(self, callback):
        # WebSocket connection to receive updates
        sio = socketio.AsyncClient()

        @sio.on('completed')
        async def on_completed(data):
            await callback('completed', data)

        @sio.on('updated')
        async def on_updated(data):
            await callback('updated', data)

        await sio.connect(self.base_url)

3. Core Workflows

3.1 Add Playlist Workflow

User → Add Playlist (URL, settings)
  ↓
Validate URL & extract playlist info via yt-dlp
  ↓
Create PlaylistSubscription record
  ↓
Fetch current video list
  ↓
For each video:
  - Create VideoRecord
  - If before start_point: status = SKIPPED
  - If after start_point: status = PENDING
  ↓
Schedule periodic check job
  ↓
Return playlist info to user

3.2 Periodic Check Workflow

Scheduler → Trigger check_playlist(playlist_id)
  ↓
Fetch latest video list from YouTube (via yt-dlp)
  ↓
Compare with existing VideoRecords
  ↓
For each new video:
  - Check if after start_point
  - Create VideoRecord with status=PENDING
  ↓
For each PENDING video (respecting order):
  - Send download request to MeTube
  - Update status to DOWNLOADING
  - Store metube_download_id
  ↓
Update playlist.last_checked timestamp

3.3 Download Completion Workflow

MeTube → WebSocket event: 'completed'
  ↓
Extract video URL from event
  ↓
Find VideoRecord by metube_download_id or video_url
  ↓
Update VideoRecord:
  - status = COMPLETED
  - download_completed_at = now()
  - original_filename = filename from MeTube
  ↓
Log completion

3.4 Manual Re-download Workflow

User → Request re-download for video_id
  ↓
Find VideoRecord
  ↓
Reset status to PENDING
  ↓
Clear error fields
  ↓
Send download request to MeTube
  ↓
Update status to DOWNLOADING

3.5 File Movement Handling

User moves file manually (outside app)
  ↓
User marks video as "file moved" in UI
  ↓
Update VideoRecord:
  - file_moved = True
  - file_location_note = optional note
  ↓
Status remains COMPLETED (prevents re-download)

4. API Design

4.1 Playlist Endpoints

# List all playlists
GET /api/playlists
Response: { playlists: [PlaylistSubscription, ...] }

# Get single playlist with videos
GET /api/playlists/{playlist_id}
Response: { 
  playlist: PlaylistSubscription,
  videos: [VideoRecord, ...],
  stats: { total, pending, completed, failed }
}

# Add new playlist
POST /api/playlists
Body: {
  url: string,
  check_interval: int (default: 60),
  start_point: string (video_id or index),
  quality: string (default: "best"),
  format: string (default: "mp4"),
  folder: string
}
Response: PlaylistSubscription

# Update playlist
PUT /api/playlists/{playlist_id}
Body: { check_interval, quality, format, enabled, ... }
Response: PlaylistSubscription

# Delete playlist
DELETE /api/playlists/{playlist_id}?delete_videos=true
Response: { status: "ok" }

# Trigger manual check
POST /api/playlists/{playlist_id}/check
Response: { new_videos: int, status: "ok" }

# Update start point
POST /api/playlists/{playlist_id}/start-point
Body: { video_id: string }
Response: { updated_videos: int }

4.2 Video Endpoints

# List videos for playlist
GET /api/playlists/{playlist_id}/videos?status=PENDING&limit=50&offset=0
Response: { videos: [VideoRecord, ...], total: int }

# Get single video
GET /api/videos/{video_id}
Response: VideoRecord

# Request download/re-download
POST /api/videos/{video_id}/download
Response: { status: "ok", metube_id: string }

# Mark file as moved
POST /api/videos/{video_id}/file-moved
Body: { location_note: string }
Response: VideoRecord

# Skip video (mark as SKIPPED)
POST /api/videos/{video_id}/skip
Response: VideoRecord

# Reset video (back to PENDING)
POST /api/videos/{video_id}/reset
Response: VideoRecord

4.3 System Endpoints

# Get system status
GET /api/status
Response: {
  total_playlists: int,
  active_playlists: int,
  total_videos: int,
  pending_downloads: int,
  active_downloads: int,
  completed_downloads: int,
  failed_downloads: int,
  metube_status: { connected: bool, version: string }
}

# Get scheduler status
GET /api/scheduler/status
Response: {
  jobs: [{ id, playlist_id, next_run, ... }],
  running: bool
}

# Trigger manual sync with MeTube
POST /api/sync-metube
Response: { synced_videos: int, status: "ok" }

5. Database Schema (SQLite)

-- Playlists
CREATE TABLE playlists (
    id TEXT PRIMARY KEY,
    url TEXT NOT NULL UNIQUE,
    title TEXT,
    check_interval INTEGER DEFAULT 60,
    last_checked TIMESTAMP,
    start_point TEXT,
    quality TEXT DEFAULT 'best',
    format TEXT DEFAULT 'mp4',
    folder TEXT,
    enabled BOOLEAN DEFAULT 1,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Videos
CREATE TABLE videos (
    id TEXT PRIMARY KEY,
    playlist_id TEXT NOT NULL,
    video_url TEXT NOT NULL,
    video_id TEXT NOT NULL,
    title TEXT,
    playlist_index INTEGER,
    upload_date TIMESTAMP,
    
    status TEXT DEFAULT 'PENDING',
    download_requested_at TIMESTAMP,
    download_completed_at TIMESTAMP,
    metube_download_id TEXT,
    
    original_filename TEXT,
    file_moved BOOLEAN DEFAULT 0,
    file_location_note TEXT,
    
    error_message TEXT,
    retry_count INTEGER DEFAULT 0,
    last_error_at TIMESTAMP,
    
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    
    FOREIGN KEY (playlist_id) REFERENCES playlists(id) ON DELETE CASCADE
);

-- Indexes
CREATE INDEX idx_videos_playlist_id ON videos(playlist_id);
CREATE INDEX idx_videos_status ON videos(status);
CREATE INDEX idx_videos_video_id ON videos(video_id);
CREATE INDEX idx_videos_playlist_index ON videos(playlist_id, playlist_index);

-- Activity Log (optional, for audit trail)
CREATE TABLE activity_log (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    event_type TEXT,  -- playlist_added, video_downloaded, check_completed, etc.
    playlist_id TEXT,
    video_id TEXT,
    details TEXT,  -- JSON blob
    FOREIGN KEY (playlist_id) REFERENCES playlists(id) ON DELETE CASCADE,
    FOREIGN KEY (video_id) REFERENCES videos(id) ON DELETE CASCADE
);

CREATE INDEX idx_activity_log_timestamp ON activity_log(timestamp);
CREATE INDEX idx_activity_log_event_type ON activity_log(event_type);

6. Technology Stack

6.1 Backend

Framework: FastAPI (Python 3.13+)
- Modern async support
- Auto-generated OpenAPI docs
- WebSocket support
- Dependency injection
Database: SQLAlchemy + SQLite
- ORM for easy data modeling
- Migration support with Alembic
Scheduler: APScheduler
- Async job scheduling
- Persistent job storage
- Cron-like intervals
MeTube Integration:
- aiohttp (HTTP client)
- python-socketio (WebSocket client)

Dependencies:

fastapi
uvicorn[standard]
sqlalchemy
alembic
apscheduler
aiohttp
python-socketio[client]
yt-dlp
pydantic
python-dotenv

6.2 Frontend (Optional - can reuse MeTube's Angular)

Option A: Extend MeTube's Angular UI
- Add new routes/components for playlists
- Integrate with existing UI
Option B: Separate Vue.js/React SPA
- Independent frontend
- Modern UI framework
Recommended: Extend MeTube's Angular for consistency

6.3 Deployment

Docker Compose:

services:
  metube:
    image: ghcr.io/alexta69/metube
    ports: ["8081:8081"]
    volumes:
      - ./downloads:/downloads

  playlist-monitor:
    build: ./playlist-monitor
    ports: ["8082:8082"]
    environment:
      - METUBE_URL=http://metube:8081
      - DATABASE_URL=sqlite:///data/playlists.db
      - CHECK_INTERVAL=60
    volumes:
      - ./monitor-data:/data
    depends_on:
      - metube

7. Configuration

# config.yaml
metube:
  url: http://localhost:8081
  reconnect_interval: 5  # seconds

database:
  url: sqlite:///data/playlists.db
  echo: false

scheduler:
  enabled: true
  default_check_interval: 60  # minutes
  max_concurrent_downloads: 3
  retry_failed_after: 24  # hours

downloads:
  default_quality: best
  default_format: mp4
  default_folder: playlists/{playlist_title}

logging:
  level: INFO
  file: logs/playlist-monitor.log

server:
  host: 0.0.0.0
  port: 8082

8. Implementation Plan (Phases)

Phase 1: Core Infrastructure (Week 1-2)

Set up FastAPI project structure
Database schema and SQLAlchemy models
Basic CRUD operations for playlists
MeTube client implementation
Configuration management

Phase 2: Playlist Management (Week 2-3)

Add playlist endpoint (with yt-dlp integration)
Fetch and parse playlist videos
Implement start_point logic
Basic video tracking

Phase 3: Scheduler & Automation (Week 3-4)

APScheduler integration
Periodic check implementation
Download triggering logic
Status synchronization with MeTube

Phase 4: Advanced Features (Week 4-5)

File movement tracking
Manual re-download
Error handling and retry logic
Activity logging

Phase 5: UI Integration (Week 5-6)

REST API documentation (OpenAPI)
Angular components (if extending MeTube UI)
Playlist listing and details view
Video status dashboard

Phase 6: Testing & Deployment (Week 6-7)

Unit tests
Integration tests
Docker containerization
Documentation and user guide

9. Key Design Decisions

9.1 Why Separate Service?

Modularity: Keep concerns separated
Independence: Can run without modifying MeTube core
Scalability: Can scale independently
Maintenance: Easier to update/maintain

9.2 Why SQLite?

Simplicity: No external DB server needed
Portability: Single file database
Sufficient: Adequate for single-user/small-team use
Upgrade Path: Can migrate to PostgreSQL if needed

9.3 File Movement Tracking

Decoupled Design: Don't check filesystem, rely on metadata
User Control: User explicitly marks files as moved
Prevents Re-downloads: Once marked completed, won't re-download
Flexibility: Files can be moved/organized freely

9.4 Start Point Implementation

Options:
- Video ID (specific video)
- Playlist index (position number)
- Upload date (date cutoff)
Recommended: Video ID (most reliable)
Behavior: Videos before start point marked as SKIPPED

10. Future Enhancements

10.1 Advanced Features

Smart Download Scheduling: Download during off-peak hours
Bandwidth Management: Rate limiting per playlist
Multi-platform Support: Support non-YouTube playlists
Notification System: Email/webhook on new videos
Archive Mode: Download only specific date ranges
Duplicate Detection: Prevent duplicate downloads across playlists

10.2 UI Enhancements

Timeline View: Visual timeline of uploads
Bulk Operations: Batch skip/reset/download
Statistics Dashboard: Charts and graphs
Search & Filters: Advanced video filtering

10.3 Integration Features

Plex/Jellyfin Integration: Auto-update media libraries
RSS Feed: Generate RSS feeds for playlists
API Webhooks: Notify external systems
Cloud Sync: Backup/sync across instances

11. Security Considerations

11.1 Authentication

Implement API key or JWT authentication
Integrate with MeTube's auth (if available)
Rate limiting to prevent abuse

11.2 Data Privacy

Encrypt sensitive data (cookies, tokens)
Regular security audits
HTTPS only in production

11.3 Resource Management

Limit number of playlists per instance
Limit number of videos per playlist
Disk space monitoring

12. Monitoring & Observability

12.1 Metrics

Playlist check frequency
Download success/failure rates
MeTube API response times
Database query performance

12.2 Logging

Structured logging (JSON)
Log levels: DEBUG, INFO, WARNING, ERROR
Log rotation and retention

12.3 Health Checks

Database connectivity
MeTube connectivity
Scheduler status
Disk space availability

13. Summary

This architecture provides a robust, scalable, and maintainable solution for automated playlist monitoring that:

✅ Integrates seamlessly with MeTube via REST API and WebSocket
✅ Tracks video status independently of filesystem
✅ Prevents re-downloads even when files are moved
✅ Supports flexible start points for granular control
✅ Provides periodic scheduling with configurable intervals
✅ Maintains persistent state across restarts
✅ Offers comprehensive API for programmatic control
✅ Scales gracefully with proper database design

The service is designed to be deployed alongside MeTube as a companion microservice, leveraging MeTube's powerful download capabilities while adding intelligent playlist monitoring on top.

Appendix A: Example Use Cases

Use Case 1: YouTube Channel Monitoring

Scenario: Monitor a YouTube channel for new uploads
Setup:

Add channel's "Uploads" playlist
Set start_point to latest video
Check interval: 30 minutes
All new videos downloaded automatically

Use Case 2: Partial Playlist Download

Scenario: Download only recent videos from a large playlist
Setup:

Add playlist URL
Set start_point to video from 6 months ago
Older videos marked as SKIPPED
New videos downloaded as they appear

Use Case 3: Archive & Organize

Scenario: Download and organize by playlist
Setup:

Multiple playlists with custom folders
Quality: best
Format: mp4
After download, move files to NAS
Mark as "file moved" to prevent re-download

Appendix B: API Examples

Adding a Playlist

curl -X POST http://localhost:8082/api/playlists \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.youtube.com/playlist?list=PLxxxxxx",
    "check_interval": 60,
    "start_point": "dQw4w9WgXcQ",
    "quality": "1080",
    "format": "mp4",
    "folder": "my-playlist"
  }'

Listing Videos

curl http://localhost:8082/api/playlists/{playlist_id}/videos?status=PENDING&limit=20

Marking File as Moved

curl -X POST http://localhost:8082/api/videos/{video_id}/file-moved \
  -H "Content-Type: application/json" \
  -d '{
    "location_note": "Moved to /mnt/nas/videos/"
  }'

Document Version: 1.0
Last Updated: 2025-01-19
Author: AI Architecture Team
Status: Draft for Review

24 KiB Raw Permalink Blame History