xorbitlab
/
tubewatch
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

docs(deploy): update image tag and add compose deployment command 

- Changed docker image tag from 1.1 to 1.3 in build and push commands
- Added `docker compose up -d --build --force-recreate` command for deployment
This commit is contained in:
tigeren 2025-11-19 17:21:21 +00:00
parent f6a9e6a1dd
commit 06a7277554
2 changed files with 852 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
DEPLOY.md
View File

@ -1,3 +1,5 @@
docker build -t 192.168.2.212:3000/tigeren/metube:1.1 . docker build -t 192.168.2.212:3000/tigeren/metube:1.3 .
docker push 192.168.2.212:3000/tigeren/metube:1.1 docker push 192.168.2.212:3000/tigeren/metube:1.3
docker compose up -d --build --force-recreate

848
PLAYLIST_MONITOR_ARCHITECTURE.md Normal file
View File

@ -0,0 +1,848 @@
# Playlist Monitor Service - Architecture & Design Document
## Executive Summary
This document outlines the architecture for a **Playlist Monitor Service** that extends MeTube's capabilities by adding automated playlist monitoring, periodic checking, and intelligent download management. The service will monitor YouTube playlists, track which videos have been downloaded, and automatically download new videos using MeTube as the download engine.
---
## 1. Current MeTube Capabilities (Base Service)
### 1.1 Core Features
- **Video Download Engine**: Uses yt-dlp to download videos from YouTube and 100+ sites
- **REST API**:
- `POST /add` - Add download with parameters (url, quality, format, folder, etc.)
- `POST /delete` - Cancel/clear downloads
- `POST /start` - Start pending downloads
- `GET /history` - Get download history
- **WebSocket Events**: Real-time updates (added, updated, completed, canceled, cleared)
- **Queue Management**:
- Sequential, concurrent, or limited concurrent download modes
- Persistent queue storage using shelve
- Pending, active, and completed download tracking
- **Cookie Support**: Authentication via cookie files stored in `STATE_DIR/cookies/`
- **Playlist Support**: Can download entire playlists with item limits
- **Custom Output Templates**: Flexible file naming and directory structure
### 1.2 Key Technical Components
- **Backend**: Python 3.13, aiohttp, socketio, yt-dlp
- **Frontend**: Angular, TypeScript
- **Storage**: Shelve-based persistent queues
- **Download Info Structure**:
```python
DownloadInfo(id, title, url, quality, format, folder,
custom_name_prefix, error, entry, playlist_item_limit)
```
### 1.3 API Integration Points
```python
# Add download to MeTube
POST /add
{
"url": "https://youtube.com/watch?v=...",
"quality": "best",
"format": "mp4",
"folder": "playlist_name",
"auto_start": true
}
```
---
## 2. New Playlist Monitor Service - Architecture
### 2.1 Service Overview
The Playlist Monitor Service is a **separate microservice** that:
1. Manages playlist subscriptions
2. Periodically checks playlists for new videos
3. Tracks download status of each video
4. Delegates actual downloads to MeTube
5. Maintains persistent state across restarts
### 2.2 High-Level Architecture
```
┌─────────────────────────────────────────────────────────────┐
│ User Interface (Web) │
│ - Add/Remove Playlists │
│ - Configure Check Intervals │
│ - Set Start Points │
│ - View Video Status │
│ - Manual Re-download │
└────────────────────┬────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Playlist Monitor Service (REST API) │
│ ┌──────────────────┐ ┌─────────────────┐ │
│ │ Playlist Manager │ │ Video Tracker │ │
│ └──────────────────┘ └─────────────────┘ │
│ ┌──────────────────┐ ┌─────────────────┐ │
│ │ Scheduler Engine │ │ State Manager │ │
│ └──────────────────┘ └─────────────────┘ │
└────────────────────┬────────────────────┬───────────────────┘
│ │
▼ ▼
┌───────────────────┐ ┌──────────────────┐
│ MeTube Service │ │ Database/Storage │
│ (Download Engine)│ │ (SQLite/Shelve) │
└───────────────────┘ └──────────────────┘
│
▼
┌───────────────────┐
│ Download Files │
│ (Filesystem) │
└───────────────────┘
```
### 2.3 Component Breakdown
#### **2.3.1 Playlist Manager**
- **Responsibilities**:
- Add/remove/update playlist subscriptions
- Configure playlist-specific settings (check interval, quality, format, folder)
- Fetch playlist metadata using yt-dlp
- Extract video list from playlists
- **Data Model**:
```python
class PlaylistSubscription:
id: str # Unique playlist ID
url: str # Playlist URL
title: str # Playlist title
check_interval: int # Minutes between checks
last_checked: datetime # Last check timestamp
start_point: str # Video ID/index to start from
quality: str # Download quality (best, 1080, etc.)
format: str # Download format (mp4, any, etc.)
folder: str # Download folder
enabled: bool # Active/paused
created_at: datetime
updated_at: datetime
```
#### **2.3.2 Video Tracker**
- **Responsibilities**:
- Track status of each video in each playlist
- Persist video metadata and download status
- Handle file movement tracking (detached from filesystem)
- Prevent re-downloads unless explicitly requested
- **Data Model**:
```python
class VideoRecord:
id: str # Unique video ID
playlist_id: str # Foreign key to playlist
video_url: str # Direct video URL
video_id: str # YouTube video ID
title: str # Video title
playlist_index: int # Position in playlist
upload_date: datetime # Video upload date
# Download tracking
status: VideoStatus # PENDING, DOWNLOADING, COMPLETED, FAILED, SKIPPED
download_requested_at: datetime
download_completed_at: datetime
metube_download_id: str # Reference to MeTube download
# File tracking (decoupled from actual file)
original_filename: str # Filename when downloaded
file_moved: bool # Whether user moved the file
file_location_note: str # Optional note about file location
# Error handling
error_message: str
retry_count: int
last_error_at: datetime
created_at: datetime
updated_at: datetime
enum VideoStatus:
PENDING # Not yet downloaded
DOWNLOADING # Currently being downloaded
COMPLETED # Successfully downloaded
FAILED # Download failed
SKIPPED # Before start_point or manually skipped
```
#### **2.3.3 Scheduler Engine**
- **Responsibilities**:
- Periodic task execution (check playlists)
- Background job queue management
- Rate limiting and retry logic
- Health monitoring
- **Implementation Options**:
- **Option A**: APScheduler (Python async scheduler)
- **Option B**: Celery + Redis (production-grade)
- **Recommended**: APScheduler for simplicity
- **Tasks**:
```python
# Scheduled Tasks
1. check_playlist(playlist_id) # Check single playlist for new videos
2. check_all_playlists() # Check all enabled playlists
3. retry_failed_downloads() # Retry failed downloads
4. cleanup_old_records() # Archive old data
5. sync_metube_status() # Sync status from MeTube
```
#### **2.3.4 State Manager**
- **Responsibilities**:
- Persistent storage of playlists and videos
- Database migrations
- Data backup and recovery
- Import/export functionality
- **Storage Options**:
- **Option A**: SQLite (recommended for simplicity)
- **Option B**: PostgreSQL (for scalability)
- **Option C**: Shelve (consistency with MeTube)
- **Recommended**: SQLite with SQLAlchemy ORM
#### **2.3.5 MeTube Client**
- **Responsibilities**:
- HTTP client to communicate with MeTube API
- WebSocket client to receive real-time updates
- Status synchronization
- Error handling and retry logic
- **Implementation**:
```python
class MeTubeClient:
def __init__(self, base_url: str):
self.base_url = base_url
self.session = aiohttp.ClientSession()
async def add_download(self, url, quality, format, folder):
response = await self.session.post(
f"{self.base_url}/add",
json={
"url": url,
"quality": quality,
"format": format,
"folder": folder,
"auto_start": True
}
)
return await response.json()
async def get_history(self):
response = await self.session.get(f"{self.base_url}/history")
return await response.json()
async def listen_to_events(self, callback):
# WebSocket connection to receive updates
sio = socketio.AsyncClient()
@sio.on('completed')
async def on_completed(data):
await callback('completed', data)
@sio.on('updated')
async def on_updated(data):
await callback('updated', data)
await sio.connect(self.base_url)
```
---
## 3. Core Workflows
### 3.1 Add Playlist Workflow
```
User → Add Playlist (URL, settings)
↓
Validate URL & extract playlist info via yt-dlp
↓
Create PlaylistSubscription record
↓
Fetch current video list
↓
For each video:
- Create VideoRecord
- If before start_point: status = SKIPPED
- If after start_point: status = PENDING
↓
Schedule periodic check job
↓
Return playlist info to user
```
### 3.2 Periodic Check Workflow
```
Scheduler → Trigger check_playlist(playlist_id)
↓
Fetch latest video list from YouTube (via yt-dlp)
↓
Compare with existing VideoRecords
↓
For each new video:
- Check if after start_point
- Create VideoRecord with status=PENDING
↓
For each PENDING video (respecting order):
- Send download request to MeTube
- Update status to DOWNLOADING
- Store metube_download_id
↓
Update playlist.last_checked timestamp
```
### 3.3 Download Completion Workflow
```
MeTube → WebSocket event: 'completed'
↓
Extract video URL from event
↓
Find VideoRecord by metube_download_id or video_url
↓
Update VideoRecord:
- status = COMPLETED
- download_completed_at = now()
- original_filename = filename from MeTube
↓
Log completion
```
### 3.4 Manual Re-download Workflow
```
User → Request re-download for video_id
↓
Find VideoRecord
↓
Reset status to PENDING
↓
Clear error fields
↓
Send download request to MeTube
↓
Update status to DOWNLOADING
```
### 3.5 File Movement Handling
```
User moves file manually (outside app)
↓
User marks video as "file moved" in UI
↓
Update VideoRecord:
- file_moved = True
- file_location_note = optional note
↓
Status remains COMPLETED (prevents re-download)
```
---
## 4. API Design
### 4.1 Playlist Endpoints
```
# List all playlists
GET /api/playlists
Response: { playlists: [PlaylistSubscription, ...] }
# Get single playlist with videos
GET /api/playlists/{playlist_id}
Response: {
playlist: PlaylistSubscription,
videos: [VideoRecord, ...],
stats: { total, pending, completed, failed }
}
# Add new playlist
POST /api/playlists
Body: {
url: string,
check_interval: int (default: 60),
start_point: string (video_id or index),
quality: string (default: "best"),
format: string (default: "mp4"),
folder: string
}
Response: PlaylistSubscription
# Update playlist
PUT /api/playlists/{playlist_id}
Body: { check_interval, quality, format, enabled, ... }
Response: PlaylistSubscription
# Delete playlist
DELETE /api/playlists/{playlist_id}?delete_videos=true
Response: { status: "ok" }
# Trigger manual check
POST /api/playlists/{playlist_id}/check
Response: { new_videos: int, status: "ok" }
# Update start point
POST /api/playlists/{playlist_id}/start-point
Body: { video_id: string }
Response: { updated_videos: int }
```
### 4.2 Video Endpoints
```
# List videos for playlist
GET /api/playlists/{playlist_id}/videos?status=PENDING&limit=50&offset=0
Response: { videos: [VideoRecord, ...], total: int }
# Get single video
GET /api/videos/{video_id}
Response: VideoRecord
# Request download/re-download
POST /api/videos/{video_id}/download
Response: { status: "ok", metube_id: string }
# Mark file as moved
POST /api/videos/{video_id}/file-moved
Body: { location_note: string }
Response: VideoRecord
# Skip video (mark as SKIPPED)
POST /api/videos/{video_id}/skip
Response: VideoRecord
# Reset video (back to PENDING)
POST /api/videos/{video_id}/reset
Response: VideoRecord
```
### 4.3 System Endpoints
```
# Get system status
GET /api/status
Response: {
total_playlists: int,
active_playlists: int,
total_videos: int,
pending_downloads: int,
active_downloads: int,
completed_downloads: int,
failed_downloads: int,
metube_status: { connected: bool, version: string }
}
# Get scheduler status
GET /api/scheduler/status
Response: {
jobs: [{ id, playlist_id, next_run, ... }],
running: bool
}
# Trigger manual sync with MeTube
POST /api/sync-metube
Response: { synced_videos: int, status: "ok" }
```
---
## 5. Database Schema (SQLite)
```sql
-- Playlists
CREATE TABLE playlists (
id TEXT PRIMARY KEY,
url TEXT NOT NULL UNIQUE,
title TEXT,
check_interval INTEGER DEFAULT 60,
last_checked TIMESTAMP,
start_point TEXT,
quality TEXT DEFAULT 'best',
format TEXT DEFAULT 'mp4',
folder TEXT,
enabled BOOLEAN DEFAULT 1,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
-- Videos
CREATE TABLE videos (
id TEXT PRIMARY KEY,
playlist_id TEXT NOT NULL,
video_url TEXT NOT NULL,
video_id TEXT NOT NULL,
title TEXT,
playlist_index INTEGER,
upload_date TIMESTAMP,
status TEXT DEFAULT 'PENDING',
download_requested_at TIMESTAMP,
download_completed_at TIMESTAMP,
metube_download_id TEXT,
original_filename TEXT,
file_moved BOOLEAN DEFAULT 0,
file_location_note TEXT,
error_message TEXT,
retry_count INTEGER DEFAULT 0,
last_error_at TIMESTAMP,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY (playlist_id) REFERENCES playlists(id) ON DELETE CASCADE
);
-- Indexes
CREATE INDEX idx_videos_playlist_id ON videos(playlist_id);
CREATE INDEX idx_videos_status ON videos(status);
CREATE INDEX idx_videos_video_id ON videos(video_id);
CREATE INDEX idx_videos_playlist_index ON videos(playlist_id, playlist_index);
-- Activity Log (optional, for audit trail)
CREATE TABLE activity_log (
id INTEGER PRIMARY KEY AUTOINCREMENT,
timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
event_type TEXT, -- playlist_added, video_downloaded, check_completed, etc.
playlist_id TEXT,
video_id TEXT,
details TEXT, -- JSON blob
FOREIGN KEY (playlist_id) REFERENCES playlists(id) ON DELETE CASCADE,
FOREIGN KEY (video_id) REFERENCES videos(id) ON DELETE CASCADE
);
CREATE INDEX idx_activity_log_timestamp ON activity_log(timestamp);
CREATE INDEX idx_activity_log_event_type ON activity_log(event_type);
```
---
## 6. Technology Stack
### 6.1 Backend
- **Framework**: FastAPI (Python 3.13+)
- Modern async support
- Auto-generated OpenAPI docs
- WebSocket support
- Dependency injection
- **Database**: SQLAlchemy + SQLite
- ORM for easy data modeling
- Migration support with Alembic
- **Scheduler**: APScheduler
- Async job scheduling
- Persistent job storage
- Cron-like intervals
- **MeTube Integration**:
- aiohttp (HTTP client)
- python-socketio (WebSocket client)
- **Dependencies**:
```
fastapi
uvicorn[standard]
sqlalchemy
alembic
apscheduler
aiohttp
python-socketio[client]
yt-dlp
pydantic
python-dotenv
```
### 6.2 Frontend (Optional - can reuse MeTube's Angular)
- **Option A**: Extend MeTube's Angular UI
- Add new routes/components for playlists
- Integrate with existing UI
- **Option B**: Separate Vue.js/React SPA
- Independent frontend
- Modern UI framework
- **Recommended**: Extend MeTube's Angular for consistency
### 6.3 Deployment
- **Docker Compose**:
```yaml
services:
metube:
image: ghcr.io/alexta69/metube
ports: ["8081:8081"]
volumes:
- ./downloads:/downloads
playlist-monitor:
build: ./playlist-monitor
ports: ["8082:8082"]
environment:
- METUBE_URL=http://metube:8081
- DATABASE_URL=sqlite:///data/playlists.db
- CHECK_INTERVAL=60
volumes:
- ./monitor-data:/data
depends_on:
- metube
```
---
## 7. Configuration
```yaml
# config.yaml
metube:
url: http://localhost:8081
reconnect_interval: 5 # seconds
database:
url: sqlite:///data/playlists.db
echo: false
scheduler:
enabled: true
default_check_interval: 60 # minutes
max_concurrent_downloads: 3
retry_failed_after: 24 # hours
downloads:
default_quality: best
default_format: mp4
default_folder: playlists/{playlist_title}
logging:
level: INFO
file: logs/playlist-monitor.log
server:
host: 0.0.0.0
port: 8082
```
---
## 8. Implementation Plan (Phases)
### **Phase 1: Core Infrastructure (Week 1-2)**
- [ ] Set up FastAPI project structure
- [ ] Database schema and SQLAlchemy models
- [ ] Basic CRUD operations for playlists
- [ ] MeTube client implementation
- [ ] Configuration management
### **Phase 2: Playlist Management (Week 2-3)**
- [ ] Add playlist endpoint (with yt-dlp integration)
- [ ] Fetch and parse playlist videos
- [ ] Implement start_point logic
- [ ] Basic video tracking
### **Phase 3: Scheduler & Automation (Week 3-4)**
- [ ] APScheduler integration
- [ ] Periodic check implementation
- [ ] Download triggering logic
- [ ] Status synchronization with MeTube
### **Phase 4: Advanced Features (Week 4-5)**
- [ ] File movement tracking
- [ ] Manual re-download
- [ ] Error handling and retry logic
- [ ] Activity logging
### **Phase 5: UI Integration (Week 5-6)**
- [ ] REST API documentation (OpenAPI)
- [ ] Angular components (if extending MeTube UI)
- [ ] Playlist listing and details view
- [ ] Video status dashboard
### **Phase 6: Testing & Deployment (Week 6-7)**
- [ ] Unit tests
- [ ] Integration tests
- [ ] Docker containerization
- [ ] Documentation and user guide
---
## 9. Key Design Decisions
### 9.1 Why Separate Service?
- **Modularity**: Keep concerns separated
- **Independence**: Can run without modifying MeTube core
- **Scalability**: Can scale independently
- **Maintenance**: Easier to update/maintain
### 9.2 Why SQLite?
- **Simplicity**: No external DB server needed
- **Portability**: Single file database
- **Sufficient**: Adequate for single-user/small-team use
- **Upgrade Path**: Can migrate to PostgreSQL if needed
### 9.3 File Movement Tracking
- **Decoupled Design**: Don't check filesystem, rely on metadata
- **User Control**: User explicitly marks files as moved
- **Prevents Re-downloads**: Once marked completed, won't re-download
- **Flexibility**: Files can be moved/organized freely
### 9.4 Start Point Implementation
- **Options**:
- Video ID (specific video)
- Playlist index (position number)
- Upload date (date cutoff)
- **Recommended**: Video ID (most reliable)
- **Behavior**: Videos before start point marked as SKIPPED
---
## 10. Future Enhancements
### 10.1 Advanced Features
- **Smart Download Scheduling**: Download during off-peak hours
- **Bandwidth Management**: Rate limiting per playlist
- **Multi-platform Support**: Support non-YouTube playlists
- **Notification System**: Email/webhook on new videos
- **Archive Mode**: Download only specific date ranges
- **Duplicate Detection**: Prevent duplicate downloads across playlists
### 10.2 UI Enhancements
- **Timeline View**: Visual timeline of uploads
- **Bulk Operations**: Batch skip/reset/download
- **Statistics Dashboard**: Charts and graphs
- **Search & Filters**: Advanced video filtering
### 10.3 Integration Features
- **Plex/Jellyfin Integration**: Auto-update media libraries
- **RSS Feed**: Generate RSS feeds for playlists
- **API Webhooks**: Notify external systems
- **Cloud Sync**: Backup/sync across instances
---
## 11. Security Considerations
### 11.1 Authentication
- Implement API key or JWT authentication
- Integrate with MeTube's auth (if available)
- Rate limiting to prevent abuse
### 11.2 Data Privacy
- Encrypt sensitive data (cookies, tokens)
- Regular security audits
- HTTPS only in production
### 11.3 Resource Management
- Limit number of playlists per instance
- Limit number of videos per playlist
- Disk space monitoring
---
## 12. Monitoring & Observability
### 12.1 Metrics
- Playlist check frequency
- Download success/failure rates
- MeTube API response times
- Database query performance
### 12.2 Logging
- Structured logging (JSON)
- Log levels: DEBUG, INFO, WARNING, ERROR
- Log rotation and retention
### 12.3 Health Checks
- Database connectivity
- MeTube connectivity
- Scheduler status
- Disk space availability
---
## 13. Summary
This architecture provides a **robust, scalable, and maintainable** solution for automated playlist monitoring that:
✅ **Integrates seamlessly** with MeTube via REST API and WebSocket
✅ **Tracks video status** independently of filesystem
✅ **Prevents re-downloads** even when files are moved
✅ **Supports flexible start points** for granular control
✅ **Provides periodic scheduling** with configurable intervals
✅ **Maintains persistent state** across restarts
✅ **Offers comprehensive API** for programmatic control
✅ **Scales gracefully** with proper database design
The service is designed to be **deployed alongside MeTube** as a companion microservice, leveraging MeTube's powerful download capabilities while adding intelligent playlist monitoring on top.
---
## Appendix A: Example Use Cases
### Use Case 1: YouTube Channel Monitoring
**Scenario**: Monitor a YouTube channel for new uploads
**Setup**:
- Add channel's "Uploads" playlist
- Set start_point to latest video
- Check interval: 30 minutes
- All new videos downloaded automatically
### Use Case 2: Partial Playlist Download
**Scenario**: Download only recent videos from a large playlist
**Setup**:
- Add playlist URL
- Set start_point to video from 6 months ago
- Older videos marked as SKIPPED
- New videos downloaded as they appear
### Use Case 3: Archive & Organize
**Scenario**: Download and organize by playlist
**Setup**:
- Multiple playlists with custom folders
- Quality: best
- Format: mp4
- After download, move files to NAS
- Mark as "file moved" to prevent re-download
---
## Appendix B: API Examples
### Adding a Playlist
```bash
curl -X POST http://localhost:8082/api/playlists \
-H "Content-Type: application/json" \
-d '{
"url": "https://www.youtube.com/playlist?list=PLxxxxxx",
"check_interval": 60,
"start_point": "dQw4w9WgXcQ",
"quality": "1080",
"format": "mp4",
"folder": "my-playlist"
}'
```
### Listing Videos
```bash
curl http://localhost:8082/api/playlists/{playlist_id}/videos?status=PENDING&limit=20
```
### Marking File as Moved
```bash
curl -X POST http://localhost:8082/api/videos/{video_id}/file-moved \
-H "Content-Type: application/json" \
-d '{
"location_note": "Moved to /mnt/nas/videos/"
}'
```
---
**Document Version**: 1.0
**Last Updated**: 2025-01-19
**Author**: AI Architecture Team
**Status**: Draft for Review