580 lines
32 KiB
Markdown
580 lines
32 KiB
Markdown
# Live Transcoding Reintroduction — Comprehensive Implementation Proposal
|
||
|
||
> Status: **Backlog / Design Proposal**
|
||
> Author: Engineering
|
||
> Target Branch: `feature/live-transcode-v2`
|
||
> Related Docs:
|
||
> - `docs/active/media-streaming/TRANSCODING_REMOVAL_DESIGN.md`
|
||
> - `docs/active/media-streaming/TRANSCODING_REMOVAL_SUMMARY.md`
|
||
> - `docs/archive/transcoding-legacy/JELLYFIN_TRANSCODING_ARCHITECTURE.md`
|
||
> - `docs/archive/transcoding-legacy/STASH-ANALYSIS-AND-SOLUTION-PLAN.md`
|
||
> - `docs/archive/transcoding-legacy/03-process-management-transcoding.md`
|
||
> - `docs/archive/transcoding-legacy/ANTI-JITTER-IMPLEMENTATION.md`
|
||
> - `docs/archive/transcoding-legacy/VIDEO-PLAYER-REPLACEMENT-PLAN.md`
|
||
|
||
---
|
||
|
||
## 1. Why Bring Live Transcoding Back?
|
||
|
||
NextAV currently routes any non‑natively‑supported format (MKV/AVI/WMV/FLV/MOV/TS/etc.) to a **Local Player Launcher** (VLC/IINA/PotPlayer). This eliminated server CPU usage but came at three real costs:
|
||
|
||
| Loss | User Impact |
|
||
|------|-------------|
|
||
| Cross‑device playback (phone, tablet, TV browser) | "I can't watch my MKVs on iPad" |
|
||
| Single‑click in‑browser playback for all formats | Friction: install app, allow protocol, retry |
|
||
| Adaptive bitrate over slow networks | Large files unplayable over LAN/WAN |
|
||
|
||
The goal of v2 is to bring back **YouTube/Jellyfin‑class live transcoding** that does **not** repeat the historical pain points (process leaks, jitter, broken seek, duration corruption, runaway CPU).
|
||
|
||
---
|
||
|
||
## 2. Honest Postmortem of v1 (Why It Was Removed)
|
||
|
||
From the legacy docs and `archive/transcoding-legacy/*`, the v1 transcoder failed at six things at once:
|
||
|
||
### 2.1 Process Lifecycle
|
||
- FFmpeg child processes orphaned on browser close / route change.
|
||
- No central registry → could not enumerate or kill cleanly.
|
||
- No client heartbeat → processes ran forever when client died silently.
|
||
- `kill('SIGKILL')` raced with the HTTP stream close, leading to half-shut pipes and `EPIPE`/`ECONNRESET`.
|
||
|
||
### 2.2 Seeking
|
||
- Old design transcoded **from t=0** on every seek. A seek to 1h00m wasted 1h of CPU before producing one byte.
|
||
- The Stash analysis (`STASH-ANALYSIS-AND-SOLUTION-PLAN.md`) correctly identified that the fix is `-ss <seek> -i <file>` (input seek) + restart, but it was never properly wired into the player.
|
||
|
||
### 2.3 Duration & Progress Jitter
|
||
- The browser's `HTMLMediaElement.duration` reflects **buffered** duration when the upstream is a piped MP4 with `frag_keyframe+empty_moov`. The progress bar therefore reported absurd durations (e.g. "6s" for a 9‑min video) and jumped backwards as new fragments arrived.
|
||
- `ANTI-JITTER-IMPLEMENTATION.md` patched the symptom in the UI, not the cause (lack of a known duration / HLS playlist with `#EXT-X-ENDLIST`).
|
||
|
||
### 2.4 Format Detection
|
||
- Decision was made per request based on a JSON blob in `media.codec_info`. That blob was sometimes missing, sometimes wrong (the scanner only ran ffprobe once and never re‑probed), so the same file would sometimes stream direct, sometimes transcode.
|
||
|
||
### 2.5 Resource Contention
|
||
- No concurrency cap. Three simultaneous 1080p `libx264 -preset veryfast` jobs would saturate any consumer CPU and starve thumbnail generation and the Next.js event loop.
|
||
- No throttling: FFmpeg encoded as fast as possible and ran 60+ seconds ahead of the player, wasting CPU if the user closed the tab.
|
||
|
||
### 2.6 HLS Half-Implementation
|
||
- `src/app/api/stream/hls/...` exists today and generates a **fake playlist** that points at byte‑range slices of the source file (`ts-segmentation-service.ts`). For real `.ts` Transport Streams this can work, but for arbitrary inputs (MKV/AVI) the resulting "segments" are not valid TS and players fail with `bufferAppendError`.
|
||
- The `hls-session-manager.ts` heartbeat exists but nothing in the player calls it.
|
||
|
||
---
|
||
|
||
## 3. Design Principles for v2
|
||
|
||
1. **HLS-first, MP4-fragment second.** All live transcoding produces an HLS playlist + MPEG-TS (or fMP4) segments on disk. The browser sees a finite playlist with a real `#EXT-X-PLAYLIST-TYPE:VOD` and `#EXT-X-ENDLIST`, so `duration` is correct from frame 1 and seeking works without restarts inside the encoded window.
|
||
2. **Stash-style seek = restart.** Seeks outside the already-encoded window kill the FFmpeg job and restart with `-ss <t>` at the new position. The playlist is regenerated with a virtual timeline that maps to real timestamps via `#EXT-X-DISCONTINUITY`.
|
||
3. **One job per (videoId, profile).** A Job is keyed by `videoId + quality + audioTrack + subTrack`. Re-requests of the same key attach to the existing job; different keys evict the older one for that videoId after a grace period.
|
||
4. **Heartbeats are mandatory.** The client pings every 20s. Jobs without a ping for 60s (HLS) / 10s (progressive) are killed. This is Jellyfin's model and the only one proven to survive real users.
|
||
5. **Server-Sent Events for control.** A single SSE channel per session reports `ready`, `segment-available`, `transcode-error`, `transcode-progress`, replacing polling.
|
||
6. **Hard concurrency cap.** Default `MAX_CONCURRENT_TRANSCODES=2` per node. Additional requests get HTTP 503 with `Retry-After`, plus a clear UI message.
|
||
7. **Direct streaming is always tried first.** Live transcoding is a fallback, not the default. H.264/AAC MP4 over range requests stays the happy path.
|
||
8. **Disk is the cache.** Every produced segment is written to `/tmp/nextav-hls/<videoId>/<profile>/seg_%05d.ts`. After the job is killed, segments live another 5 minutes so the user can rewind without re-encoding.
|
||
9. **Observability is shipped, not patched on.** Every job emits structured logs + a `/api/transcode/status` snapshot suitable for a tiny admin page.
|
||
10. **Feature-flagged rollout.** Live transcoding is behind `ENABLE_LIVE_TRANSCODE=false` until validated on a Linux box with both software and hardware encoders.
|
||
|
||
---
|
||
|
||
## 4. End-to-End Architecture
|
||
|
||
```
|
||
┌──────────────────────────────────────────────────────────────────────┐
|
||
│ Browser │
|
||
│ ArtPlayer + hls.js ──► /api/transcode/start (POST, returns job) │
|
||
│ │ │
|
||
│ │ loads m3u8 │
|
||
│ ▼ │
|
||
│ /api/transcode/[jobId]/master.m3u8 ──► VOD playlist (finite) │
|
||
│ /api/transcode/[jobId]/seg/000.ts ──► segment file (disk) │
|
||
│ │ │
|
||
│ │ every 20s │
|
||
│ /api/transcode/[jobId]/ping (POST) │
|
||
│ │ │
|
||
│ /api/transcode/[jobId] (DELETE) on unmount / close │
|
||
└──────────────────────────────────────────────────────────────────────┘
|
||
│
|
||
▼
|
||
┌──────────────────────────────────────────────────────────────────────┐
|
||
│ Next.js Server │
|
||
│ │
|
||
│ TranscodeOrchestrator (singleton, src/lib/transcode/orchestrator) │
|
||
│ ├─ JobRegistry (Map<jobId, TranscodeJob>) │
|
||
│ ├─ HeartbeatManager (kill-timers per job) │
|
||
│ ├─ ConcurrencyGate (semaphore, MAX_CONCURRENT_TRANSCODES) │
|
||
│ ├─ SegmentJanitor (LRU eviction, disk-budget aware) │
|
||
│ └─ HwCapabilityProbe (one-time on boot, picks h264_videotoolbox│
|
||
│ / h264_nvenc / h264_qsv / libx264) │
|
||
│ │
|
||
│ TranscodeJob │
|
||
│ ├─ ffmpeg ChildProcess (spawned, attached to AbortController) │
|
||
│ ├─ outDir (/tmp/nextav-hls/<videoId>/<profile>) │
|
||
│ ├─ playlistPath, segmentCount, lastSegmentMtime │
|
||
│ ├─ lastPing, isPaused (throttler input) │
|
||
│ └─ status: pending | encoding | ready | paused | failed | killed │
|
||
│ │
|
||
│ FormatDecisionEngine (src/lib/transcode/decision.ts) │
|
||
│ direct | hls-remux (no re-encode) | hls-transcode │
|
||
└──────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## 5. Data & API Surface
|
||
|
||
### 5.1 New / Modified Tables
|
||
|
||
No schema changes to user data, but a small operational table to survive process restarts:
|
||
|
||
```sql
|
||
CREATE TABLE IF NOT EXISTS transcode_jobs (
|
||
job_id TEXT PRIMARY KEY,
|
||
media_id INTEGER NOT NULL REFERENCES media(id) ON DELETE CASCADE,
|
||
profile TEXT NOT NULL, -- e.g. '720p-h264-aac'
|
||
out_dir TEXT NOT NULL,
|
||
start_ts REAL NOT NULL DEFAULT 0, -- the -ss value
|
||
status TEXT NOT NULL, -- pending|encoding|ready|paused|failed|killed
|
||
pid INTEGER,
|
||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||
last_ping_at DATETIME,
|
||
error TEXT
|
||
);
|
||
CREATE INDEX IF NOT EXISTS idx_transcode_jobs_media ON transcode_jobs(media_id);
|
||
CREATE INDEX IF NOT EXISTS idx_transcode_jobs_status ON transcode_jobs(status);
|
||
```
|
||
|
||
On boot, `TranscodeOrchestrator.recover()` reads `status IN ('encoding','paused')`, attempts `process.kill(pid, 0)` to see if the process is alive; if not, mark `killed`, schedule disk cleanup.
|
||
|
||
### 5.2 HTTP API
|
||
|
||
| Method | Path | Purpose |
|
||
|--------|------|---------|
|
||
| `POST` | `/api/transcode/start` | Body: `{ mediaId, profile?, startTime? }`. Returns `{ jobId, playlistUrl, etaSeconds, profile }`. Idempotent on `(mediaId, profile)` within 5s. |
|
||
| `GET` | `/api/transcode/:jobId/master.m3u8` | The VOD playlist. Regenerated on every request from disk state — always reflects what segments actually exist. Includes `#EXT-X-PLAYLIST-TYPE:VOD` and `#EXT-X-ENDLIST` once encoding finishes. |
|
||
| `GET` | `/api/transcode/:jobId/seg/:n.ts` | Streams a segment file. Sets `Cache-Control: private, max-age=3600` and strong ETag. |
|
||
| `POST` | `/api/transcode/:jobId/ping` | Body: `{ position, isPaused }`. Resets kill-timer, feeds throttler. |
|
||
| `POST` | `/api/transcode/:jobId/seek` | Body: `{ position }`. If the target is **inside** the encoded window → 204 (client just seeks). If **outside** → orchestrator kills and restarts with `-ss position`, responds with a **new** `jobId` and the client reloads. |
|
||
| `DELETE` | `/api/transcode/:jobId` | Kills the job and schedules cleanup. |
|
||
| `GET` | `/api/transcode/:jobId/events` | SSE: `ready`, `segment`, `progress`, `error`, `killed`. |
|
||
| `GET` | `/api/transcode/status` | Admin/debug snapshot of `JobRegistry`. |
|
||
|
||
### 5.3 Profiles
|
||
|
||
```ts
|
||
// src/lib/transcode/profiles.ts
|
||
export const PROFILES = {
|
||
'1080p-h264-aac': { vCodec:'h264', vBitrate:'5000k', maxHeight:1080, aCodec:'aac', aBitrate:'192k' },
|
||
'720p-h264-aac': { vCodec:'h264', vBitrate:'2800k', maxHeight: 720, aCodec:'aac', aBitrate:'160k' },
|
||
'480p-h264-aac': { vCodec:'h264', vBitrate:'1200k', maxHeight: 480, aCodec:'aac', aBitrate:'128k' },
|
||
'remux-copy': { vCodec:'copy', aCodec:'copy' }, // for AVC+AAC in MKV — wrap into HLS without re-encoding
|
||
} as const;
|
||
```
|
||
|
||
The client picks an initial profile from the saved user preference; ABR is **not** in v2 scope (single‑variant playlist only). Quality switching = new job.
|
||
|
||
---
|
||
|
||
## 6. Format Decision Engine
|
||
|
||
```ts
|
||
// src/lib/transcode/decision.ts
|
||
type Decision =
|
||
| { kind: 'direct' } // browser can play the file as-is
|
||
| { kind: 'hls-remux'; profile: 'remux-copy' } // codecs OK, container not
|
||
| { kind: 'hls-transcode'; profile: '720p-h264-aac' | ... };
|
||
```
|
||
|
||
Rules, evaluated in order:
|
||
|
||
1. If `extension ∈ {mp4,m4v,webm,ogg,ogv}` **and** `videoCodec ∈ {h264,vp8,vp9,av1}` **and** `audioCodec ∈ {aac,opus,vorbis}` → `direct`.
|
||
2. If `videoCodec ∈ {h264}` **and** `audioCodec ∈ {aac}` but container is anything else (mkv, ts, mov) → `hls-remux` (`-c copy`, basically free CPU).
|
||
3. If `videoCodec ∈ {hevc,vp9,av1}` and the requesting User-Agent indicates support (Safari/Chrome with hardware decode) → `direct` with a probe header (HTTP 200 + `X-Probe: 1`); on player error within 5s, client posts `/api/transcode/start` to fall back.
|
||
4. Otherwise → `hls-transcode` with the user's preferred profile (default 720p).
|
||
|
||
The decision is computed **fresh** on every `/api/transcode/start` using a server-side ffprobe of the file (cached in `media.codec_info` with a TTL of 7 days and a `codec_info_version` integer so we can invalidate the whole cache atomically).
|
||
|
||
---
|
||
|
||
## 7. FFmpeg Command Templates
|
||
|
||
### 7.1 Transcode (the hot path)
|
||
|
||
```bash
|
||
ffmpeg \
|
||
-hide_banner -loglevel error -nostats \
|
||
-ss <START_TS> \ # input seek for fast start
|
||
-copyts \ # keep original timestamps for accurate playlist
|
||
-i "<INPUT_PATH>" \
|
||
-map 0:v:0 -map 0:a:0? \
|
||
-c:v <VCODEC> \ # h264_videotoolbox|h264_nvenc|h264_qsv|libx264
|
||
-preset <PRESET> \ # libx264: veryfast | hw: p4/quality
|
||
-profile:v main -level 4.1 \
|
||
-pix_fmt yuv420p \
|
||
-vf "scale=-2:'min(<MAX_H>,ih)':flags=lanczos" \
|
||
-b:v <VBITRATE> -maxrate <VBITRATE> -bufsize <2*VBITRATE> \
|
||
-force_key_frames "expr:gte(t,n_forced*4)" \
|
||
-c:a aac -b:a <ABITRATE> -ac 2 \
|
||
-f hls \
|
||
-hls_time 4 \ # 4s segments — fast startup, good seek granularity
|
||
-hls_list_size 0 \ # keep all segments in the playlist
|
||
-hls_segment_type mpegts \
|
||
-hls_flags independent_segments+temp_file \
|
||
-hls_segment_filename "<OUT_DIR>/seg_%05d.ts" \
|
||
-start_number 0 \
|
||
"<OUT_DIR>/index.m3u8"
|
||
```
|
||
|
||
Key choices and **why**:
|
||
|
||
- `-ss` **before** `-i` (input seek): O(1) on indexed containers; correctly drops everything before the seek point.
|
||
- `-copyts` + `-start_at_zero` are NOT used together — we keep original timestamps so when we discontinuity-merge after a seek-restart the player handles it via `#EXT-X-DISCONTINUITY`.
|
||
- `-force_key_frames expr:gte(t,n_forced*4)` makes every segment independently decodable, which lets `-hls_flags independent_segments` work and gives clean seeks.
|
||
- `-hls_flags temp_file` writes `seg_00042.ts.tmp` and renames atomically, so the playlist regenerator never sees a half-written segment.
|
||
- We do **not** use `-re` — we want to encode as fast as the CPU allows, then **throttle** via stdin `p`/`u` once we are >60s ahead of playback (Jellyfin pattern).
|
||
|
||
### 7.2 Remux (free path for h264+aac in mkv/mov)
|
||
|
||
Same skeleton but `-c:v copy -c:a copy -bsf:v h264_mp4toannexb` (the bitstream filter is required to convert AVC's length-prefixed NAL units into Annex‑B for MPEG‑TS).
|
||
|
||
### 7.3 Hardware Acceleration
|
||
|
||
Probed once at boot by running `ffmpeg -hide_banner -encoders | grep h264_`. Selection priority:
|
||
|
||
1. **macOS dev**: `h264_videotoolbox` (zero CPU, ships with macOS)
|
||
2. **Linux + NVIDIA**: `h264_nvenc`
|
||
3. **Linux + Intel iGPU**: `h264_qsv` (requires `libmfx`)
|
||
4. **Fallback everywhere**: `libx264 -preset veryfast`
|
||
|
||
The choice is recorded in `/api/transcode/status` and surfaced in Settings → "Transcoding Backend: h264_videotoolbox (auto)".
|
||
|
||
---
|
||
|
||
## 8. Process Lifecycle (the part v1 got wrong)
|
||
|
||
```
|
||
client click play
|
||
│
|
||
▼
|
||
POST /api/transcode/start
|
||
│ ┌─────────────────────────────────────────────────────────┐
|
||
│ │ ConcurrencyGate.acquire() ── may 503 with Retry-After │
|
||
│ └─────────────────────────────────────────────────────────┘
|
||
▼
|
||
orchestrator.startJob(mediaId, profile, startTs)
|
||
│
|
||
├─ pick existing job? (same key, alive, ping <30s) → return it
|
||
├─ ffprobe file (cached) → decide remux vs transcode
|
||
├─ mkdir <out_dir>
|
||
├─ spawn ffmpeg, attach { stdout: ignore, stderr: pipe-to-ring-buffer, stdio:[pipe,ignore,pipe] }
|
||
├─ insert into DB (status=encoding, pid)
|
||
├─ HeartbeatManager.armKillTimer(jobId, 60s)
|
||
├─ wait for first segment to appear (fs.watch on out_dir)
|
||
│ ── on timeout 15s → kill + fail with TRANSCODE_STARTUP_TIMEOUT
|
||
└─ return { jobId, playlistUrl: /api/transcode/<jobId>/master.m3u8 }
|
||
|
||
client loads master.m3u8 every ~segment
|
||
client posts /ping every 20s ── resets kill-timer
|
||
client posts /seek when user drags ── may trigger restart
|
||
client posts DELETE on close ── immediate kill
|
||
|
||
orchestrator monitors:
|
||
- ffmpeg exit code → status=ready (0) | failed (non-zero, capture stderr ring)
|
||
- stderr ring → parse "frame=… time=00:01:23.45" for progress events
|
||
- throttler tick (1Hz):
|
||
if (encoder_position - last_ping_position) > 60s and !isPaused
|
||
ffmpeg.stdin.write('p\n') // pause
|
||
else if was paused and gap < 30s
|
||
ffmpeg.stdin.write('u\n') // resume
|
||
```
|
||
|
||
### 8.1 Kill Is a State Machine, Not a Signal
|
||
|
||
```ts
|
||
async function killJob(jobId: string, reason: string) {
|
||
const job = registry.get(jobId);
|
||
if (!job || job.status === 'killed') return;
|
||
job.status = 'killed';
|
||
// 1. Try graceful: write 'q' to stdin (ffmpeg writes trailers, exits 0)
|
||
try { job.proc.stdin.write('q\n'); } catch {}
|
||
// 2. Wait up to 3s
|
||
const exited = await waitForExit(job.proc, 3_000);
|
||
// 3. SIGTERM, then 2s grace
|
||
if (!exited) { job.proc.kill('SIGTERM'); await waitForExit(job.proc, 2_000); }
|
||
// 4. SIGKILL
|
||
if (!job.proc.killed) job.proc.kill('SIGKILL');
|
||
// 5. DB + concurrency release + janitor schedule
|
||
db.prepare('UPDATE transcode_jobs SET status=? WHERE job_id=?').run('killed', jobId);
|
||
gate.release();
|
||
janitor.scheduleCleanup(job.outDir, /* delayMs */ 5 * 60_000);
|
||
}
|
||
```
|
||
|
||
This is the three-layer pattern from `archive/transcoding-legacy/03-process-management-transcoding.md` (HTTP context → registry → forced kill), ported faithfully.
|
||
|
||
### 8.2 Seek Handling
|
||
|
||
```ts
|
||
// /api/transcode/[jobId]/seek
|
||
if (target >= job.encodedFromTs && target <= job.lastSegmentTs - 8) {
|
||
// Inside the encoded window — the client already has segments. No action.
|
||
return new Response(null, { status: 204 });
|
||
}
|
||
// Outside → restart. Old job dies; new job starts at target.
|
||
const newJob = await orchestrator.restart(job, /* startTs */ target);
|
||
return Response.json({ jobId: newJob.id, playlistUrl: newJob.playlistUrl });
|
||
```
|
||
|
||
---
|
||
|
||
## 9. Playlist Regeneration (eliminates duration jitter)
|
||
|
||
The legacy implementation streamed the playlist FFmpeg wrote directly. That playlist grew over time, which is why the browser reported a growing duration and the bar jumped.
|
||
|
||
v2 regenerates the playlist on **every GET** from the on-disk segment files plus the known total duration:
|
||
|
||
```ts
|
||
function buildPlaylist(job: TranscodeJob): string {
|
||
const segs = fs.readdirSync(job.outDir)
|
||
.filter(f => /^seg_\d{5}\.ts$/.test(f))
|
||
.sort();
|
||
const totalDuration = job.knownDuration; // from ffprobe, REAL duration
|
||
|
||
const lines = [
|
||
'#EXTM3U',
|
||
'#EXT-X-VERSION:3',
|
||
'#EXT-X-TARGETDURATION:5',
|
||
'#EXT-X-MEDIA-SEQUENCE:0',
|
||
'#EXT-X-PLAYLIST-TYPE:VOD',
|
||
];
|
||
|
||
// Emit all real segments
|
||
for (const seg of segs) lines.push(`#EXTINF:4.000,`, `seg/${seg}`);
|
||
|
||
// If FFmpeg has not yet produced the tail, append a "gap" sentinel so the
|
||
// browser knows total duration. We DO NOT emit #EXT-X-ENDLIST until the
|
||
// encoder exits cleanly — that flips the playlist into final VOD mode.
|
||
if (job.status === 'ready') lines.push('#EXT-X-ENDLIST');
|
||
|
||
return lines.join('\n');
|
||
}
|
||
```
|
||
|
||
The trick: even before `#EXT-X-ENDLIST` is emitted, the player has a `#EXT-X-PLAYLIST-TYPE:VOD` hint and the real ffprobe duration is stored in the segment count expectation. We also send `Content-Duration` header for ArtPlayer to consume.
|
||
|
||
---
|
||
|
||
## 10. UI / Player Integration
|
||
|
||
### 10.1 Component Flow
|
||
|
||
`unified-video-player.tsx`:
|
||
|
||
```
|
||
detectVideoFormat(video)
|
||
│
|
||
├─ direct → ArtPlayer with src=/api/stream/direct/<id>
|
||
├─ hls-* → POST /api/transcode/start
|
||
│ ArtPlayer + hls.js with src=<playlistUrl>
|
||
│ on artplayer "seek" event → POST /seek (may swap source)
|
||
│ every 20s timer → POST /ping
|
||
│ on unmount / onClose → DELETE
|
||
└─ local-player → existing LocalPlayerLauncher (kept as final fallback
|
||
for formats we explicitly refuse, e.g. raw VOB / RealMedia)
|
||
```
|
||
|
||
### 10.2 UI States
|
||
|
||
| State | UI |
|
||
|-------|-----|
|
||
| `requesting` | Spinner + "Preparing video…" + ETA from `etaSeconds` |
|
||
| `encoding`, no segments yet | Same spinner; auto-timeout 15s → error UI |
|
||
| `encoding`, playing | Hidden, but a tiny "Transcoding" badge in the player chrome (toggle in settings) |
|
||
| `503 Retry-After` | Toast: "Server busy transcoding 2 other videos. Try again in 30s." |
|
||
| `failed` | Error card with stderr snippet + button "Open in local player" (degrades to existing LocalPlayerLauncher) |
|
||
|
||
### 10.3 Settings
|
||
|
||
New Settings → Playback section:
|
||
|
||
- **Transcoding**: Off / Auto / Always
|
||
- **Preferred quality**: 480p / 720p / 1080p / Source
|
||
- **Hardware acceleration**: read-only label of detected backend
|
||
- **Show transcoding badge in player**: toggle
|
||
|
||
Stored in `localStorage` and mirrored to `user_preferences` (table TBD or reuse existing settings store).
|
||
|
||
---
|
||
|
||
## 11. Concurrency, Throttling, Disk
|
||
|
||
### 11.1 Concurrency Gate
|
||
|
||
```ts
|
||
class Semaphore {
|
||
constructor(public max: number) {}
|
||
// acquire returns immediately or rejects with { code: 'BUSY', retryAfterSec }
|
||
}
|
||
```
|
||
|
||
`MAX_CONCURRENT_TRANSCODES` env var, default 2 for dev, configurable in Settings → Admin.
|
||
|
||
### 11.2 Throttling (the Jellyfin pattern)
|
||
|
||
A 1Hz ticker per active job:
|
||
|
||
```
|
||
encoderPosition = parse from stderr 'time=00:hh:mm:ss.ms'
|
||
playheadPosition = job.lastPingPosition
|
||
gap = encoderPosition - playheadPosition
|
||
if (gap > 60 && !job.paused) send 'p\n'; job.paused = true
|
||
if (gap < 30 && job.paused) send 'u\n'; job.paused = false
|
||
```
|
||
|
||
This caps wasted work at ~60s of encode beyond playback.
|
||
|
||
### 11.3 Disk Budget (Segment Janitor)
|
||
|
||
- Per-job soft cap: `5 minutes * profile_bitrate` ≈ 100 MB for 720p.
|
||
- Global soft cap: `TRANSCODE_DISK_BUDGET_MB`, default 5000 MB.
|
||
- LRU eviction of **whole jobs** (never partial — a half-deleted job breaks the playlist) when global cap is exceeded.
|
||
- After job kill: keep `outDir` alive 5 minutes so a quick reopen reuses the cache; then `rm -rf`.
|
||
|
||
---
|
||
|
||
## 12. Failure Modes & Recovery
|
||
|
||
| Failure | Detection | Response |
|
||
|---------|-----------|----------|
|
||
| FFmpeg won't start (binary missing) | spawn `ENOENT` | Hard 500 + log; surface in Settings "FFmpeg not found at <path>" |
|
||
| First segment never appears | 15s fs.watch timeout | Kill job, return `TRANSCODE_STARTUP_TIMEOUT`, client falls back to local-player UI |
|
||
| FFmpeg crashes mid-stream | `exit` event with code ≠ 0 | Capture last 50 stderr lines, mark `failed`, push SSE error, client offers retry or local-player |
|
||
| Disk full | `ENOSPC` in stderr | Janitor force-cleanup, then either restart job or fail with explicit message |
|
||
| Client tab closed silently | No ping for 60s | Auto-kill via HeartbeatManager |
|
||
| Next.js process restart | DB has `status=encoding` rows with dead PIDs | `recover()` on boot marks them killed, cleans disks |
|
||
| Two clients open same video | Same `(mediaId, profile)` key | Both attach to one job; ping is per-session, kill only when **all** sessions are gone (refcount on job) |
|
||
| H.264 in MKV thought to be HEVC | Bad codec_info cache | `decision.ts` re-probes if codec_info is older than 7 days or absent |
|
||
|
||
---
|
||
|
||
## 13. Cleanup of v1 Debris (must happen before v2)
|
||
|
||
The following files are dead code from v1 and must be deleted (not patched), so v2 starts from a clean slate:
|
||
|
||
- `src/lib/ffmpeg/process-registry.ts` (will be replaced by `src/lib/transcode/orchestrator.ts`)
|
||
- `src/lib/hls-session-manager.ts`
|
||
- `src/lib/ts-segmentation-service.ts`
|
||
- `src/lib/hls-error-handler.ts`
|
||
- `src/lib/ts-converter.ts`
|
||
- `src/app/api/stream/hls/**`
|
||
- `src/app/api/stream/[id]/transcode/**` (will be replaced by `/api/transcode/**`)
|
||
- `src/app/api/videos/[id]/convert-ts/**`
|
||
- `src/app/api/ffmpeg/status/**` (replaced by `/api/transcode/status`)
|
||
- `src/lib/hooks/use-protected-duration.ts` (no longer needed once the playlist reports correct duration)
|
||
|
||
The `local-player-launcher` UI and `/api/external-stream/[id]` stay — they remain the user-visible fallback when transcoding is disabled, fails, or is rate-limited.
|
||
|
||
---
|
||
|
||
## 14. New Files (proposed layout)
|
||
|
||
```
|
||
src/lib/transcode/
|
||
orchestrator.ts # JobRegistry, start/kill/recover, singleton
|
||
job.ts # TranscodeJob class
|
||
heartbeat.ts # HeartbeatManager
|
||
concurrency.ts # Semaphore
|
||
janitor.ts # disk LRU cleanup
|
||
throttler.ts # 1Hz stdin p/u
|
||
hw-probe.ts # one-time encoder probe, cached
|
||
profiles.ts # PROFILES constant
|
||
decision.ts # FormatDecisionEngine
|
||
ffmpeg-args.ts # buildTranscodeArgs / buildRemuxArgs
|
||
playlist.ts # buildPlaylist (regenerates m3u8 from disk)
|
||
ffprobe.ts # cached probe, writes media.codec_info
|
||
|
||
src/app/api/transcode/
|
||
start/route.ts # POST
|
||
[jobId]/master.m3u8/route.ts # GET
|
||
[jobId]/seg/[name]/route.ts # GET
|
||
[jobId]/ping/route.ts # POST
|
||
[jobId]/seek/route.ts # POST
|
||
[jobId]/route.ts # DELETE
|
||
[jobId]/events/route.ts # GET (SSE)
|
||
status/route.ts # GET admin
|
||
|
||
src/components/
|
||
unified-video-player.tsx # MODIFIED: branches on decision
|
||
transcode-status-badge.tsx # NEW: tiny chrome badge
|
||
```
|
||
|
||
---
|
||
|
||
## 15. Phased Roadmap
|
||
|
||
| Phase | Scope | Exit Criteria | Risk |
|
||
|-------|-------|--------------|------|
|
||
| **0. Cleanup** | Delete dead v1 code (§13), drop `fluent-ffmpeg` dep, keep only the `ffmpeg` binary for thumbnails. | `pnpm build` green; existing direct-stream + local-player flow untouched. | Low |
|
||
| **1. Spike: orchestrator + remux only** | `decision.ts` for `hls-remux` only (h264+aac in mkv/mov). Full orchestrator, kill state machine, heartbeat, single-segment playlist regen. No transcoding yet — copy only. | Play a 2 GB MKV (h264+aac) in Chrome/Safari/Firefox. Seek, pause, close. `lsof` shows no leaked FDs after 5 minutes. | **High** (the lifecycle code is the hard part) |
|
||
| **2. Software transcoding** | Add `libx264 -preset veryfast` path. 720p only. Throttler + concurrency gate. | Play an AVI (mpeg4+mp3) end-to-end. Two concurrent jobs run, third gets 503. Kill -9 the server: on restart, dead-job rows are cleaned, no orphaned ffmpeg. | High |
|
||
| **3. Hardware encoder + profiles** | `hw-probe.ts`, 480p/720p/1080p, settings UI. | macOS dev box uses `h264_videotoolbox`, CPU < 30% on 1080p. Linux box auto-picks `h264_nvenc` if available. | Medium |
|
||
| **4. Seek-restart + discontinuity** | `/seek` endpoint, restart-on-large-seek, `#EXT-X-DISCONTINUITY` in regenerated playlist. | Seek to t=1h in a 2h MKV completes within 3s on hw encoder; progress bar shows correct global timeline. | Medium |
|
||
| **5. SSE + observability** | `/events` SSE channel, `/status` admin page, structured stderr capture. | A dev panel at `/admin/transcode` shows live jobs, CPU%, ETA. | Low |
|
||
| **6. Polish + flags** | Feature flag default-on per env; user-facing error UI; docs. | E2E run-book passes (see §17). Move this doc to `docs/active/media-streaming/`. | Low |
|
||
|
||
Estimated effort: **6 calendar weeks** at one engineer, dominated by Phases 1 and 2.
|
||
|
||
---
|
||
|
||
## 16. Configuration (env)
|
||
|
||
```env
|
||
ENABLE_LIVE_TRANSCODE=false # master switch
|
||
FFMPEG_PATH=/usr/bin/ffmpeg
|
||
FFPROBE_PATH=/usr/bin/ffprobe
|
||
TRANSCODE_TMP_DIR=/tmp/nextav-hls
|
||
MAX_CONCURRENT_TRANSCODES=2
|
||
TRANSCODE_DISK_BUDGET_MB=5000
|
||
TRANSCODE_HEARTBEAT_TIMEOUT_MS=60000
|
||
TRANSCODE_STARTUP_TIMEOUT_MS=15000
|
||
TRANSCODE_HWACCEL=auto # auto|videotoolbox|nvenc|qsv|none
|
||
TRANSCODE_DEFAULT_PROFILE=720p-h264-aac
|
||
```
|
||
|
||
---
|
||
|
||
## 17. Acceptance / Run-Book
|
||
|
||
Manual E2E (must all pass on macOS + Linux):
|
||
|
||
1. **Direct path untouched**: play an MP4(H.264+AAC), confirm `/api/stream/direct/<id>` is used and no transcode rows appear.
|
||
2. **Remux happy path**: play an MKV (H.264+AAC). Confirm `hls-remux`, CPU < 5%.
|
||
3. **Transcode happy path**: play an AVI (MPEG-4 ASP). Confirm software encoder, CPU usage matches `nproc/2` budget, video plays smoothly.
|
||
4. **HW encoder**: on a Mac, confirm `h264_videotoolbox` is chosen, CPU < 30% on 1080p.
|
||
5. **Seek inside window**: seek +30s, no new job spawned.
|
||
6. **Seek outside window**: seek +30 minutes, old job dies within 3s, new job spawns, video resumes within 5s.
|
||
7. **Concurrency**: open three transcoding videos simultaneously; third receives 503 + UI toast.
|
||
8. **Heartbeat death**: close tab during transcoding, wait 90s, confirm process is killed (`ps aux | grep ffmpeg` empty) and segments are cleaned within 5 min.
|
||
9. **Crash recovery**: `kill -9` the Next.js server during transcoding, restart, confirm DB rows cleaned, disk cleaned, no orphan ffmpeg.
|
||
10. **Disk pressure**: fill disk to 95%, request a transcode → fails with explicit `ENOSPC` message, not a hang.
|
||
11. **Player error fallback**: corrupt a segment on disk while playing → SSE error event → UI offers "Open in local player" without page reload.
|
||
12. **Duration correctness**: For a 9:00 video, the progress bar reports 9:00 from the first frame and never moves backward.
|
||
|
||
---
|
||
|
||
## 18. Risks & Open Questions
|
||
|
||
- **Q1**: ABR is out of scope for v2. Should we plan a v3 with multi-variant playlists, or is single-variant "good enough" for a personal media library? *Recommendation: defer; single-variant with manual quality switch covers 95% of the use case.*
|
||
- **Q2**: Subtitles. MKVs frequently carry SRT/SSA tracks. Initial proposal: extract on demand to WebVTT via `ffmpeg -map 0:s:0 -c:s webvtt`, expose as `/api/transcode/<jobId>/sub/<idx>.vtt`. Not in Phase 1–4; track as a follow-up.
|
||
- **Q3**: Multi-audio tracks. Same story as subtitles — out of scope for v2, but the profile key already supports `+audioTrack` for future use.
|
||
- **R1**: Process management is **the** historical failure point. Phase 1 must be ruthlessly tested with `lsof`, `ps`, and crash injection before any encoding work begins. If Phase 1 cannot demonstrate zero leaks under chaos testing, **abort the whole effort** and keep the local-player approach.
|
||
- **R2**: Browser HLS behavior varies. Safari handles HLS natively; Chrome/Firefox need hls.js. The chosen `MEDIA-SEQUENCE` / `DISCONTINUITY` strategy must be validated on all three. Reserve 1 week of Phase 4 just for cross-browser bug triage.
|
||
|
||
---
|
||
|
||
## 19. Decision Required Before Coding
|
||
|
||
1. Approve the **HLS-on-disk** approach (vs. the v1 piped-MP4 approach). *Strongly recommended; everything else in this doc depends on it.*
|
||
2. Approve **single-variant playlists** for v2 (no ABR).
|
||
3. Confirm the **delete-first** stance on legacy code (§13). If we keep v1 stubs around, the new code paths will be ambiguous and the old bugs will resurface.
|
||
4. Confirm `MAX_CONCURRENT_TRANSCODES=2` as the default for a personal/home server. Adjust upward only when the target deployment is a multi-user server.
|
||
|
||
Once these four are signed off, work can start with Phase 0 cleanup.
|