diff --git a/.gitignore b/.gitignore index ea5edfd..33b7b32 100644 --- a/.gitignore +++ b/.gitignore @@ -55,6 +55,9 @@ __pycache__ # Downloaded media /downloads +# Private yt-dlp wheel (built by build-ytdlp.sh and baked into the image) +/vendor/yt_dlp-*.whl + # Runtime data /library /metube-config diff --git a/DEPLOY.md b/DEPLOY.md index 62e3651..5aa9203 100644 --- a/DEPLOY.md +++ b/DEPLOY.md @@ -23,9 +23,29 @@ Ubuntu 22.04 x86_64, so images are cross-built with `docker buildx` for buildx builder with amd64 support (OrbStack / Docker Desktop provide one by default, e.g. via QEMU emulation). +## Private yt-dlp build + +Until upstream yt-dlp fixes the extractor bug, the image bakes a wheel built +from a local patched checkout instead of the pinned PyPI release: + +- `./build-ytdlp.sh` builds the wheel from `~/Dev/github/yt-dlp` (override the + path with `YTDLP_REPO=...`) into `vendor/yt_dlp-*.whl`. +- The Dockerfile force-reinstalls that wheel over the version pinned in + `uv.lock` when the wheel is present; otherwise it silently falls back to the + pinned release. +- The wheel is git-ignored, so it is not committed. +- `publish.sh` runs `./build-ytdlp.sh` automatically and aborts if the local + checkout is missing. + +See **`YTDLP_PRIVATE_BUILD.md`** for full details, including a step-by-step +guide to switching back to the official build once the upstream fix lands. + ## Build & push (manual) ```sh +# Optional: stage the private yt-dlp wheel (see "Private yt-dlp build") +./build-ytdlp.sh + docker buildx build --platform linux/amd64 \ --build-arg VERSION=1.22 \ -t 192.168.2.212:3000/tigeren/metube:1.22 \ diff --git a/Dockerfile b/Dockerfile index 37d1157..e552d15 100644 --- a/Dockerfile +++ b/Dockerfile @@ -34,6 +34,17 @@ COPY pyproject.toml uv.lock ./ RUN --mount=type=cache,target=/root/.cache/uv \ UV_PROJECT_ENVIRONMENT=/usr/local uv sync --frozen --no-dev --compile-bytecode --extra headless +# Private yt-dlp build: bake the locally-patched wheel (built by build-ytdlp.sh +# from a local yt-dlp checkout) over the pinned PyPI release until the upstream +# fix lands. The wheel is git-ignored; vendor/.gitkeep keeps the dir present so +# this COPY always succeeds and the RUN is a no-op when no wheel is staged. +COPY vendor ./vendor +RUN if [ -n "$(ls /app/vendor/yt_dlp-*.whl 2>/dev/null || true)" ]; then \ + uv pip install --python /usr/local/bin/python --force-reinstall --no-deps /app/vendor/yt_dlp-*.whl; \ + else \ + echo "No private yt-dlp wheel in vendor/ - using pinned yt-dlp from uv.lock"; \ + fi + # Use sed to strip carriage-return characters from the entrypoint script (in case building on Windows) COPY docker-entrypoint.sh ./ RUN sed -i 's/\r$//g' docker-entrypoint.sh && chmod +x docker-entrypoint.sh diff --git a/YTDLP_PRIVATE_BUILD.md b/YTDLP_PRIVATE_BUILD.md new file mode 100644 index 0000000..ea7fdec --- /dev/null +++ b/YTDLP_PRIVATE_BUILD.md @@ -0,0 +1,126 @@ +# Private yt-dlp build + +MeTube uses [yt-dlp](https://github.com/yt-dlp/yt-dlp) as its core downloader, +pinned to the official PyPI release via `pyproject.toml` / `uv.lock`. + +There is currently an extractor bug in the official release that is not yet +fixed upstream. Until it is, the Docker image bakes a wheel built from a local, +patched yt-dlp checkout (`~/Dev/github/yt-dlp`) instead of the pinned release. + +This document explains how the override works, how to build/update the private +wheel, and — most importantly — **how to switch back to the official build once +the upstream fix lands**. + +--- + +## How the override works + +1. `./build-ytdlp.sh` builds a wheel from a local yt-dlp checkout and stages it + as `vendor/yt_dlp-*.whl` (the versioned filename is required by `uv`). +2. The `Dockerfile`, right after `uv sync`, force-reinstalls that wheel over the + version pinned in `uv.lock`: + + ```dockerfile + COPY vendor ./vendor + RUN if [ -n "$(ls /app/vendor/yt_dlp-*.whl 2>/dev/null || true)" ]; then \ + uv pip install --python /usr/local/bin/python --force-reinstall --no-deps /app/vendor/yt_dlp-*.whl; \ + else \ + echo "No private yt-dlp wheel in vendor/ - using pinned yt-dlp from uv.lock"; \ + fi + ``` + +3. `publish.sh` runs `./build-ytdlp.sh` before every build, so published images + always carry the patched build (and the publish aborts loudly if the local + checkout is missing). +4. The wheel is **git-ignored** (`/vendor/yt_dlp-*.whl`). `vendor/.gitkeep` and + `vendor/.gitignore` are committed so the directory always exists for + `COPY vendor ./vendor`. + +## Building / updating the private wheel + +```sh +./build-ytdlp.sh # builds from ~/Dev/github/yt-dlp +YTDLP_REPO=/path/to/yt-dlp ./build-ytdlp.sh # custom checkout path +``` + +Rebuild whenever the local checkout changes, then rebuild the image: + +```sh +docker compose up -d --build --force-recreate +``` + +The script fails with a clear error if the configured checkout is missing. + +--- + +## Switching back to the official build (when upstream fixes it) + +### 1. Remove the Dockerfile override + +Delete the `COPY vendor` / `RUN uv pip install` block from `Dockerfile` +(added in the "Private yt-dlp build" section, right after `uv sync`). + +### 2. Remove the build step from `publish.sh` + +Delete the block: + +```sh +# Bake the private yt-dlp build (see build-ytdlp.sh) into the image until the +# upstream fix lands. Fails loudly if the local checkout is missing. +./build-ytdlp.sh +``` + +### 3. Update the pinned version + +Bump `yt-dlp` to the fixed release and re-resolve the lockfile: + +```sh +uv add "yt-dlp[default,curl-cffi]@latest" # or pin the exact fixed version +uv lock +``` + +Commit the updated `pyproject.toml` and `uv.lock`. + +### 4. Clean up the override artifacts + +```sh +rm build-ytdlp.sh # no longer needed +rm -f vendor/yt_dlp-*.whl # delete any staged wheel +``` + +You may also remove the `vendor/` directory (`.gitkeep`, `.gitignore`) and the +`/vendor/yt_dlp-*.whl` entry from `.gitignore`. + +### 5. Update the docs + +Remove the "Private yt-dlp build" section from `DEPLOY.md` (and delete this file +if you no longer want the guide around). + +### 6. Rebuild and verify + +```sh +docker compose up -d --build --force-recreate +``` + +Confirm the installed version is the official release: + +```sh +docker compose exec metube python3 -c "import yt_dlp; print(yt_dlp.version.__version__)" +``` + +### 7. Publish + +When you next publish, `./publish.sh` will no longer stage a private wheel and +the image will use the official yt-dlp release. + +--- + +## Verifying which build is running + +The `/version` endpoint reports `METUBE_VERSION`; the yt-dlp version is logged +by the app at startup. To check inside a running container: + +```sh +docker compose exec metube python3 -c \ + "import yt_dlp, yt_dlp.extractor.spankbang as s; import inspect; print(yt_dlp.version.__version__, 'patched:', 'spankbang' in inspect.getsource(s).lower())" +``` diff --git a/build-ytdlp.sh b/build-ytdlp.sh new file mode 100755 index 0000000..9c9d071 --- /dev/null +++ b/build-ytdlp.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +# Build the private yt-dlp wheel (locally-patched build) and stage it for the +# Docker image. +# +# MeTube pins the official yt-dlp release via uv.lock. Until the upstream fix +# lands, a wheel built from a local yt-dlp checkout is baked into the image +# instead (see the Dockerfile). This script produces vendor/yt_dlp.whl. +# +# Usage: +# ./build-ytdlp.sh # uses ~/Dev/github/yt-dlp +# YTDLP_REPO=/path/to/yt-dlp ./build-ytdlp.sh + +set -euo pipefail +cd "$(dirname "$0")" + +YTDLP_REPO="${YTDLP_REPO:-$HOME/Dev/github/yt-dlp}" +OUT_DIR="$(pwd)/vendor" +mkdir -p "$OUT_DIR" + +if [[ ! -d "$YTDLP_REPO" ]]; then + echo "Error: local yt-dlp checkout not found at $YTDLP_REPO" >&2 + echo "Set YTDLP_REPO to your patched checkout, or drop the override (see DEPLOY.md)." >&2 + exit 1 +fi + +echo "Building yt-dlp wheel from $YTDLP_REPO ..." +(cd "$YTDLP_REPO" && uv build --wheel -o "$OUT_DIR") + +# Keep the original wheel filename (uv requires it to encode the version). +echo "Wheel ready:" +ls -1 "$OUT_DIR"/yt_dlp-*.whl diff --git a/publish.sh b/publish.sh index a9dca97..3b88a53 100755 --- a/publish.sh +++ b/publish.sh @@ -52,6 +52,10 @@ fi echo "Publishing $IMAGE:$NEW_VERSION (current: $CURRENT_VERSION)" +# Bake the private yt-dlp build (see build-ytdlp.sh) into the image until the +# upstream fix lands. Fails loudly if the local checkout is missing. +./build-ytdlp.sh + # Record the new version as the current online version. BSD sed has no `\b`, # so require a non-digit (or end of line) after the version to avoid matching # e.g. 1.1 inside 1.10. diff --git a/vendor/.gitignore b/vendor/.gitignore new file mode 100644 index 0000000..377ccd3 --- /dev/null +++ b/vendor/.gitignore @@ -0,0 +1,2 @@ +* +!.gitkeep diff --git a/vendor/.gitkeep b/vendor/.gitkeep new file mode 100644 index 0000000..e69de29