Add artwork self-hosting: local-disk download worker + storage
CI / lint (push) Canceled after 0s
CI / typecheck (push) Canceled after 0s
CI / unit-tests (push) Canceled after 0s
CI / integration-tests (push) Canceled after 0s
CI / container-build (push) Canceled after 0s
CI / dependency-audit (push) Canceled after 0s
CI / lint (push) Canceled after 0s
CI / typecheck (push) Canceled after 0s
CI / unit-tests (push) Canceled after 0s
CI / integration-tests (push) Canceled after 0s
CI / container-build (push) Canceled after 0s
CI / dependency-audit (push) Canceled after 0s
Section 10: every image API responses reference has pointed straight at TMDb's CDN since day one -- Aurora never actually downloaded or served artwork itself, despite storing dimensions/checksum/local_path columns for exactly that. Closes the gap: - is_primary was declared on the Artwork model but never actually set anywhere -- every row defaulted to False. Now the normalizer marks the one image each provider designates as *the* image for an entity (TMDb's poster_path/backdrop_path/profile_path) as primary, distinct from the dozens of alternates also ingested, including correctly marking an existing row primary (not inserting a duplicate) when that image already appears in the alternates list. - app/services/artwork/: an ArtworkStorage interface with a LocalDiskStorage implementation (checksum-named files, single-host deployment via a Docker volume shared between api and worker -- see docs/architecture.md for the S3/R2 upgrade path), and a download pipeline with content-type validation, a size cap, and path-traversal protection. - After movie/series/person ingestion succeeds, tasks.py cascades one download_artwork_task per primary artwork id via arq's ctx['redis'] (same fan-out pattern as the season cascade) -- fire-and-forget, not tracked in ingestion_jobs, since a failed image cache is a known-safe fallback to the provider URL, not a user-facing failure. - app/main.py serves MEDIA_ROOT at MEDIA_URL_PREFIX; local_path is preferred over original_url once set (already-existing fallback logic in app/services/common.py:artwork_url, now actually exercised). Also fixes a real interaction bug this surfaced: every ingestion pipeline deleted and recreated all of an entity's Artwork rows on each re-sync, which would have silently wiped local_path/checksum on every re-ingestion, forcing a full re-download every time and defeating the point of caching locally at all. app/repositories/polymorphic.py's new sync_artwork upserts by provider_asset_id (TMDb's stable per-image path) instead, preserving downloaded copies across re-sync, and replaces three near-identical delete+recreate helpers that used to live separately in movie_ingestion.py, series_ingestion.py, and person_ingestion.py. Co-Authored-By: Claude Sonnet 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
cd4dd98adb
commit
fef914db36
@@ -90,6 +90,40 @@ star can race to create the same `Person` row (see
|
||||
recovers from that specific race directly; the arq retry here is the
|
||||
backstop for whatever it doesn't).
|
||||
|
||||
## Artwork self-hosting (section 10)
|
||||
|
||||
```
|
||||
ingest_movie/ingest_series/ingest_person succeeds
|
||||
-> tasks.py: get_primary_artwork_for_download(entity_type, [entity_id, ...])
|
||||
-> one download_artwork_task enqueued per primary artwork id
|
||||
-> download_artwork(): fetch original_url, verify it's an image, cap
|
||||
size, sha256 it, storage.save(), set Artwork.local_path + checksum
|
||||
```
|
||||
|
||||
Every image API responses reference is stored with its provider CDN URL
|
||||
(`original_url`) from the moment it's ingested; `local_path` starts empty
|
||||
and is filled in by this pipeline running asynchronously afterward, so
|
||||
responses transparently point at TMDb until a local copy exists, then
|
||||
switch over. Only artwork marked `is_primary` (the one image TMDb itself
|
||||
designates for an entity, not the dozens of alternates also ingested) gets
|
||||
downloaded automatically — see `docs/database.md#artwork-storage-self-hosting`.
|
||||
|
||||
Storage is behind an `ArtworkStorage` interface
|
||||
(`app/services/artwork/storage.py`) with one implementation,
|
||||
`LocalDiskStorage` — a shared Docker volume between the `api` and `worker`
|
||||
containers (`docker-compose.yml`'s `aurora_media_data`), which is enough
|
||||
for this project's actual deployment shape (one Docker host, principle 9:
|
||||
avoid premature complexity) but assumes exactly that. A deployment that
|
||||
outgrew a single host would implement the same interface against S3/R2
|
||||
instead, without the download pipeline or callers changing.
|
||||
|
||||
This is also why `app/repositories/polymorphic.py:sync_artwork` upserts
|
||||
artwork by `provider_asset_id` instead of the delete-then-recreate pattern
|
||||
every other satellite table uses (movie/series/person credits, videos,
|
||||
genre/keyword/company links): a delete-then-recreate would generate a fresh
|
||||
`Artwork.id` with `local_path=None` on every re-ingestion, silently
|
||||
discarding every downloaded copy and forcing a full re-download each sync.
|
||||
|
||||
## Search (section 16)
|
||||
|
||||
`app/services/search/base.py:SearchProvider` is the abstraction; API
|
||||
|
||||
+43
-12
@@ -71,6 +71,31 @@ externally-sourced images can be freely redistributed. Aurora never
|
||||
downloads video media; `videos.url` is a reference to the provider's
|
||||
hosted copy (YouTube, Vimeo, ...).
|
||||
|
||||
### Artwork storage (self-hosting)
|
||||
|
||||
`artwork.original_url` (the provider's CDN link) is always populated by
|
||||
ingestion; `artwork.local_path` starts `NULL` and is filled in
|
||||
asynchronously by `app/services/artwork/`. `app/services/common.py:artwork_url()`
|
||||
prefers `local_path` and falls back to `original_url`, so API responses
|
||||
point at TMDb's CDN until a copy has been downloaded, then switch over
|
||||
automatically once it has.
|
||||
|
||||
Only artwork marked `is_primary` gets downloaded automatically — the single
|
||||
image the provider designates as *the* image for an entity (TMDb's
|
||||
`poster_path`/`backdrop_path`/`profile_path`), not the dozens of alternates
|
||||
also stored. `app/services/ingestion/runner.py:get_primary_artwork_for_download`
|
||||
finds them and `app/workers/tasks.py` enqueues a `download_artwork_task` per
|
||||
id once movie/series/person/season ingestion succeeds — see
|
||||
`docs/architecture.md` for the full flow.
|
||||
|
||||
Storage is local disk (`app/services/artwork/storage.py:LocalDiskStorage`)
|
||||
behind an `ArtworkStorage` interface, deliberately simple for a
|
||||
single-Docker-host deployment where the `api` and `worker` containers share
|
||||
one volume (`docker-compose.yml`'s `aurora_media_data`). A deployment that
|
||||
outgrows one host would need an S3/R2-backed implementation of the same
|
||||
interface instead. Files are named by content (`<type>/<sha256[:2]>/<sha256>.<ext>`),
|
||||
so the same image referenced by two entities is only ever stored once.
|
||||
|
||||
## Polymorphic associations
|
||||
|
||||
`credits`, `artwork`, `videos`, `external_ids`, and `release_dates` attach to
|
||||
@@ -120,14 +145,15 @@ same lowercase strings.
|
||||
|
||||
Alembic (section 27); the application **never** creates tables at startup —
|
||||
see `app/core/database.py`. The initial migration
|
||||
(`migrations/versions/0001_initial_schema.py`) is hand-written rather than
|
||||
autogenerated (no throwaway Postgres was available in the environment this
|
||||
was built in — see the root README's Known Limitations), but was written
|
||||
directly from the same models this document describes and validated with
|
||||
`alembic history`; treat a first `alembic upgrade head` against a scratch
|
||||
database as the real acceptance test, and autogenerate future migrations
|
||||
against it (`alembic revision --autogenerate`) rather than the empty
|
||||
database this one was written from.
|
||||
(`migrations/versions/0001_initial_schema.py`) was hand-written rather than
|
||||
autogenerated (no throwaway Postgres was available in the environment it
|
||||
was originally written in), then verified by actually running
|
||||
`alembic upgrade head` against a genuinely empty PostgreSQL on a real
|
||||
devbox — that run caught a real bug (pre-creating enum types and then also
|
||||
letting `CREATE TABLE` auto-create them raised `DuplicateObject`; fixed by
|
||||
`create_type=False` on each enum), now fixed and verified clean. Autogenerate
|
||||
future migrations against a database that already has this one applied
|
||||
(`alembic revision --autogenerate`), not from scratch.
|
||||
|
||||
```bash
|
||||
alembic upgrade head # apply all migrations
|
||||
@@ -159,10 +185,15 @@ indexes on the alternate-title tables. See
|
||||
- Field-level provenance (which provider supplied *this* value) is not
|
||||
tracked, only entity-level (`source_provider_id`) and per-provider raw
|
||||
payloads (`provider_records.raw_data`).
|
||||
- Satellite associations (credits, artwork, videos, genre/keyword/company
|
||||
links) are replaced wholesale on each re-sync rather than diffed, so their
|
||||
Aurora IDs are not stable across re-ingestion — see
|
||||
`app/services/ingestion/movie_ingestion.py`.
|
||||
- Satellite associations (credits, videos, genre/keyword/company links) are
|
||||
replaced wholesale on each re-sync rather than diffed, so their Aurora IDs
|
||||
are not stable across re-ingestion — see
|
||||
`app/services/ingestion/movie_ingestion.py`. **Artwork is the exception:**
|
||||
`app/repositories/polymorphic.py:sync_artwork` upserts by
|
||||
`provider_asset_id` instead, specifically so a previously-downloaded
|
||||
`local_path`/`checksum` (see "Artwork storage" below) survives
|
||||
re-ingestion instead of being silently reset to `None` and re-downloaded
|
||||
every sync.
|
||||
- Seasons don't get their own `external_ids` row (identified instead by the
|
||||
natural `(series_id, season_number)` key) — episodes do, since TMDb gives
|
||||
each episode a distinct global id. Revisit if a provider ever needs a
|
||||
|
||||
Reference in New Issue
Block a user