NixReaperandClaude Sonnet 5
fef914db36
CI / lint (push) Canceled after 0s
CI / typecheck (push) Canceled after 0s
CI / unit-tests (push) Canceled after 0s
CI / integration-tests (push) Canceled after 0s
CI / container-build (push) Canceled after 0s
CI / dependency-audit (push) Canceled after 0s
Section 10: every image API responses reference has pointed straight at TMDb's CDN since day one -- Aurora never actually downloaded or served artwork itself, despite storing dimensions/checksum/local_path columns for exactly that. Closes the gap: - is_primary was declared on the Artwork model but never actually set anywhere -- every row defaulted to False. Now the normalizer marks the one image each provider designates as *the* image for an entity (TMDb's poster_path/backdrop_path/profile_path) as primary, distinct from the dozens of alternates also ingested, including correctly marking an existing row primary (not inserting a duplicate) when that image already appears in the alternates list. - app/services/artwork/: an ArtworkStorage interface with a LocalDiskStorage implementation (checksum-named files, single-host deployment via a Docker volume shared between api and worker -- see docs/architecture.md for the S3/R2 upgrade path), and a download pipeline with content-type validation, a size cap, and path-traversal protection. - After movie/series/person ingestion succeeds, tasks.py cascades one download_artwork_task per primary artwork id via arq's ctx['redis'] (same fan-out pattern as the season cascade) -- fire-and-forget, not tracked in ingestion_jobs, since a failed image cache is a known-safe fallback to the provider URL, not a user-facing failure. - app/main.py serves MEDIA_ROOT at MEDIA_URL_PREFIX; local_path is preferred over original_url once set (already-existing fallback logic in app/services/common.py:artwork_url, now actually exercised). Also fixes a real interaction bug this surfaced: every ingestion pipeline deleted and recreated all of an entity's Artwork rows on each re-sync, which would have silently wiped local_path/checksum on every re-ingestion, forcing a full re-download every time and defeating the point of caching locally at all. app/repositories/polymorphic.py's new sync_artwork upserts by provider_asset_id (TMDb's stable per-image path) instead, preserving downloaded copies across re-sync, and replaces three near-identical delete+recreate helpers that used to live separately in movie_ingestion.py, series_ingestion.py, and person_ingestion.py. Co-Authored-By: Claude Sonnet 5 <[email protected]>
104 lines
4.2 KiB
Python
104 lines
4.2 KiB
Python
"""arq task functions. Thin wrappers around the ingestion runner so retry/
|
|
concurrency/observability behavior lives in arq's worker loop, not here.
|
|
|
|
Each ingest_* task receives the `IngestionJob` row's id (created
|
|
synchronously by the enqueuing endpoint via `get_or_create_ingestion_job`,
|
|
*not* generated here) so status polling has a real, immediately-valid id to
|
|
look up from the moment the HTTP response comes back. `download_artwork_task`
|
|
is the exception — it's an internal side effect of ingestion (see
|
|
`_cascade_artwork_downloads`), not something a client triggers or polls, so
|
|
it isn't tracked in `ingestion_jobs` at all; arq's own per-function retries
|
|
are enough for it (section 10: artwork self-hosting is best-effort caching,
|
|
not a user-facing operation with a status to report).
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import uuid
|
|
from typing import Any
|
|
|
|
from app.core.database import session_scope
|
|
from app.core.logging import get_logger
|
|
from app.services.artwork.download import download_artwork
|
|
from app.services.artwork.storage import get_artwork_storage
|
|
from app.services.ingestion.runner import (
|
|
get_or_create_ingestion_job,
|
|
get_primary_artwork_for_download,
|
|
get_series_season_ids,
|
|
get_series_season_numbers,
|
|
run_movie_ingestion,
|
|
run_person_ingestion,
|
|
run_season_ingestion,
|
|
run_series_ingestion,
|
|
)
|
|
|
|
logger = get_logger(__name__)
|
|
|
|
|
|
async def _cascade_artwork_downloads(
|
|
ctx: dict[str, Any], entity_type: str, entity_ids: list[uuid.UUID]
|
|
) -> None:
|
|
artwork_ids = await get_primary_artwork_for_download(entity_type, entity_ids)
|
|
if not artwork_ids:
|
|
return
|
|
redis = ctx["redis"]
|
|
for artwork_id in artwork_ids:
|
|
await redis.enqueue_job("download_artwork_task", str(artwork_id))
|
|
logger.info(
|
|
"artwork_downloads_cascaded", entity_type=entity_type, count=len(artwork_ids)
|
|
)
|
|
|
|
|
|
async def ingest_movie_task(ctx: dict[str, Any], job_id: str, external_id: str) -> str:
|
|
movie_id = await run_movie_ingestion(uuid.UUID(job_id), external_id)
|
|
await _cascade_artwork_downloads(ctx, "movie", [movie_id])
|
|
return str(movie_id)
|
|
|
|
|
|
async def ingest_person_task(ctx: dict[str, Any], job_id: str, external_id: str) -> str:
|
|
person_id = await run_person_ingestion(uuid.UUID(job_id), external_id)
|
|
await _cascade_artwork_downloads(ctx, "person", [person_id])
|
|
return str(person_id)
|
|
|
|
|
|
async def ingest_series_task(
|
|
ctx: dict[str, Any], job_id: str, external_id: str, include_seasons: bool = False
|
|
) -> str:
|
|
series_id = await run_series_ingestion(uuid.UUID(job_id), external_id)
|
|
await _cascade_artwork_downloads(ctx, "series", [series_id])
|
|
# Season posters come from the season *stubs* ingest_series already
|
|
# creates, independent of include_seasons (which only controls whether
|
|
# full per-season episode data gets fetched) -- so this always runs.
|
|
await _cascade_artwork_downloads(ctx, "season", await get_series_season_ids(series_id))
|
|
|
|
if include_seasons:
|
|
# Fan out one ingest_season job per season this series has, using
|
|
# the same arq connection this task is running on (arq hands every
|
|
# job's ctx a "redis" entry precisely for chaining/fan-out like
|
|
# this) — not a second, unrelated connection.
|
|
redis = ctx["redis"]
|
|
for season_number in await get_series_season_numbers(series_id):
|
|
season_job_id = await get_or_create_ingestion_job(
|
|
"season_sync", "season", f"{external_id}:{season_number}"
|
|
)
|
|
await redis.enqueue_job(
|
|
"ingest_season_task", str(season_job_id), external_id, season_number
|
|
)
|
|
logger.info(
|
|
"series_seasons_cascaded", series_id=str(series_id), external_id=external_id
|
|
)
|
|
|
|
return str(series_id)
|
|
|
|
|
|
async def ingest_season_task(
|
|
ctx: dict[str, Any], job_id: str, series_external_id: str, season_number: int
|
|
) -> str:
|
|
season_id = await run_season_ingestion(uuid.UUID(job_id), series_external_id, season_number)
|
|
return str(season_id)
|
|
|
|
|
|
async def download_artwork_task(ctx: dict[str, Any], artwork_id: str) -> bool:
|
|
async with session_scope() as session:
|
|
return await download_artwork(session, get_artwork_storage(), uuid.UUID(artwork_id))
|