13 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
NixReaperandClaude Sonnet 5	fef914db36	Add artwork self-hosting: local-disk download worker + storage ... Section 10: every image API responses reference has pointed straight at TMDb's CDN since day one -- Aurora never actually downloaded or served artwork itself, despite storing dimensions/checksum/local_path columns for exactly that. Closes the gap: - is_primary was declared on the Artwork model but never actually set anywhere -- every row defaulted to False. Now the normalizer marks the one image each provider designates as *the* image for an entity (TMDb's poster_path/backdrop_path/profile_path) as primary, distinct from the dozens of alternates also ingested, including correctly marking an existing row primary (not inserting a duplicate) when that image already appears in the alternates list. - app/services/artwork/: an ArtworkStorage interface with a LocalDiskStorage implementation (checksum-named files, single-host deployment via a Docker volume shared between api and worker -- see docs/architecture.md for the S3/R2 upgrade path), and a download pipeline with content-type validation, a size cap, and path-traversal protection. - After movie/series/person ingestion succeeds, tasks.py cascades one download_artwork_task per primary artwork id via arq's ctx['redis'] (same fan-out pattern as the season cascade) -- fire-and-forget, not tracked in ingestion_jobs, since a failed image cache is a known-safe fallback to the provider URL, not a user-facing failure. - app/main.py serves MEDIA_ROOT at MEDIA_URL_PREFIX; local_path is preferred over original_url once set (already-existing fallback logic in app/services/common.py:artwork_url, now actually exercised). Also fixes a real interaction bug this surfaced: every ingestion pipeline deleted and recreated all of an entity's Artwork rows on each re-sync, which would have silently wiped local_path/checksum on every re-ingestion, forcing a full re-download every time and defeating the point of caching locally at all. app/repositories/polymorphic.py's new sync_artwork upserts by provider_asset_id (TMDb's stable per-image path) instead, preserving downloaded copies across re-sync, and replaces three near-identical delete+recreate helpers that used to live separately in movie_ingestion.py, series_ingestion.py, and person_ingestion.py. Co-Authored-By: Claude Sonnet 5 <[email protected]>	2026-09-05 03:26:28 -05:00
NixReaperandClaude Sonnet 5	cd4dd98adb	Make failed ingestion jobs actually retry, not just say "retrying" ... arq only auto-retries a job on its own Retry exception (or CancelledError) -- never on an arbitrary exception a task raises. _run_ingestion caught failures, recorded them via mark_job_failed (which sets ingestion_jobs.status to "retrying" when attempts hasn't hit max_attempts), and then just re-raised the original exception -- which arq treats as a terminal failure. The DB said "retrying" forever; nothing ever put the job back on the queue. Confirmed live: cascading Better Call Saul's 6 seasons hit the get_or_create race fixed in the previous commit on 4 of them, and the worker's own shutdown summary showed "4 failed, 0 retries" -- every one of those was permanently lost despite max_attempts=5 suggesting otherwise. _run_ingestion now raises arq.Retry (capped exponential backoff) when mark_job_failed left the job in RETRYING state, so arq actually requeues it; only once attempts are exhausted does the original exception propagate as a genuine terminal failure. Co-Authored-By: Claude Sonnet 5 <[email protected]>	2026-09-05 02:31:43 -05:00
NixReaperandClaude Sonnet 5	dd3ff9764b	Fix race condition in get_or_create_* under concurrent ingestion ... Every get_or_create_* helper (genre, keyword, language, country, company, collection, person) followed check-then-act: look up by natural key, insert if missing. That was safe as long as nothing ingested concurrently for overlapping data -- which stopped being true the moment ingest_series_task could cascade season ingestion (include_seasons), since arq runs multiple queued jobs concurrently. Two seasons of the same series sharing a guest star, crew member, or company can now race to create it: both see "not found", both try to INSERT, and the loser hits a unique-constraint violation instead of finding the winner's row. Reproduced live: cascading Better Call Saul (60059) with include_seasons crashed a season job with "duplicate key value violates unique constraint uq_external_id_source" on a recurring guest actor shared across two seasons ingesting at the same time. _create_or_recover wraps each insert in a SAVEPOINT and, on that specific failure, rolls back just the savepoint and re-queries for what the concurrent job committed -- recovering instead of aborting the whole ingestion job. Added unit tests for the retry mechanism itself; genuinely reproducing the underlying race in an automated test would need real Postgres MVCC timing, which isn't something a deterministic test can force reliably. Co-Authored-By: Claude Sonnet 5 <[email protected]>	2026-09-05 02:27:07 -05:00
NixReaperandClaude Sonnet 5	64169c4aee	Fix test factory Provider collision from ingestion-cascade tests ... tests/integration/factories.py:make_provider did a raw INSERT assuming no "tmdb" Provider row existed yet. That held as long as every test used the db_session fixture's isolated, rolled-back transaction -- but the new ingestion-cascade tests call get_or_create_ingestion_job, which really commits via session_scope()'s own connection, outside that per-test isolation. Once that ran, every later test using the raw-insert factory hit "duplicate key value violates unique constraint uq_providers_slug". Fixed by using the app's own get_or_create_provider in the factory instead of a bare insert, which is also just correct: provider slugs are unique in production too, so the test helper should behave the same way real ingestion code does rather than assume a pristine table. Co-Authored-By: Claude Sonnet 5 <[email protected]>	2026-09-05 02:21:43 -05:00
NixReaperandClaude Sonnet 5	b4d1e8f755	Add opt-in season cascade to series ingestion ... Ingesting a series only ever produced season stubs -- getting actual episodes meant calling POST /admin/ingest/season once per season by hand, which is exactly what manual end-to-end testing kept requiring for every multi-season show. POST /api/v1/admin/ingest/series now takes include_seasons (default false, since a show can have dozens of seasons and that's dozens of TMDb calls fired from one request). When true, ingest_series_task fans out one ingest_season job per season once the series succeeds, using arq's own ctx['redis'] connection for the job (the same pattern arq's docs recommend for job chaining) rather than a second unrelated connection. Each fanned-out season job still goes through get_or_create_ingestion_job, so re-running a cascaded series ingestion reuses the same season job rows instead of creating duplicates. Co-Authored-By: Claude Sonnet 5 <[email protected]>	2026-09-05 02:03:25 -05:00
NixReaperandClaude Sonnet 5	ec6b5bdd7d	Surface series/episode credits in person filmography ... /people/{id}/credits only ever looked at entity_type == "movie", silently dropping the series-level cast/crew and episode-level guest-star Credit rows that series_ingestion.py has been writing all along -- a person ingested via a TV show would show an empty or movie-only filmography no matter how much TV work was actually in the database. Regular series credits map straight to a filmography row, same shape as movies. Guest-star episode credits are aggregated to one row per (series, department/job/character) with an episode_count, rather than one row per episode -- a prolific guest actor would otherwise flood the list with dozens of near-identical entries. Added tv_repository.get_series_by_ids/get_episodes_by_ids (bulk, mirroring the existing movie_repository pattern) to support it. Co-Authored-By: Claude Sonnet 5 <[email protected]>	2026-09-04 20:00:15 -05:00
NixReaperandClaude Sonnet 5	e6841badf8	Fix worker never picking up code changes after restart ... docker compose restart worker looked like it should reflect new code (docker-compose.yml bind-mounts ./app:/app/app for dev), but the worker Dockerfile installed the app package non-editably (pip install .), which copies a frozen snapshot into site-packages at image build time. The bare `arq` console-script just imports whatever site-packages says `app` is -- unlike uvicorn --reload, which explicitly resolves its import string against the CWD -- so the worker silently kept running whatever code existed at the last `docker compose up --build`, no matter how many times it was restarted or how recent the git pull was. Confirmed live: after pulling the series/season ingestion pipeline and restarting, the worker log still read "Starting worker for 2 functions: ingest_movie_task, ingest_person_task" -- proof it never saw ingest_series_task/ingest_season_task at all. Switching to `pip install -e .` makes site-packages point at /app by absolute path instead of a frozen copy, so the same bind mount that already worked for the API now actually reaches the worker process too -- a plain restart after `git pull` is enough going forward. Applied to both Dockerfiles for consistency, since the API's working behavior was incidental to uvicorn, not something to rely on either. Co-Authored-By: Claude Sonnet 5 <[email protected]>	2026-09-04 19:51:52 -05:00
NixReaperandClaude Sonnet 5	3acf00e9f0	Fix test bug: await inside a generator expression passed to all() ... Python turns a generator expression containing await into an async generator (since it's evaluated inside an async def), which the built-in all() can't consume -- it raised "TypeError: 'async_generator' object is not iterable" rather than testing anything. Replaced with a plain loop. Found by actually running the new integration tests. Co-Authored-By: Claude Sonnet 5 <[email protected]>	2026-09-04 19:45:29 -05:00
NixReaperandClaude Sonnet 5	3f6332559a	Add series/season/episode ingestion pipeline ... Movie and person ingestion existed, but the series/season/episode API endpoints had no way to actually populate them from TMDb -- the biggest remaining gap in Core 0.1's television scope. - normalize_series/normalize_season in the metadata normalizer, including two TMDb quirks that would otherwise silently produce empty data: /tv nests keywords and alternative_titles under "results" (not "keywords"/"titles" like /movie), and series-level creators come from a separate created_by array, not the credits cast/crew lists. - ingest_series upserts the Series row plus season *stubs* (number, title, air date, poster -- everything TMDb inlines on /tv/{id}); ingest_season fetches and upserts one season's full episode list, and requires the series to already exist (TMDb has no single endpoint for a whole series' episodes, so this has to be a separate per-season call). - Wired through the same job-lifecycle path as movie/person ingestion (runner.py, arq tasks, admin endpoints), with the shared boilerplate between them factored into runner._run_ingestion and admin.py:_enqueue rather than copy-pasted a third and fourth time. - Unit tests for both normalizers (runnable without a DB) and integration tests for the full ingest_series -> ingest_season flow, including the series-must-exist-first error and idempotency on re-ingestion. Co-Authored-By: Claude Sonnet 5 <[email protected]>	2026-09-04 19:43:56 -05:00
NixReaperandClaude Sonnet 5	669034896a	Fix job_id mismatch between ingestion enqueue and status endpoints ... POST /api/v1/admin/ingest/movie returned arq's own internal queue job id, but GET /api/v1/admin/ingest/jobs/{job_id} looked up our IngestionJob DB row by its own UUID -- a completely different id that didn't even exist yet at enqueue time (the row was only created once the worker picked up the task). Every status poll 404'd. Fix: create the IngestionJob row synchronously in the enqueue endpoint (reusing the existing idempotency-key logic, so repeat enqueues for the same tmdb_id reuse the same row rather than creating duplicates), pass its real id into the arq task, and have the worker update that row instead of creating a new one. Found by actually running the milestone end-to-end: enqueue a real TMDb ingestion, then poll its status. Co-Authored-By: Claude Sonnet 5 <[email protected]>	2026-09-04 19:28:31 -05:00
NixReaperandClaude Sonnet 5	dd6ecce78d	Fix asyncpg cross-event-loop errors in the integration test suite ... tests/conftest.py's `engine` fixture is session-scoped (one asyncpg engine/schema for the whole run, for speed), but pytest-asyncio's default is a fresh event loop per test function. asyncpg connections are bound to the event loop that created them, so every test after the first failed with "cannot perform operation: another operation is in progress" or "attached to a different loop" the moment it touched the database. Pinning both the fixture and test loop scope to "session" in pyproject.toml keeps everything on one loop, matching the session-scoped engine. Found by actually running the integration suite for the first time, against real PostgreSQL + Redis on a devbox. Co-Authored-By: Claude Sonnet 5 <[email protected]>	2026-09-04 19:23:18 -05:00
NixReaperandClaude Sonnet 5	5b38226818	Fix duplicate enum type creation in initial migration ... Postgres ENUM columns auto-create their type as a side effect of the first CREATE TABLE that uses them; since the migration also explicitly pre-creates all 7 enum types up front (to group and control that DDL alongside their explicit drop in downgrade()), the two collided with "type already exists" on the first table past the first enum's owner. create_type=False on each ENUM tells SQLAlchemy the migration manages that DDL itself. Found by running the migration against a real PostgreSQL for the first time (docker compose up on a devbox) — this repo had no Docker/Postgres available to test against when the migration was originally written. Co-Authored-By: Claude Sonnet 5 <[email protected]>	2026-09-04 18:48:44 -05:00
NixReaperandClaude Sonnet 5	c20df1c503	Initial Aurora Core 0.1 implementation ... Production-grade media metadata backend per docs/aurora-core-spec.md: Aurora-owned UUIDv7 identity system, normalized movie/TV/people/credits schema, provider abstraction with a TMDb adapter, an idempotent async ingestion pipeline (normalize -> match -> upsert -> provenance), a Postgres FTS/trigram search layer, a stateless FastAPI v1 API with Redis caching/rate-limiting, arq workers, dev/prod Docker Compose topologies, unit + integration + load tests, CI, and architecture/ database/API/ERD documentation. Co-Authored-By: Claude Sonnet 5 <[email protected]>	2026-09-04 17:34:03 -05:00