11 Commits

Include renames

Author	SHA1	Message	Date
NixReaperandClaude Sonnet 5	6c95e18aad	Add watch providers (where to stream/rent/buy) ... New TMDb integration end to end: append_to_response "watch/providers", normalizer parsing grouped by country/monetization type, a WatchProvider catalog entry (external-id keyed, same get_or_create pattern as Company/ Collection/Person) + polymorphic WatchProviderOffering table (new migration 0002), ingestion write (including provider logos via the existing Artwork table), and API exposure -- embedded on movie/series detail plus a dedicated /watch-providers sub-route, shaped to mirror TMDb's own response for easy porting. Note: this data comes from JustWatch via TMDb and carries its own attribution requirement separate from TMDb's general one -- flagged in docs/api.md, must be satisfied before showing this in the frontend. Co-Authored-By: Claude Sonnet 5 <[email protected]>	2026-09-06 04:59:02 -05:00
NixReaperandClaude Sonnet 5	567b4ef068	Add keyword browsing/filtering and expose alternate titles ... Keywords were ingested and shown read-only on detail pages, but had no directory (GET /keywords) or filter param -- the same gap genres had before today. Mirrors the genre browse pattern exactly: repository, service, route, plus a `keyword` filter on GET /movies and GET /series. Alternate/translated titles ("Also Known As") were normalized down to bare title strings and never written to MovieAlternateTitle/ SeriesAlternateTitle at all. Normalizer now captures country_code + type per TMDb's actual shape; ingestion writes them; movie/series detail now expose an alternate_titles list. Co-Authored-By: Claude Sonnet 5 <[email protected]>	2026-09-06 04:40:32 -05:00
NixReaperandClaude Sonnet 5	e1a36ce227	Expose certifications and per-country release dates ... TMDb's release_dates/content_ratings blocks were already fetched via append_to_response but discarded at normalization -- every downstream layer (ingestion write, repository read, schema, service) was unbuilt. Adds the full pipeline: NormalizedReleaseDate + get_or_create_certification, a shared polymorphic sync/read (movie and series both use it), and MovieDetailOut/SeriesDetailOut certification + release_dates fields. Series content_ratings has no date of its own, so each rating is anchored to the show's first_air_date with release_type="tv". Co-Authored-By: Claude Sonnet 5 <[email protected]>	2026-09-06 04:07:18 -05:00
NixReaperandClaude Sonnet 5	b05d34db5f	Fix bulk_import.py stalling on real-scale exports (1M+ ids) ... asyncio.gather(*(_worker(id) for id in ids)) eagerly wraps every argument into a Task before awaiting any of them -- with a semaphore bounding concurrent *execution* but not concurrent Task *creation*, running this against the real movie export (1,240,787 ids) meant constructing 1.24 million live Task + coroutine objects up front, which looked indistinguishable from hanging for minutes before any work could start, confirmed live in production tonight. Replaced with a fixed pool of _CONCURRENCY (50) consumer tasks pulling from a plain asyncio.Queue pre-loaded with every id -- only 50 Task objects ever exist regardless of catalog size. Verified: 500K enqueues against no-op stubs now complete in 0.59s (previously: the setup loop alone dominated), and the real bottleneck is correctly back to the per-id DB round-trip + Redis push, gated by the pool of 50 -- not Python-level Task scheduling overhead. Co-Authored-By: Claude Sonnet 5 <[email protected]>	2026-09-05 23:08:37 -05:00
NixReaperandClaude Sonnet 5	a52ac676ff	Add full-catalog bulk import from TMDb's daily ID exports ... app/services/bulk_import.py fetches and parses TMDb's daily ID export files (published once a day specifically for full-catalog sync, unlike /discover which caps at 10,000 results per query) -- today's file with a fallback to yesterday's, since TMDb publishes at a fixed time and "today's" may not exist yet. app/scripts/bulk_import.py is a standalone ops script (not an arq task, since enqueueing millions of jobs would blow past arq's job_timeout) that fans every id from the export into the existing per-entity ingestion pipeline -- get_or_create_ingestion_job + the same arq task each admin endpoint already uses, so a single id's behavior is identical whether it came from the admin API or this script. Bounded concurrency (50) on the enqueue phase; idempotency-keyed jobs make interrupting and re-running the script safe. Real TMDb catalog size (checked live): ~1.24M movies, 231K series, 4.9M people, 9.8K collections -- ~6.4M entities. At default worker concurrency this is realistically weeks of continuous processing, not hours, documented as such in README.md. Also added 429 (rate-limited) handling to TMDbProvider before running this at volume: previously a 429 fell into the generic >=400 branch and failed the request outright with zero retry, since only httpx.TransportError triggered the existing retry decorator. Now a 429 raises ProviderRateLimitedError carrying the response's own Retry-After hint, and tenacity's wait strategy honors that hint directly instead of guessing via exponential backoff. Co-Authored-By: Claude Sonnet 5 <[email protected]>	2026-09-05 22:12:12 -05:00
NixReaperandClaude Sonnet 5	892f78016c	Add collection/franchise ingestion ... Collections were bare stubs (name only) lazily created the moment a movie's belongs_to_collection reference was seen. Adds a real ingest_collection pipeline: TMDb's /collection/{id} for overview and poster/backdrop artwork, plus ordered CollectionMovie membership -- linking only movies already ingested into Aurora, the same never-fabricate-a-stub rule movie recommendations already follow. get_or_create_collection now refreshes name/overview on an existing stub instead of only creating, guarded so a bare movie-stub call (overview=None) never clobbers real data a full ingestion already filled in. New POST /api/v1/admin/ingest/collection endpoint and ingest_collection_task; GET /api/v1/collections/{id} needed no changes since its read path (collection_service.py) already assembled the full detail shape. Co-Authored-By: Claude Sonnet 5 <[email protected]>	2026-09-05 08:37:27 -05:00
NixReaperandClaude Sonnet 5	c03a75989d	Ingest and self-host company/network logos ... CompanyBriefOut.logo_path and CompanyDetailOut.logo_path have existed in the API schemas since Core 0.1's first pass, but nothing ever actually populated them -- TMDb's production_companies/networks entries carry a logo_path field that the normalizer simply never read, so every response has silently returned null there. - NormalizedCompanyRef now captures logo_path from both movie production_companies and series networks/production_companies. - normalizer.company_logo_artwork() builds the single-image artwork entry for it (a company has exactly one logo ingested, so it's always is_primary) -- centralized there rather than duplicated in both ingestion modules, matching how movie/series artwork URLs are already built in one place. - movie_ingestion.py and series_ingestion.py sync it via the same sync_artwork() the rest of section 10's self-hosting already uses, so it gets the same re-ingestion-safe upsert behavior for free. - tasks.py cascades a download_artwork_task per referenced company after movie/series ingestion, same fan-out pattern as movie/series/season/person artwork. Co-Authored-By: Claude Sonnet 5 <[email protected]>	2026-09-05 07:46:30 -05:00
NixReaperandClaude Sonnet 5	fef914db36	Add artwork self-hosting: local-disk download worker + storage ... Section 10: every image API responses reference has pointed straight at TMDb's CDN since day one -- Aurora never actually downloaded or served artwork itself, despite storing dimensions/checksum/local_path columns for exactly that. Closes the gap: - is_primary was declared on the Artwork model but never actually set anywhere -- every row defaulted to False. Now the normalizer marks the one image each provider designates as *the* image for an entity (TMDb's poster_path/backdrop_path/profile_path) as primary, distinct from the dozens of alternates also ingested, including correctly marking an existing row primary (not inserting a duplicate) when that image already appears in the alternates list. - app/services/artwork/: an ArtworkStorage interface with a LocalDiskStorage implementation (checksum-named files, single-host deployment via a Docker volume shared between api and worker -- see docs/architecture.md for the S3/R2 upgrade path), and a download pipeline with content-type validation, a size cap, and path-traversal protection. - After movie/series/person ingestion succeeds, tasks.py cascades one download_artwork_task per primary artwork id via arq's ctx['redis'] (same fan-out pattern as the season cascade) -- fire-and-forget, not tracked in ingestion_jobs, since a failed image cache is a known-safe fallback to the provider URL, not a user-facing failure. - app/main.py serves MEDIA_ROOT at MEDIA_URL_PREFIX; local_path is preferred over original_url once set (already-existing fallback logic in app/services/common.py:artwork_url, now actually exercised). Also fixes a real interaction bug this surfaced: every ingestion pipeline deleted and recreated all of an entity's Artwork rows on each re-sync, which would have silently wiped local_path/checksum on every re-ingestion, forcing a full re-download every time and defeating the point of caching locally at all. app/repositories/polymorphic.py's new sync_artwork upserts by provider_asset_id (TMDb's stable per-image path) instead, preserving downloaded copies across re-sync, and replaces three near-identical delete+recreate helpers that used to live separately in movie_ingestion.py, series_ingestion.py, and person_ingestion.py. Co-Authored-By: Claude Sonnet 5 <[email protected]>	2026-09-05 03:26:28 -05:00
NixReaperandClaude Sonnet 5	dd3ff9764b	Fix race condition in get_or_create_* under concurrent ingestion ... Every get_or_create_* helper (genre, keyword, language, country, company, collection, person) followed check-then-act: look up by natural key, insert if missing. That was safe as long as nothing ingested concurrently for overlapping data -- which stopped being true the moment ingest_series_task could cascade season ingestion (include_seasons), since arq runs multiple queued jobs concurrently. Two seasons of the same series sharing a guest star, crew member, or company can now race to create it: both see "not found", both try to INSERT, and the loser hits a unique-constraint violation instead of finding the winner's row. Reproduced live: cascading Better Call Saul (60059) with include_seasons crashed a season job with "duplicate key value violates unique constraint uq_external_id_source" on a recurring guest actor shared across two seasons ingesting at the same time. _create_or_recover wraps each insert in a SAVEPOINT and, on that specific failure, rolls back just the savepoint and re-queries for what the concurrent job committed -- recovering instead of aborting the whole ingestion job. Added unit tests for the retry mechanism itself; genuinely reproducing the underlying race in an automated test would need real Postgres MVCC timing, which isn't something a deterministic test can force reliably. Co-Authored-By: Claude Sonnet 5 <[email protected]>	2026-09-05 02:27:07 -05:00
NixReaperandClaude Sonnet 5	3f6332559a	Add series/season/episode ingestion pipeline ... Movie and person ingestion existed, but the series/season/episode API endpoints had no way to actually populate them from TMDb -- the biggest remaining gap in Core 0.1's television scope. - normalize_series/normalize_season in the metadata normalizer, including two TMDb quirks that would otherwise silently produce empty data: /tv nests keywords and alternative_titles under "results" (not "keywords"/"titles" like /movie), and series-level creators come from a separate created_by array, not the credits cast/crew lists. - ingest_series upserts the Series row plus season *stubs* (number, title, air date, poster -- everything TMDb inlines on /tv/{id}); ingest_season fetches and upserts one season's full episode list, and requires the series to already exist (TMDb has no single endpoint for a whole series' episodes, so this has to be a separate per-season call). - Wired through the same job-lifecycle path as movie/person ingestion (runner.py, arq tasks, admin endpoints), with the shared boilerplate between them factored into runner._run_ingestion and admin.py:_enqueue rather than copy-pasted a third and fourth time. - Unit tests for both normalizers (runnable without a DB) and integration tests for the full ingest_series -> ingest_season flow, including the series-must-exist-first error and idempotency on re-ingestion. Co-Authored-By: Claude Sonnet 5 <[email protected]>	2026-09-04 19:43:56 -05:00
NixReaperandClaude Sonnet 5	c20df1c503	Initial Aurora Core 0.1 implementation ... Production-grade media metadata backend per docs/aurora-core-spec.md: Aurora-owned UUIDv7 identity system, normalized movie/TV/people/credits schema, provider abstraction with a TMDb adapter, an idempotent async ingestion pipeline (normalize -> match -> upsert -> provenance), a Postgres FTS/trigram search layer, a stateless FastAPI v1 API with Redis caching/rate-limiting, arq workers, dev/prod Docker Compose topologies, unit + integration + load tests, CI, and architecture/ database/API/ERD documentation. Co-Authored-By: Claude Sonnet 5 <[email protected]>	2026-09-04 17:34:03 -05:00