app/services/bulk_import.py fetches and parses TMDb's daily ID export
files (published once a day specifically for full-catalog sync, unlike
/discover which caps at 10,000 results per query) -- today's file with
a fallback to yesterday's, since TMDb publishes at a fixed time and
"today's" may not exist yet.
app/scripts/bulk_import.py is a standalone ops script (not an arq task,
since enqueueing millions of jobs would blow past arq's job_timeout)
that fans every id from the export into the existing per-entity
ingestion pipeline -- get_or_create_ingestion_job + the same arq task
each admin endpoint already uses, so a single id's behavior is
identical whether it came from the admin API or this script. Bounded
concurrency (50) on the enqueue phase; idempotency-keyed jobs make
interrupting and re-running the script safe.
Real TMDb catalog size (checked live): ~1.24M movies, 231K series,
4.9M people, 9.8K collections -- ~6.4M entities. At default worker
concurrency this is realistically weeks of continuous processing, not
hours, documented as such in README.md.
Also added 429 (rate-limited) handling to TMDbProvider before running
this at volume: previously a 429 fell into the generic >=400 branch and
failed the request outright with zero retry, since only httpx.TransportError
triggered the existing retry decorator. Now a 429 raises
ProviderRateLimitedError carrying the response's own Retry-After hint,
and tenacity's wait strategy honors that hint directly instead of
guessing via exponential backoff.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Two real bugs hit during tonight's first production deploy (found and
fixed live on the server, folded back into the tracked config so the
next deploy doesn't hit either wall):
1. pgbouncer defaults to md5 auth; postgres:17-alpine defaults password_encryption
to scram-sha-256. Every connection failed with "wrong password type"
until AUTH_TYPE=scram-sha-256 was set explicitly on the pgbouncer service.
2. POSTGRES_USER/PASSWORD/DB were never documented anywhere (not in
.env.example, despite docker-compose.prod.yml requiring them) --
added, with a note against `openssl rand -base64` for the password
specifically: its output can contain /+=, which breaks pgbouncer's
(and any DATABASE_URL parser's) naive URL parsing. Pointed at
token_urlsafe instead, matching the other generated secrets.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
The production host runs cPanel, which already owns ports 80/443 for
other sites on the box -- there's no Traefik there. docker-compose.prod.yml
now binds the api container to 127.0.0.1:8000 only; the host's own
Apache/LiteSpeed (via cPanel) reverse-proxies api.amdbs.net to it and
terminates TLS itself (AutoSSL). Dropped the external traefik-public
network requirement and Traefik labels entirely.
Also corrected a separate latent inaccuracy while touching this: a bare
`docker compose up` (not `docker stack deploy`) never actually honored
deploy.replicas -- docs now say so and point at `--scale` instead of
implying free multi-replica scaling that was never really happening.
README's Production deployment section rewritten with concrete steps:
Cloudflare DNS record, cPanel subdomain + reverse proxy, then compose up.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
api.amdbs.net (backend) / www.amdbs.net (frontend) -- records the
required AURORA_API_HOST and CORS_ORIGINS values for the production
.env, and the external-Traefik prerequisite docker-compose.prod.yml
already assumed but never spelled out.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
FilmographyCreditOut already aggregated a guest star's episode
appearances into one row per series with an episode_count, but never
exposed which episodes those were even though the individual episode
Credit rows exist in the database. Adds an `episodes` list (season/
episode number, title, air date, sorted chronologically) alongside the
existing count -- closes a known limitation tracked in docs/api.md.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Section 10: every image API responses reference has pointed straight
at TMDb's CDN since day one -- Aurora never actually downloaded or
served artwork itself, despite storing dimensions/checksum/local_path
columns for exactly that. Closes the gap:
- is_primary was declared on the Artwork model but never actually set
anywhere -- every row defaulted to False. Now the normalizer marks
the one image each provider designates as *the* image for an entity
(TMDb's poster_path/backdrop_path/profile_path) as primary,
distinct from the dozens of alternates also ingested, including
correctly marking an existing row primary (not inserting a
duplicate) when that image already appears in the alternates list.
- app/services/artwork/: an ArtworkStorage interface with a
LocalDiskStorage implementation (checksum-named files, single-host
deployment via a Docker volume shared between api and worker -- see
docs/architecture.md for the S3/R2 upgrade path), and a download
pipeline with content-type validation, a size cap, and path-traversal
protection.
- After movie/series/person ingestion succeeds, tasks.py cascades one
download_artwork_task per primary artwork id via arq's ctx['redis']
(same fan-out pattern as the season cascade) -- fire-and-forget, not
tracked in ingestion_jobs, since a failed image cache is a
known-safe fallback to the provider URL, not a user-facing failure.
- app/main.py serves MEDIA_ROOT at MEDIA_URL_PREFIX; local_path is
preferred over original_url once set (already-existing fallback
logic in app/services/common.py:artwork_url, now actually exercised).
Also fixes a real interaction bug this surfaced: every ingestion
pipeline deleted and recreated all of an entity's Artwork rows on each
re-sync, which would have silently wiped local_path/checksum on every
re-ingestion, forcing a full re-download every time and defeating the
point of caching locally at all. app/repositories/polymorphic.py's new
sync_artwork upserts by provider_asset_id (TMDb's stable per-image
path) instead, preserving downloaded copies across re-sync, and
replaces three near-identical delete+recreate helpers that used to
live separately in movie_ingestion.py, series_ingestion.py, and
person_ingestion.py.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
