Aurora-Core/docs/database.md

Aurora Core — Database

PostgreSQL is authoritative (principle 4). Nothing important lives only in Redis. See erd.md for the full entity-relationship diagram.

Aurora IDs vs. primary keys

Every table's primary key is a UUIDv7 (app/models/base.py:UUIDPrimaryKeyMixin), generated in the application, not the database. UUIDv7 is time-ordered, so it stays btree-friendly (unlike UUIDv4) and can double as a rough creation-order cursor.

The public "Aurora ID" strings clients see (aur_movie_01m1q7p1c...) are a lossless, sortable Crockford-base32 encoding of that same UUID, produced by app/core/ids.py:aurora_id(). Encoding (not a second stored column) keeps the public ID and the primary key from ever drifting apart. See section 3 of docs/aurora-core-spec.md for why Aurora never uses a provider's ID (TMDb, IMDb, ...) as a primary key.

Reference data

• languages (ISO 639-1), countries (ISO 3166-1 alpha-2), certifications (country + rating system + rating, e.g. US/MPA/PG-13) — populated incrementally as ingestion encounters new codes (get_or_create_* in app/repositories/reference_repository.py), not preloaded. A curated seed list is a reasonable follow-up (see Known limitations).
• genres, keywords — deduplicated by a slugified name. Two providers' "Sci-Fi" and "Science Fiction" are not currently merged; that requires a curated synonym table, deliberately deferred (section 33 lists dedup foundations, not a finished thesaurus).

Movies / TV

movies and series hold the record itself; per-locale and per-relationship data lives in satellite tables (movie_alternate_titles, movie_spoken_languages, movie_production_countries, release_dates, movie_recommendations, and the series_* equivalents) rather than JSON blobs, per section 26. seasons and episodes form a strict series -> season -> episode hierarchy; episodes.series_id is denormalized (in addition to season_id) purely so episode queries don't need to hop through seasons to filter/join by series.

movies.status / series.status are native Postgres enums, values .value of app/models/enums.py:MovieStatus/SeriesStatus — see "Enum storage" below.

Credits

credits is the flexible cast/crew system from section 7: one row per (person, media entity, department/job/character), not per-movie text blobs. department == "Acting" rows carry character/billing_order; everything else carries job. Series-level "Creator" credits and episode is_guest (guest stars) reuse the same table rather than adding new ones.

Companies / Collections

A single companies table serves production companies, studios, networks, and distributors, distinguished by company_type (section 8) — a TV network and a movie studio have the same shape (name, logo, country, parent company), so a separate Network entity would just duplicate columns. collections (franchises) are first-class, with their own artwork and collection_relations for shared-universe links between collections (section 9).

Artwork / Videos

Never a poster_url column (section 10/11). artwork and videos are their own tables carrying provider, provider-native asset ID, dimensions, and (for artwork) attribution/license fields — Aurora doesn't assume externally-sourced images can be freely redistributed. Aurora never downloads video media; videos.url is a reference to the provider's hosted copy (YouTube, Vimeo, ...).

Artwork storage (self-hosting)

artwork.original_url (the provider's CDN link) is always populated by ingestion; artwork.local_path starts NULL and is filled in asynchronously by app/services/artwork/. app/services/common.py:artwork_url() prefers local_path and falls back to original_url, so API responses point at TMDb's CDN until a copy has been downloaded, then switch over automatically once it has.

Only artwork marked is_primary gets downloaded automatically — the single image the provider designates as the image for an entity (TMDb's poster_path/backdrop_path/profile_path/logo_path), not the dozens of alternates also stored. This includes company/network logos (a company has exactly one ingested, so it's always primary) — cascaded per-movie/series since companies aren't independently ingested the way movies/series/people are. app/services/ingestion/runner.py:get_primary_artwork_for_download finds them and app/workers/tasks.py enqueues a download_artwork_task per id once movie/series/person/season ingestion succeeds — see docs/architecture.md for the full flow.

Storage is local disk (app/services/artwork/storage.py:LocalDiskStorage) behind an ArtworkStorage interface, deliberately simple for a single-Docker-host deployment where the api and worker containers share one volume (docker-compose.yml's aurora_media_data). A deployment that outgrows one host would need an S3/R2-backed implementation of the same interface instead. Files are named by content (<type>/<sha256[:2]>/<sha256>.<ext>), so the same image referenced by two entities is only ever stored once.

Polymorphic associations

credits, artwork, videos, external_ids, and release_dates attach to more than one entity type via an (entity_type, entity_id) pair instead of a typed foreign key, because a single column can't reference "movies.id or series.id or people.id depending on a sibling column" in Postgres. The tradeoff: the schema does not enforce that entity_id actually points at a real row of entity_type. That integrity is enforced by the repository/service layer (app/repositories/polymorphic.py, app/services/ingestion/*), which is the only code path allowed to write these tables. If Core grows enough that this becomes a real risk, the standard fix is per-type junction tables (movie_credits, series_credits, ...) at the cost of losing the single shared query path — deliberately not done for 0.1.

External IDs & provenance

external_ids is the generalized external-identifier system (section 12): (provider, entity_type, external_id) is unique (one Aurora entity per provider-native ID), and (entity_type, entity_id, provider) is unique (one mapping per provider per entity).

provider_records is the full provenance ledger (section 14): one row per (provider, entity_type, external_id) holding the raw payload (raw_data JSONB), match_confidence, and which ingestion_job produced it. Entity-level provenance is coarse — movies.source_provider_id / series.source_provider_id point at "the provider that was authoritative at creation time" for quick display; full field-level provenance (which provider supplied this specific field) is not tracked in 0.1. See Known limitations.

ingestion_jobs tracks async work with a unique idempotency_key (e.g. tmdb:movie:603) so re-triggering the same sync doesn't double-enqueue — see app/services/ingestion/runner.py.

Enum storage

Python enums in app/models/enums.py are enum.StrEnum (member value == member name's meaning, e.g. MovieStatus.RELEASED == "released"). SQLAlchemy's Enum type stores .name ("RELEASED") by default, which would silently diverge from the lowercase strings the API and ingestion code use everywhere. app/models/base.py:_enum_column() overrides this via values_callable so the database, the ORM, and the API all agree on the same lowercase strings.

Migrations

Alembic (section 27); the application never creates tables at startup — see app/core/database.py. The initial migration (migrations/versions/0001_initial_schema.py) was hand-written rather than autogenerated (no throwaway Postgres was available in the environment it was originally written in), then verified by actually running alembic upgrade head against a genuinely empty PostgreSQL on a real devbox — that run caught a real bug (pre-creating enum types and then also letting CREATE TABLE auto-create them raised DuplicateObject; fixed by create_type=False on each enum), now fixed and verified clean. Autogenerate future migrations against a database that already has this one applied (alembic revision --autogenerate), not from scratch.

alembic upgrade head          # apply all migrations
alembic revision --autogenerate -m "add xyz"   # generate the next one
alembic downgrade -1          # roll back one revision

Integration/API tests build schema directly from Base.metadata.create_all (see tests/conftest.py) for speed and to catch model drift quickly — this is a testing shortcut, not how the running application ever gets its schema.

Search indexes

pg_trgm (trigram similarity, for typo-tolerant matching) plus functional GIN indexes over to_tsvector('simple', title || ' ' || original_title) on movies/series, name on people/collections/companies, and trigram indexes on the alternate-title tables. See app/services/search/postgresql_provider.py and section 16 of the spec.

Known limitations (tracked, not hidden)

• No curated language/country seed data — rows are created on first encounter during ingestion, so english_name/name are sometimes just the code uppercased until backfilled.
• No fuzzy/candidate entity matching — only exact external-ID matches are auto-linked (section 14: never silently merge ambiguous matches). A review queue for AMBIGUOUS matches is the natural next step once Core has an admin UI to act on it.
• Field-level provenance (which provider supplied this value) is not tracked, only entity-level (source_provider_id) and per-provider raw payloads (provider_records.raw_data).
• Satellite associations (credits, videos, genre/keyword/company links) are replaced wholesale on each re-sync rather than diffed, so their Aurora IDs are not stable across re-ingestion — see app/services/ingestion/movie_ingestion.py. Artwork is the exception: app/repositories/polymorphic.py:sync_artwork upserts by provider_asset_id instead, specifically so a previously-downloaded local_path/checksum (see "Artwork storage" below) survives re-ingestion instead of being silently reset to None and re-downloaded every sync.
• Seasons don't get their own external_ids row (identified instead by the natural (series_id, season_number) key) — episodes do, since TMDb gives each episode a distinct global id. Revisit if a provider ever needs a season-level external reference Aurora doesn't already have another way to resolve.
• ingest_series only upserts season stubs (number, title, air date, poster) — episodes require a separate ingest_season call per season (TMDb has no single endpoint returning a whole series' episodes), and that call requires the series to already exist. POST /api/v1/admin/ingest/series can cascade this automatically (include_seasons: true fans out one ingest_season job per season from ingest_series_task once the series succeeds, via arq's own ctx['redis'] — see app/workers/tasks.py), but it's opt-in: some shows have dozens of seasons, each a separate TMDb call, so cascading isn't the default.