Aurora-Core/docs/database.md

# Aurora Core — Database

PostgreSQL is authoritative (principle 4). Nothing important lives only in
Redis. See [erd.md](erd.md) for the full entity-relationship diagram.

## Aurora IDs vs. primary keys

Every table's primary key is a UUIDv7 (`app/models/base.py:UUIDPrimaryKeyMixin`),
generated in the application, not the database. UUIDv7 is time-ordered, so it
stays btree-friendly (unlike UUIDv4) and can double as a rough creation-order
cursor.

The public "Aurora ID" strings clients see (`aur_movie_01m1q7p1c...`) are a
lossless, sortable Crockford-base32 encoding of that same UUID, produced by
`app/core/ids.py:aurora_id()`. Encoding (not a second stored column) keeps
the public ID and the primary key from ever drifting apart. See section 3 of
`docs/aurora-core-spec.md` for why Aurora never uses a provider's ID (TMDb,
IMDb, ...) as a primary key.

## Reference data

- `languages` (ISO 639-1), `countries` (ISO 3166-1 alpha-2), `certifications`
  (country + rating system + rating, e.g. US/MPA/PG-13) — populated
  incrementally as ingestion encounters new codes (`get_or_create_*` in
  `app/repositories/reference_repository.py`), not preloaded. A curated
  seed list is a reasonable follow-up (see Known limitations).
- `genres`, `keywords` — deduplicated by a slugified name. Two providers'
  "Sci-Fi" and "Science Fiction" are **not** currently merged; that requires
  a curated synonym table, deliberately deferred (section 33 lists dedup
  *foundations*, not a finished thesaurus).

## Movies / TV

`movies` and `series` hold the record itself; per-locale and per-relationship
data lives in satellite tables (`movie_alternate_titles`,
`movie_spoken_languages`, `movie_production_countries`, `release_dates`,
`movie_recommendations`, and the `series_*` equivalents) rather than JSON
blobs, per section 26. `seasons` and `episodes` form a strict
`series -> season -> episode` hierarchy; `episodes.series_id` is denormalized
(in addition to `season_id`) purely so episode queries don't need to hop
through `seasons` to filter/join by series.

`movies.status` / `series.status` are native Postgres enums, values `.value`
of `app/models/enums.py:MovieStatus`/`SeriesStatus` — see "Enum storage"
below.

## Credits

`credits` is the flexible cast/crew system from section 7: one row per
(person, media entity, department/job/character), not per-movie text blobs.
`department == "Acting"` rows carry `character`/`billing_order`; everything
else carries `job`. Series-level "Creator" credits and episode
`is_guest` (guest stars) reuse the same table rather than adding new ones.

## Companies / Collections

A single `companies` table serves production companies, studios, networks,
and distributors, distinguished by `company_type` (section 8) — a TV network
and a movie studio have the same shape (name, logo, country, parent
company), so a separate `Network` entity would just duplicate columns. A
company's own filmography (`GET /api/v1/companies/{id}`'s `movies`/`series`
lists) is read straight off `movie_companies`/`series_companies` — no
separate sync step, since those rows only ever exist as a side effect of
ingesting the movie or series itself (`app/repositories/company_repository.py`).

`collections` (franchises) are first-class, with their own artwork and
`collection_relations` for shared-universe links between collections
(section 9). A collection starts as a bare name-only stub the first time
movie ingestion sees a `belongs_to_collection` reference; its real overview,
poster/backdrop, and ordered `collection_movies` membership are filled in by
`app/services/ingestion/collection_ingestion.py:ingest_collection`
(`POST /api/v1/admin/ingest/collection`), which — like recommendations —
only links movies Aurora has already ingested rather than fabricating stubs
for the rest of the franchise. `collection_relations` and `collection_series`
exist in the schema for future use but nothing populates them yet.

## Artwork / Videos

Never a `poster_url` column (section 10/11). `artwork` and `videos` are
their own tables carrying provider, provider-native asset ID, dimensions,
and (for artwork) `attribution`/`license` fields — Aurora doesn't assume
externally-sourced images can be freely redistributed. Aurora never
downloads video media; `videos.url` is a reference to the provider's
hosted copy (YouTube, Vimeo, ...).

### Artwork storage (self-hosting)

`artwork.original_url` (the provider's CDN link) is always populated by
ingestion; `artwork.local_path` starts `NULL` and is filled in
asynchronously by `app/services/artwork/`. `app/services/common.py:artwork_url()`
prefers `local_path` and falls back to `original_url`, so API responses
point at TMDb's CDN until a copy has been downloaded, then switch over
automatically once it has.

Only artwork marked `is_primary` gets downloaded automatically — the single
image the provider designates as *the* image for an entity (TMDb's
`poster_path`/`backdrop_path`/`profile_path`/`logo_path`), not the dozens of
alternates also stored. This includes company/network logos (a company has
exactly one ingested, so it's always primary) — cascaded per-movie/series
since companies aren't independently ingested the way movies/series/people
are. `app/services/ingestion/runner.py:get_primary_artwork_for_download`
finds them and `app/workers/tasks.py` enqueues a `download_artwork_task` per
id once movie/series/person/season ingestion succeeds — see
`docs/architecture.md` for the full flow.

Storage is local disk (`app/services/artwork/storage.py:LocalDiskStorage`)
behind an `ArtworkStorage` interface, deliberately simple for a
single-Docker-host deployment where the `api` and `worker` containers share
one volume (`docker-compose.yml`'s `aurora_media_data`). A deployment that
outgrows one host would need an S3/R2-backed implementation of the same
interface instead. Files are named by content (`<type>/<sha256[:2]>/<sha256>.<ext>`),
so the same image referenced by two entities is only ever stored once.

## Polymorphic associations

`credits`, `artwork`, `videos`, `external_ids`, and `release_dates` attach to
more than one entity type via an `(entity_type, entity_id)` pair instead of a
typed foreign key, because a single column can't reference "movies.id or
series.id or people.id depending on a sibling column" in Postgres. The
tradeoff: **the schema does not enforce that `entity_id` actually points at
a real row of `entity_type`.** That integrity is enforced by the
repository/service layer (`app/repositories/polymorphic.py`,
`app/services/ingestion/*`), which is the only code path allowed to write
these tables. If Core grows enough that this becomes a real risk, the
standard fix is per-type junction tables (`movie_credits`, `series_credits`,
...) at the cost of losing the single shared query path — deliberately not
done for 0.1.

## External IDs & provenance

`external_ids` is the generalized external-identifier system (section 12):
`(provider, entity_type, external_id)` is unique (one Aurora entity per
provider-native ID), and `(entity_type, entity_id, provider)` is unique (one
mapping per provider per entity).

`provider_records` is the full provenance ledger (section 14): one row per
`(provider, entity_type, external_id)` holding the raw payload
(`raw_data` JSONB), `match_confidence`, and which `ingestion_job` produced
it. Entity-level provenance is coarse — `movies.source_provider_id` /
`series.source_provider_id` point at "the provider that was authoritative at
creation time" for quick display; full field-level provenance (which
provider supplied *this specific* field) is not tracked in 0.1. See Known
limitations.

`ingestion_jobs` tracks async work with a unique `idempotency_key` (e.g.
`tmdb:movie:603`) so re-triggering the same sync doesn't double-enqueue —
see `app/services/ingestion/runner.py`.

## Enum storage

Python enums in `app/models/enums.py` are `enum.StrEnum` (member value ==
member name's meaning, e.g. `MovieStatus.RELEASED == "released"`).
SQLAlchemy's `Enum` type stores `.name` (`"RELEASED"`) by default, which
would silently diverge from the lowercase strings the API and ingestion code
use everywhere. `app/models/base.py:_enum_column()` overrides this via
`values_callable` so the database, the ORM, and the API all agree on the
same lowercase strings.

## Migrations

Alembic (section 27); the application **never** creates tables at startup —
see `app/core/database.py`. The initial migration
(`migrations/versions/0001_initial_schema.py`) was hand-written rather than
autogenerated (no throwaway Postgres was available in the environment it
was originally written in), then verified by actually running
`alembic upgrade head` against a genuinely empty PostgreSQL on a real
devbox — that run caught a real bug (pre-creating enum types and then also
letting `CREATE TABLE` auto-create them raised `DuplicateObject`; fixed by
`create_type=False` on each enum), now fixed and verified clean. Autogenerate
future migrations against a database that already has this one applied
(`alembic revision --autogenerate`), not from scratch.

```bash
alembic upgrade head          # apply all migrations
alembic revision --autogenerate -m "add xyz"   # generate the next one
alembic downgrade -1          # roll back one revision
```

Integration/API tests build schema directly from `Base.metadata.create_all`
(see `tests/conftest.py`) for speed and to catch model drift quickly — this
is a testing shortcut, not how the running application ever gets its schema.

## Search indexes

`pg_trgm` (trigram similarity, for typo-tolerant matching) plus functional
GIN indexes over `to_tsvector('simple', title || ' ' || original_title)` on
`movies`/`series`, `name` on `people`/`collections`/`companies`, and trigram
indexes on the alternate-title tables. See
`app/services/search/postgresql_provider.py` and section 16 of the spec.

## Known limitations (tracked, not hidden)

- No curated language/country seed data — rows are created on first
  encounter during ingestion, so `english_name`/`name` are sometimes just
  the code uppercased until backfilled.
- No fuzzy/candidate entity matching — only exact external-ID matches are
  auto-linked (section 14: never silently merge ambiguous matches). A
  review queue for AMBIGUOUS matches is the natural next step once Core has
  an admin UI to act on it.
- Field-level provenance (which provider supplied *this* value) is not
  tracked, only entity-level (`source_provider_id`) and per-provider raw
  payloads (`provider_records.raw_data`).
- Satellite associations (credits, videos, genre/keyword/company links) are
  replaced wholesale on each re-sync rather than diffed, so their Aurora IDs
  are not stable across re-ingestion — see
  `app/services/ingestion/movie_ingestion.py`. **Artwork is the exception:**
  `app/repositories/polymorphic.py:sync_artwork` upserts by
  `provider_asset_id` instead, specifically so a previously-downloaded
  `local_path`/`checksum` (see "Artwork storage" below) survives
  re-ingestion instead of being silently reset to `None` and re-downloaded
  every sync.
- Seasons don't get their own `external_ids` row (identified instead by the
  natural `(series_id, season_number)` key) — episodes do, since TMDb gives
  each episode a distinct global id. Revisit if a provider ever needs a
  season-level external reference Aurora doesn't already have another way
  to resolve.
- `ingest_series` only upserts season *stubs* (number, title, air date,
  poster) — episodes require a separate `ingest_season` call per season
  (TMDb has no single endpoint returning a whole series' episodes), and
  that call requires the series to already exist. `POST
  /api/v1/admin/ingest/series` can cascade this automatically
  (`include_seasons: true` fans out one `ingest_season` job per season from
  `ingest_series_task` once the series succeeds, via arq's own `ctx['redis']`
  — see `app/workers/tasks.py`), but it's opt-in: some shows have dozens of
  seasons, each a separate TMDb call, so cascading isn't the default.