Initial Aurora Core 0.1 implementation
CI / lint (push) Canceled after 0s
CI / typecheck (push) Canceled after 0s
CI / unit-tests (push) Canceled after 0s
CI / integration-tests (push) Canceled after 0s
CI / container-build (push) Canceled after 0s
CI / dependency-audit (push) Canceled after 0s
CI / lint (push) Canceled after 0s
CI / typecheck (push) Canceled after 0s
CI / unit-tests (push) Canceled after 0s
CI / integration-tests (push) Canceled after 0s
CI / container-build (push) Canceled after 0s
CI / dependency-audit (push) Canceled after 0s
Production-grade media metadata backend per docs/aurora-core-spec.md: Aurora-owned UUIDv7 identity system, normalized movie/TV/people/credits schema, provider abstraction with a TMDb adapter, an idempotent async ingestion pipeline (normalize -> match -> upsert -> provenance), a Postgres FTS/trigram search layer, a stateless FastAPI v1 API with Redis caching/rate-limiting, arq workers, dev/prod Docker Compose topologies, unit + integration + load tests, CI, and architecture/ database/API/ERD documentation. Co-Authored-By: Claude Sonnet 5 <[email protected]>
This commit is contained in:
+143
@@ -0,0 +1,143 @@
|
||||
# Aurora Core — API
|
||||
|
||||
Base path: `/api/v1`. Interactive OpenAPI docs are available at `/docs` in
|
||||
any environment where `AURORA_API_DOCS_ENABLED=true` (default; set to
|
||||
`false` to hide docs in production — section 32).
|
||||
|
||||
All responses are JSON, built from Pydantic response schemas
|
||||
(`app/schemas/`), never from serialized ORM objects (section 18). Errors use
|
||||
a consistent envelope:
|
||||
|
||||
```json
|
||||
{ "error": { "code": "movie_not_found", "message": "No such movie: aur_movie_..." } }
|
||||
```
|
||||
|
||||
## Health
|
||||
|
||||
| Method | Path | Notes |
|
||||
|---|---|---|
|
||||
| GET | `/health` and `/api/v1/health` | Liveness only. |
|
||||
| GET | `/ready` and `/api/v1/ready` | Checks PostgreSQL (always) and Redis (if `REDIS_REQUIRED=true`). |
|
||||
|
||||
## Search
|
||||
|
||||
| Method | Path | Notes |
|
||||
|---|---|---|
|
||||
| GET | `/api/v1/search?q=...&limit=10` | Searches movies, series, people, collections, companies. `limit` (1-50) caps results **per type**. |
|
||||
|
||||
```json
|
||||
GET /api/v1/search?q=the%20matrix
|
||||
|
||||
{
|
||||
"query": "the matrix",
|
||||
"movies": [ { "id": "aur_movie_...", "type": "movie", "title": "The Matrix", "release_date": "1999-03-30", "poster_path": "https://..." } ],
|
||||
"series": [],
|
||||
"people": [],
|
||||
"collections": [ { "id": "aur_collection_...", "name": "The Matrix Collection" } ],
|
||||
"companies": [],
|
||||
"total_results": 2
|
||||
}
|
||||
```
|
||||
|
||||
## Movies
|
||||
|
||||
| Method | Path |
|
||||
|---|---|
|
||||
| GET | `/api/v1/movies/{aurora_id}` |
|
||||
| GET | `/api/v1/movies/{aurora_id}/credits` |
|
||||
| GET | `/api/v1/movies/{aurora_id}/images` |
|
||||
| GET | `/api/v1/movies/{aurora_id}/videos` |
|
||||
|
||||
`GET /api/v1/movies/{aurora_id}` (cached, see `docs/architecture.md#caching`):
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "aur_movie_01m1q7p1c...",
|
||||
"type": "movie",
|
||||
"title": "The Matrix",
|
||||
"original_title": "The Matrix",
|
||||
"overview": "...",
|
||||
"tagline": "Welcome to the Real World",
|
||||
"status": "released",
|
||||
"original_language": "en",
|
||||
"release_date": "1999-03-30",
|
||||
"runtime": 136,
|
||||
"budget": 63000000,
|
||||
"revenue": 463517383,
|
||||
"popularity": 45.5,
|
||||
"vote_average": 8.2,
|
||||
"vote_count": 24000,
|
||||
"adult": false,
|
||||
"genres": [{ "id": "aur_genre_...", "name": "Action" }],
|
||||
"keywords": [{ "id": "aur_keyword_...", "name": "dystopia" }],
|
||||
"cast": [{ "id": "aur_credit_...", "person": { "id": "aur_person_...", "name": "Keanu Reeves", "profile_path": null }, "character": "Neo", "billing_order": 0 }],
|
||||
"crew": [{ "id": "aur_credit_...", "person": { "id": "aur_person_...", "name": "Lana Wachowski" }, "department": "Directing", "job": "Director" }],
|
||||
"companies": [{ "id": "aur_company_...", "name": "Village Roadshow Pictures", "company_type": "production_company" }],
|
||||
"collection": { "id": "aur_collection_...", "name": "The Matrix Collection" },
|
||||
"images": { "posters": [...], "backdrops": [...], "logos": [...] },
|
||||
"videos": [{ "id": "aur_vid_...", "type": "trailer", "site": "YouTube", "url": "https://www.youtube.com/watch?v=...", "is_official": true }],
|
||||
"external_ids": { "tmdb": "603", "imdb": "tt0133093" },
|
||||
"created_at": "...",
|
||||
"updated_at": "..."
|
||||
}
|
||||
```
|
||||
|
||||
## Series / Seasons / Episodes
|
||||
|
||||
| Method | Path |
|
||||
|---|---|
|
||||
| GET | `/api/v1/series/{aurora_id}` |
|
||||
| GET | `/api/v1/series/{aurora_id}/seasons` |
|
||||
| GET | `/api/v1/seasons/{aurora_id}` |
|
||||
| GET | `/api/v1/episodes/{aurora_id}` |
|
||||
|
||||
Series detail includes `networks` and `companies` as separate lists (a TV
|
||||
network and a production company are both `Company` rows distinguished by
|
||||
`company_type` — see `docs/database.md`), plus `creators` (series-level
|
||||
"Creator" credits) alongside `cast`/`crew`.
|
||||
|
||||
## People
|
||||
|
||||
| Method | Path |
|
||||
|---|---|
|
||||
| GET | `/api/v1/people/{aurora_id}` |
|
||||
| GET | `/api/v1/people/{aurora_id}/credits` (movie filmography in 0.1; series/episode credits are stored but not yet surfaced here — see Known limitations) |
|
||||
|
||||
## Collections / Companies
|
||||
|
||||
| Method | Path |
|
||||
|---|---|
|
||||
| GET | `/api/v1/collections/{aurora_id}` |
|
||||
| GET | `/api/v1/companies/{aurora_id}` |
|
||||
|
||||
## Administrative (section 17: "separately protected")
|
||||
|
||||
Require header `X-API-Key: <one of AURORA_ADMIN_API_KEYS>`. Never exposed to
|
||||
public clients — mount behind a separate Traefik router/ACL in production if
|
||||
you want network-level isolation too.
|
||||
|
||||
| Method | Path | Body | Notes |
|
||||
|---|---|---|---|
|
||||
| POST | `/api/v1/admin/ingest/movie` | `{"tmdb_id": "603"}` | Enqueues an arq job; returns `{"job_id": "...", "status": "enqueued"}` immediately (section 15: ingestion never runs inline in a request). |
|
||||
| POST | `/api/v1/admin/ingest/person` | `{"tmdb_id": "6384"}` | Same, for a person profile. |
|
||||
| GET | `/api/v1/admin/ingest/jobs/{job_id}` | — | Poll job status: `pending` → `running` → `succeeded`/`failed`/`retrying`. |
|
||||
|
||||
## Errors
|
||||
|
||||
| Status | When |
|
||||
|---|---|
|
||||
| 404 | Entity not found, or the ID is well-formed but the wrong entity type (`aur_person_...` passed to `/movies/{id}`) |
|
||||
| 422 | Request validation failure (e.g. missing/invalid query params) |
|
||||
| 429 | Rate limit exceeded (`Retry-After` header set) |
|
||||
| 401 | Missing/invalid `X-API-Key` on an admin endpoint |
|
||||
| 503 | Admin API called with `AURORA_ADMIN_API_KEYS` unset, or the job queue is unavailable |
|
||||
|
||||
## Known limitations
|
||||
|
||||
- No pagination on list-shaped responses yet beyond search's `limit`
|
||||
(movie/series lists like recommendations return everything). Cursor-based
|
||||
pagination using the Aurora ID's inherent creation-order sort (see
|
||||
`docs/database.md#aurora-ids-vs-primary-keys`) is the natural next step.
|
||||
- `/people/{id}/credits` only returns movie credits today; series/episode
|
||||
credits exist in the `credits` table but the response assembly for them
|
||||
is not yet wired up (`app/services/person_service.py`).
|
||||
@@ -0,0 +1,154 @@
|
||||
# Aurora Core — Architecture
|
||||
|
||||
Aurora Core is a production-grade **modular monolith** (section 1/9 of
|
||||
`docs/aurora-core-spec.md`) — one deployable API codebase with clear internal
|
||||
module boundaries, not a set of microservices. It is the canonical metadata
|
||||
layer for the Aurora ecosystem; Plex, Overseerr, Radarr, Sonarr, and future
|
||||
Aurora applications are consumers of it, never dependencies of it.
|
||||
|
||||
```
|
||||
Internet
|
||||
|
|
||||
Traefik
|
||||
|
|
||||
Aurora Core API replicas (stateless)
|
||||
|
|
||||
+--- PostgreSQL (authoritative)
|
||||
|
|
||||
+--- PgBouncer (production connection pooling)
|
||||
|
|
||||
+--- Redis (cache / rate limit / job queue — disposable)
|
||||
|
|
||||
+--- Background Workers (arq, scale independently)
|
||||
|
|
||||
+--- Metadata ingestion
|
||||
+--- Search index maintenance (implicit: Postgres FTS/trigram indexes)
|
||||
```
|
||||
|
||||
## Module boundaries (`app/`)
|
||||
|
||||
| Module | Responsibility |
|
||||
|---|---|
|
||||
| `app/core/` | Config, DB engine, Redis client, structured logging, security/rate-limiting, job queue connection. No business logic. |
|
||||
| `app/models/` | SQLAlchemy 2.0 ORM models — the schema. See `docs/database.md`. |
|
||||
| `app/schemas/` | Pydantic **response** models. API responses are always built from these, never from serialized ORM objects directly (section 18). |
|
||||
| `app/repositories/` | Raw data access (SQLAlchemy queries) per aggregate, plus `polymorphic.py` for the shared credit/artwork/video/external-id tables. No response shaping. |
|
||||
| `app/services/` | Business logic: assembling API responses from repositories (`movie_service.py`, ...), caching (`cache.py`), search (`search/`), and the ingestion pipeline (`metadata/`, `ingestion/`). |
|
||||
| `app/providers/` | Provider adapters (`MetadataProvider` ABC + `TMDbProvider`). Fetch **raw** data only — never write to the database. |
|
||||
| `app/api/v1/` | FastAPI routers. Thin: parse request, call a service, translate service exceptions to HTTP errors. |
|
||||
| `app/workers/` | arq worker process + task functions, thin wrappers around `app/services/ingestion/runner.py`. |
|
||||
|
||||
Data flows in one direction: `providers` → `services/metadata` (normalize) →
|
||||
`services/metadata/matcher` (match) → `repositories` (persist) →
|
||||
`services/*_service` (assemble response) → `schemas` (validate/serialize) →
|
||||
`api/v1` (HTTP). Nothing skips a layer; in particular, provider adapters
|
||||
never touch the database, and routers never touch SQLAlchemy models
|
||||
directly.
|
||||
|
||||
## Provider abstraction & ingestion pipeline (sections 13–15)
|
||||
|
||||
```
|
||||
Provider.get_movie(external_id) # raw dict, provider-shaped
|
||||
-> normalize_movie() # -> NormalizedMovie, provider-agnostic
|
||||
-> match_entity() # exact external-ID match, or NEW
|
||||
-> ingest_movie() # upsert Movie + satellite tables
|
||||
-> upsert_provider_record() # provenance: raw payload + match confidence
|
||||
```
|
||||
|
||||
`MetadataProvider` (`app/providers/base.py`) is the only thing that knows
|
||||
TMDb-specific request shapes. A second provider (IMDb dataset export,
|
||||
Wikidata, ...) means writing a new adapter that returns the same raw shape
|
||||
TMDb's endpoints return, or a new `normalize_*` sibling if its shape differs
|
||||
— nothing in `app/services/ingestion/` or downstream needs to change.
|
||||
|
||||
**Matching policy is deliberately conservative** (section 14): only an exact
|
||||
external-ID match reuses an existing Aurora entity. Everything else creates
|
||||
a new entity. Aurora never guesses via fuzzy title matching that two
|
||||
providers' records are "the same movie" — that risk (wrongly merging two
|
||||
different films) is worse than the cost of a duplicate until a human-
|
||||
reviewable matching workflow exists (tracked in `docs/database.md#known-limitations`).
|
||||
|
||||
Ingestion always runs through workers, never inline in an HTTP request
|
||||
(section 15). The admin API (`POST /api/v1/admin/ingest/movie`) only
|
||||
enqueues an arq job and returns immediately; `app/workers/tasks.py` picks it
|
||||
up and calls `app/services/ingestion/runner.py`, which tracks the job's
|
||||
lifecycle in `ingestion_jobs` (idempotent via a unique `idempotency_key` —
|
||||
re-triggering ingestion for the same provider + external ID reuses the same
|
||||
job record rather than double-processing).
|
||||
|
||||
## Search (section 16)
|
||||
|
||||
`app/services/search/base.py:SearchProvider` is the abstraction; API
|
||||
handlers and `SearchService` only depend on it, never on SQL directly.
|
||||
`PostgreSQLSearchProvider` blends `ts_rank_cd` (full-text) with `pg_trgm`
|
||||
`similarity()` (typo tolerance) across primary titles/names and alternate
|
||||
titles. Swapping in `OpenSearchProvider` later means implementing the same
|
||||
`SearchProvider.search()` contract — no caller changes.
|
||||
|
||||
## Caching (section 21)
|
||||
|
||||
`app/services/cache.py` wraps Redis with fail-open semantics: every read/
|
||||
write function catches Redis errors and behaves as a cache miss / no-op
|
||||
rather than raising, so Redis being down degrades performance, not
|
||||
correctness. `app/api/deps.py:cached_response()` is the read-through helper
|
||||
used by the movie/person detail endpoints (the read-heaviest, per section
|
||||
21). Cache keys are versioned (`aurora:v1:...`) so a response-shape change
|
||||
(bumping `CACHE_VERSION`) doesn't require a manual flush.
|
||||
|
||||
Cache invalidation on write: 0.1 has no write API yet (ingestion writes
|
||||
directly to Postgres via workers, not through the cached read endpoints), so
|
||||
there is nothing to invalidate today. The moment a write endpoint exists,
|
||||
it must call `app/services/cache.py:cache_delete` for the affected keys —
|
||||
tracked as a Known limitation.
|
||||
|
||||
## Security (section 22)
|
||||
|
||||
- CORS restricted to `CORS_ORIGINS` (comma-separated env var), not `*`.
|
||||
- Every response carries `X-Content-Type-Options`, `X-Frame-Options`,
|
||||
`Referrer-Policy` (see `app/main.py` middleware).
|
||||
- Request bodies over 2 MiB are rejected before parsing.
|
||||
- Redis-backed fixed-window rate limiting (`app/core/security.py`), fails
|
||||
open if Redis is unreachable (same reasoning as caching).
|
||||
- Admin/provider endpoints (`/api/v1/admin/*`) require `X-API-Key` matching
|
||||
`AURORA_ADMIN_API_KEYS`; public read endpoints require no auth but are
|
||||
rate-limited.
|
||||
- No provider credentials are hard-coded; `TMDB_API_KEY` is read from the
|
||||
environment and the TMDb adapter raises a clear
|
||||
`ProviderNotConfiguredError` (not a fake success) if it's missing.
|
||||
- Secrets never committed — `.env.example` holds placeholders only.
|
||||
|
||||
Not yet implemented (tracked): encryption-at-rest for provider credentials
|
||||
beyond "not committed to Git" (there's currently exactly one credential,
|
||||
`TMDB_API_KEY`, sourced from the environment — a secrets manager integration
|
||||
is the natural next step once there are more); audit logging for admin
|
||||
actions beyond structured request logs.
|
||||
|
||||
## Observability (section 23)
|
||||
|
||||
Structured logging via `structlog`, JSON in non-development environments.
|
||||
Every request gets a correlation ID (`X-Request-ID`, generated if the client
|
||||
doesn't send one) bound into the log context for that request's lifetime
|
||||
(`app/core/logging.py`, `app/main.py:request_context_middleware`).
|
||||
`GET /metrics` exposes Prometheus counters/histograms (HTTP request count,
|
||||
latency by route) — compatible with Prometheus/Grafana; nothing in local
|
||||
development requires the observability stack to be running.
|
||||
|
||||
## Health checks (section 24)
|
||||
|
||||
`GET /health` — liveness only (process is up). `GET /ready` — checks
|
||||
PostgreSQL connectivity (always required) and Redis (required only if
|
||||
`REDIS_REQUIRED=true`, since Redis is disposable — see `docs/database.md`).
|
||||
Both are mounted twice: unprefixed (`/health`, `/ready`, for
|
||||
Docker/Traefik health checks) and under `/api/v1` (per section 17's endpoint
|
||||
list).
|
||||
|
||||
## Statelessness & horizontal scaling (section 1/2)
|
||||
|
||||
The API holds no in-process state across requests — no server-side
|
||||
sessions, no in-memory caches keyed by request. Every dependency
|
||||
(`get_db`, `get_redis`, `get_queue`) is request-scoped and backed by a
|
||||
shared pool. This is what makes `docker-compose.prod.yml`'s `api: replicas:
|
||||
3` behind Traefik correct without sticky sessions. Workers
|
||||
(`docker-compose.prod.yml`'s `worker: replicas: 2`) scale independently of
|
||||
the API — arq's Redis-backed queue ensures each job is claimed by exactly
|
||||
one worker.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,165 @@
|
||||
# Aurora Core — Database
|
||||
|
||||
PostgreSQL is authoritative (principle 4). Nothing important lives only in
|
||||
Redis. See [erd.md](erd.md) for the full entity-relationship diagram.
|
||||
|
||||
## Aurora IDs vs. primary keys
|
||||
|
||||
Every table's primary key is a UUIDv7 (`app/models/base.py:UUIDPrimaryKeyMixin`),
|
||||
generated in the application, not the database. UUIDv7 is time-ordered, so it
|
||||
stays btree-friendly (unlike UUIDv4) and can double as a rough creation-order
|
||||
cursor.
|
||||
|
||||
The public "Aurora ID" strings clients see (`aur_movie_01m1q7p1c...`) are a
|
||||
lossless, sortable Crockford-base32 encoding of that same UUID, produced by
|
||||
`app/core/ids.py:aurora_id()`. Encoding (not a second stored column) keeps
|
||||
the public ID and the primary key from ever drifting apart. See section 3 of
|
||||
`docs/aurora-core-spec.md` for why Aurora never uses a provider's ID (TMDb,
|
||||
IMDb, ...) as a primary key.
|
||||
|
||||
## Reference data
|
||||
|
||||
- `languages` (ISO 639-1), `countries` (ISO 3166-1 alpha-2), `certifications`
|
||||
(country + rating system + rating, e.g. US/MPA/PG-13) — populated
|
||||
incrementally as ingestion encounters new codes (`get_or_create_*` in
|
||||
`app/repositories/reference_repository.py`), not preloaded. A curated
|
||||
seed list is a reasonable follow-up (see Known limitations).
|
||||
- `genres`, `keywords` — deduplicated by a slugified name. Two providers'
|
||||
"Sci-Fi" and "Science Fiction" are **not** currently merged; that requires
|
||||
a curated synonym table, deliberately deferred (section 33 lists dedup
|
||||
*foundations*, not a finished thesaurus).
|
||||
|
||||
## Movies / TV
|
||||
|
||||
`movies` and `series` hold the record itself; per-locale and per-relationship
|
||||
data lives in satellite tables (`movie_alternate_titles`,
|
||||
`movie_spoken_languages`, `movie_production_countries`, `release_dates`,
|
||||
`movie_recommendations`, and the `series_*` equivalents) rather than JSON
|
||||
blobs, per section 26. `seasons` and `episodes` form a strict
|
||||
`series -> season -> episode` hierarchy; `episodes.series_id` is denormalized
|
||||
(in addition to `season_id`) purely so episode queries don't need to hop
|
||||
through `seasons` to filter/join by series.
|
||||
|
||||
`movies.status` / `series.status` are native Postgres enums, values `.value`
|
||||
of `app/models/enums.py:MovieStatus`/`SeriesStatus` — see "Enum storage"
|
||||
below.
|
||||
|
||||
## Credits
|
||||
|
||||
`credits` is the flexible cast/crew system from section 7: one row per
|
||||
(person, media entity, department/job/character), not per-movie text blobs.
|
||||
`department == "Acting"` rows carry `character`/`billing_order`; everything
|
||||
else carries `job`. Series-level "Creator" credits and episode
|
||||
`is_guest` (guest stars) reuse the same table rather than adding new ones.
|
||||
|
||||
## Companies / Collections
|
||||
|
||||
A single `companies` table serves production companies, studios, networks,
|
||||
and distributors, distinguished by `company_type` (section 8) — a TV network
|
||||
and a movie studio have the same shape (name, logo, country, parent
|
||||
company), so a separate `Network` entity would just duplicate columns.
|
||||
`collections` (franchises) are first-class, with their own artwork and
|
||||
`collection_relations` for shared-universe links between collections
|
||||
(section 9).
|
||||
|
||||
## Artwork / Videos
|
||||
|
||||
Never a `poster_url` column (section 10/11). `artwork` and `videos` are
|
||||
their own tables carrying provider, provider-native asset ID, dimensions,
|
||||
and (for artwork) `attribution`/`license` fields — Aurora doesn't assume
|
||||
externally-sourced images can be freely redistributed. Aurora never
|
||||
downloads video media; `videos.url` is a reference to the provider's
|
||||
hosted copy (YouTube, Vimeo, ...).
|
||||
|
||||
## Polymorphic associations
|
||||
|
||||
`credits`, `artwork`, `videos`, `external_ids`, and `release_dates` attach to
|
||||
more than one entity type via an `(entity_type, entity_id)` pair instead of a
|
||||
typed foreign key, because a single column can't reference "movies.id or
|
||||
series.id or people.id depending on a sibling column" in Postgres. The
|
||||
tradeoff: **the schema does not enforce that `entity_id` actually points at
|
||||
a real row of `entity_type`.** That integrity is enforced by the
|
||||
repository/service layer (`app/repositories/polymorphic.py`,
|
||||
`app/services/ingestion/*`), which is the only code path allowed to write
|
||||
these tables. If Core grows enough that this becomes a real risk, the
|
||||
standard fix is per-type junction tables (`movie_credits`, `series_credits`,
|
||||
...) at the cost of losing the single shared query path — deliberately not
|
||||
done for 0.1.
|
||||
|
||||
## External IDs & provenance
|
||||
|
||||
`external_ids` is the generalized external-identifier system (section 12):
|
||||
`(provider, entity_type, external_id)` is unique (one Aurora entity per
|
||||
provider-native ID), and `(entity_type, entity_id, provider)` is unique (one
|
||||
mapping per provider per entity).
|
||||
|
||||
`provider_records` is the full provenance ledger (section 14): one row per
|
||||
`(provider, entity_type, external_id)` holding the raw payload
|
||||
(`raw_data` JSONB), `match_confidence`, and which `ingestion_job` produced
|
||||
it. Entity-level provenance is coarse — `movies.source_provider_id` /
|
||||
`series.source_provider_id` point at "the provider that was authoritative at
|
||||
creation time" for quick display; full field-level provenance (which
|
||||
provider supplied *this specific* field) is not tracked in 0.1. See Known
|
||||
limitations.
|
||||
|
||||
`ingestion_jobs` tracks async work with a unique `idempotency_key` (e.g.
|
||||
`tmdb:movie:603`) so re-triggering the same sync doesn't double-enqueue —
|
||||
see `app/services/ingestion/runner.py`.
|
||||
|
||||
## Enum storage
|
||||
|
||||
Python enums in `app/models/enums.py` are `enum.StrEnum` (member value ==
|
||||
member name's meaning, e.g. `MovieStatus.RELEASED == "released"`).
|
||||
SQLAlchemy's `Enum` type stores `.name` (`"RELEASED"`) by default, which
|
||||
would silently diverge from the lowercase strings the API and ingestion code
|
||||
use everywhere. `app/models/base.py:_enum_column()` overrides this via
|
||||
`values_callable` so the database, the ORM, and the API all agree on the
|
||||
same lowercase strings.
|
||||
|
||||
## Migrations
|
||||
|
||||
Alembic (section 27); the application **never** creates tables at startup —
|
||||
see `app/core/database.py`. The initial migration
|
||||
(`migrations/versions/0001_initial_schema.py`) is hand-written rather than
|
||||
autogenerated (no throwaway Postgres was available in the environment this
|
||||
was built in — see the root README's Known Limitations), but was written
|
||||
directly from the same models this document describes and validated with
|
||||
`alembic history`; treat a first `alembic upgrade head` against a scratch
|
||||
database as the real acceptance test, and autogenerate future migrations
|
||||
against it (`alembic revision --autogenerate`) rather than the empty
|
||||
database this one was written from.
|
||||
|
||||
```bash
|
||||
alembic upgrade head # apply all migrations
|
||||
alembic revision --autogenerate -m "add xyz" # generate the next one
|
||||
alembic downgrade -1 # roll back one revision
|
||||
```
|
||||
|
||||
Integration/API tests build schema directly from `Base.metadata.create_all`
|
||||
(see `tests/conftest.py`) for speed and to catch model drift quickly — this
|
||||
is a testing shortcut, not how the running application ever gets its schema.
|
||||
|
||||
## Search indexes
|
||||
|
||||
`pg_trgm` (trigram similarity, for typo-tolerant matching) plus functional
|
||||
GIN indexes over `to_tsvector('simple', title || ' ' || original_title)` on
|
||||
`movies`/`series`, `name` on `people`/`collections`/`companies`, and trigram
|
||||
indexes on the alternate-title tables. See
|
||||
`app/services/search/postgresql_provider.py` and section 16 of the spec.
|
||||
|
||||
## Known limitations (tracked, not hidden)
|
||||
|
||||
- No curated language/country seed data — rows are created on first
|
||||
encounter during ingestion, so `english_name`/`name` are sometimes just
|
||||
the code uppercased until backfilled.
|
||||
- No fuzzy/candidate entity matching — only exact external-ID matches are
|
||||
auto-linked (section 14: never silently merge ambiguous matches). A
|
||||
review queue for AMBIGUOUS matches is the natural next step once Core has
|
||||
an admin UI to act on it.
|
||||
- Field-level provenance (which provider supplied *this* value) is not
|
||||
tracked, only entity-level (`source_provider_id`) and per-provider raw
|
||||
payloads (`provider_records.raw_data`).
|
||||
- Satellite associations (credits, artwork, videos, genre/keyword/company
|
||||
links) are replaced wholesale on each re-sync rather than diffed, so their
|
||||
Aurora IDs are not stable across re-ingestion — see
|
||||
`app/services/ingestion/movie_ingestion.py`.
|
||||
+201
@@ -0,0 +1,201 @@
|
||||
# Aurora Core — Entity-Relationship Diagram
|
||||
|
||||
This is the Core 0.1 schema (36 tables, see [database.md](database.md) for
|
||||
column-level detail). Association/junction tables are drawn as relationships
|
||||
directly where Mermaid supports it; a few many-to-many tables with their own
|
||||
attributes (`movie_companies.role`, `credits.*`) are drawn as entities.
|
||||
|
||||
```mermaid
|
||||
erDiagram
|
||||
MOVIE ||--o{ MOVIE_ALTERNATE_TITLE : has
|
||||
MOVIE ||--o{ MOVIE_SPOKEN_LANGUAGE : has
|
||||
MOVIE ||--o{ MOVIE_PRODUCTION_COUNTRY : has
|
||||
MOVIE ||--o{ RELEASE_DATE : has
|
||||
MOVIE ||--o{ MOVIE_RECOMMENDATION : recommends
|
||||
MOVIE }o--o{ GENRE : "movie_genres"
|
||||
MOVIE }o--o{ KEYWORD : "movie_keywords"
|
||||
MOVIE }o--o{ COMPANY : "movie_companies"
|
||||
MOVIE }o--|| COLLECTION : "belongs to"
|
||||
MOVIE ||--o{ CREDIT : "cast/crew"
|
||||
MOVIE ||--o{ ARTWORK : "images"
|
||||
MOVIE ||--o{ VIDEO : "trailers"
|
||||
MOVIE ||--o{ EXTERNAL_ID : "provider refs"
|
||||
|
||||
SERIES ||--o{ SEASON : has
|
||||
SEASON ||--o{ EPISODE : has
|
||||
SERIES ||--o{ SERIES_ALTERNATE_TITLE : has
|
||||
SERIES ||--o{ SERIES_SPOKEN_LANGUAGE : has
|
||||
SERIES ||--o{ SERIES_PRODUCTION_COUNTRY : has
|
||||
SERIES }o--o{ GENRE : "series_genres"
|
||||
SERIES }o--o{ KEYWORD : "series_keywords"
|
||||
SERIES }o--o{ COMPANY : "series_companies (network/studio)"
|
||||
SERIES ||--o{ CREDIT : "cast/crew/creators"
|
||||
SERIES ||--o{ ARTWORK : "images"
|
||||
SERIES ||--o{ VIDEO : "trailers"
|
||||
SERIES ||--o{ EXTERNAL_ID : "provider refs"
|
||||
EPISODE ||--o{ CREDIT : "cast/guest stars/crew"
|
||||
EPISODE ||--o{ ARTWORK : "stills"
|
||||
EPISODE ||--o{ EXTERNAL_ID : "provider refs"
|
||||
SEASON ||--o{ ARTWORK : "posters"
|
||||
SEASON ||--o{ EXTERNAL_ID : "provider refs"
|
||||
|
||||
COLLECTION ||--o{ MOVIE : "collection_movies"
|
||||
COLLECTION ||--o{ SERIES : "collection_series"
|
||||
COLLECTION ||--o{ COLLECTION : "collection_relations"
|
||||
COLLECTION ||--o{ ARTWORK : "images"
|
||||
|
||||
COMPANY ||--o{ COMPANY : "parent_company"
|
||||
COMPANY ||--o{ ARTWORK : "logos"
|
||||
COMPANY ||--o{ EXTERNAL_ID : "provider refs"
|
||||
COMPANY }o--o| COUNTRY : "based in"
|
||||
|
||||
PERSON ||--o{ CREDIT : "credited on"
|
||||
PERSON ||--o{ ARTWORK : "profile images"
|
||||
PERSON ||--o{ EXTERNAL_ID : "provider refs"
|
||||
|
||||
PROVIDER ||--o{ PROVIDER_RECORD : "provenance"
|
||||
PROVIDER ||--o{ INGESTION_JOB : "runs"
|
||||
PROVIDER ||--o{ EXTERNAL_ID : "issues"
|
||||
PROVIDER ||--o{ ARTWORK : "sourced"
|
||||
PROVIDER ||--o{ VIDEO : "sourced"
|
||||
PROVIDER ||--o{ CREDIT : "sourced"
|
||||
INGESTION_JOB ||--o{ PROVIDER_RECORD : produces
|
||||
|
||||
COUNTRY ||--o{ CERTIFICATION : defines
|
||||
RELEASE_DATE }o--o| CERTIFICATION : rated
|
||||
|
||||
MOVIE {
|
||||
uuid id PK
|
||||
string title
|
||||
string sort_title
|
||||
enum status
|
||||
date primary_release_date
|
||||
int runtime_minutes
|
||||
uuid collection_id FK
|
||||
uuid source_provider_id FK
|
||||
}
|
||||
SERIES {
|
||||
uuid id PK
|
||||
string title
|
||||
enum status
|
||||
date first_air_date
|
||||
int number_of_seasons
|
||||
}
|
||||
SEASON {
|
||||
uuid id PK
|
||||
uuid series_id FK
|
||||
int season_number
|
||||
}
|
||||
EPISODE {
|
||||
uuid id PK
|
||||
uuid season_id FK
|
||||
uuid series_id FK
|
||||
int episode_number
|
||||
}
|
||||
PERSON {
|
||||
uuid id PK
|
||||
string name
|
||||
date birthday
|
||||
string known_for_department
|
||||
}
|
||||
CREDIT {
|
||||
uuid id PK
|
||||
uuid person_id FK
|
||||
string entity_type
|
||||
uuid entity_id
|
||||
string department
|
||||
string job
|
||||
string character
|
||||
}
|
||||
COMPANY {
|
||||
uuid id PK
|
||||
string name
|
||||
enum company_type
|
||||
uuid parent_company_id FK
|
||||
}
|
||||
COLLECTION {
|
||||
uuid id PK
|
||||
string name
|
||||
string slug
|
||||
}
|
||||
GENRE {
|
||||
uuid id PK
|
||||
string name
|
||||
}
|
||||
KEYWORD {
|
||||
uuid id PK
|
||||
string name
|
||||
}
|
||||
ARTWORK {
|
||||
uuid id PK
|
||||
string entity_type
|
||||
uuid entity_id
|
||||
enum artwork_type
|
||||
uuid provider_id FK
|
||||
string original_url
|
||||
}
|
||||
VIDEO {
|
||||
uuid id PK
|
||||
string entity_type
|
||||
uuid entity_id
|
||||
enum video_type
|
||||
uuid provider_id FK
|
||||
string url
|
||||
}
|
||||
EXTERNAL_ID {
|
||||
uuid id PK
|
||||
string entity_type
|
||||
uuid entity_id
|
||||
uuid provider_id FK
|
||||
string external_id
|
||||
}
|
||||
PROVIDER {
|
||||
uuid id PK
|
||||
string slug
|
||||
string name
|
||||
}
|
||||
PROVIDER_RECORD {
|
||||
uuid id PK
|
||||
uuid provider_id FK
|
||||
string entity_type
|
||||
string external_id
|
||||
uuid aurora_entity_id
|
||||
enum match_confidence
|
||||
jsonb raw_data
|
||||
}
|
||||
INGESTION_JOB {
|
||||
uuid id PK
|
||||
uuid provider_id FK
|
||||
string job_type
|
||||
enum status
|
||||
string idempotency_key
|
||||
}
|
||||
COUNTRY {
|
||||
string code PK
|
||||
string name
|
||||
}
|
||||
CERTIFICATION {
|
||||
uuid id PK
|
||||
string country_code FK
|
||||
string system
|
||||
string rating
|
||||
}
|
||||
RELEASE_DATE {
|
||||
uuid id PK
|
||||
uuid entity_id
|
||||
string country_code FK
|
||||
date release_date
|
||||
uuid certification_id FK
|
||||
}
|
||||
```
|
||||
|
||||
## Notes on the polymorphic tables
|
||||
|
||||
`CREDIT`, `ARTWORK`, `VIDEO`, `EXTERNAL_ID`, and `RELEASE_DATE` attach to more
|
||||
than one entity type (movie, series, season, episode, person, company,
|
||||
collection) via an `(entity_type, entity_id)` pair rather than a typed
|
||||
foreign key. Postgres can't enforce referential integrity across a
|
||||
polymorphic pair with a single `FOREIGN KEY`, so integrity there is enforced
|
||||
by the repository/service layer (`app/repositories/polymorphic.py`), not the
|
||||
schema. See [database.md](database.md#polymorphic-associations) for the
|
||||
tradeoff.
|
||||
Reference in New Issue
Block a user