2026-09-04 17:34:03 -05:00
# Aurora Core — Architecture
Aurora Core is a production-grade **modular monolith** (section 1/9 of
`docs/aurora-core-spec.md` ) — one deployable API codebase with clear internal
module boundaries, not a set of microservices. It is the canonical metadata
layer for the Aurora ecosystem; Plex, Overseerr, Radarr, Sonarr, and future
Aurora applications are consumers of it, never dependencies of it.
```
Internet
|
Traefik
|
Aurora Core API replicas (stateless)
|
+--- PostgreSQL (authoritative)
|
+--- PgBouncer (production connection pooling)
|
+--- Redis (cache / rate limit / job queue — disposable)
|
+--- Background Workers (arq, scale independently)
|
+--- Metadata ingestion
+--- Search index maintenance (implicit: Postgres FTS/trigram indexes)
```
## Module boundaries (`app/`)
| Module | Responsibility |
|---|---|
| `app/core/` | Config, DB engine, Redis client, structured logging, security/rate-limiting, job queue connection. No business logic. |
| `app/models/` | SQLAlchemy 2.0 ORM models — the schema. See `docs/database.md` . |
| `app/schemas/` | Pydantic **response** models. API responses are always built from these, never from serialized ORM objects directly (section 18). |
| `app/repositories/` | Raw data access (SQLAlchemy queries) per aggregate, plus `polymorphic.py` for the shared credit/artwork/video/external-id tables. No response shaping. |
| `app/services/` | Business logic: assembling API responses from repositories (`movie_service.py` , ...), caching (`cache.py` ), search (`search/` ), and the ingestion pipeline (`metadata/` , `ingestion/` ). |
| `app/providers/` | Provider adapters (`MetadataProvider` ABC + `TMDbProvider` ). Fetch **raw** data only — never write to the database. |
| `app/api/v1/` | FastAPI routers. Thin: parse request, call a service, translate service exceptions to HTTP errors. |
| `app/workers/` | arq worker process + task functions, thin wrappers around `app/services/ingestion/runner.py` . |
Data flows in one direction: `providers` → `services/metadata` (normalize) →
`services/metadata/matcher` (match) → `repositories` (persist) →
`services/*_service` (assemble response) → `schemas` (validate/serialize) →
`api/v1` (HTTP). Nothing skips a layer; in particular, provider adapters
never touch the database, and routers never touch SQLAlchemy models
directly.
## Provider abstraction & ingestion pipeline (sections 13–
```
Provider.get_movie(external_id) # raw dict, provider-shaped
-> normalize_movie() # -> NormalizedMovie, provider-agnostic
-> match_entity() # exact external-ID match, or NEW
-> ingest_movie() # upsert Movie + satellite tables
-> upsert_provider_record() # provenance: raw payload + match confidence
```
`MetadataProvider` (`app/providers/base.py` ) is the only thing that knows
TMDb-specific request shapes. A second provider (IMDb dataset export,
Wikidata, ...) means writing a new adapter that returns the same raw shape
TMDb's endpoints return, or a new `normalize_*` sibling if its shape differs
— nothing in `app/services/ingestion/` or downstream needs to change.
**Matching policy is deliberately conservative** (section 14): only an exact
external-ID match reuses an existing Aurora entity. Everything else creates
a new entity. Aurora never guesses via fuzzy title matching that two
providers' records are "the same movie" — that risk (wrongly merging two
different films) is worse than the cost of a duplicate until a human-
reviewable matching workflow exists (tracked in `docs/database.md#known-limitations` ).
Ingestion always runs through workers, never inline in an HTTP request
(section 15). The admin API (`POST /api/v1/admin/ingest/movie` ) only
enqueues an arq job and returns immediately; `app/workers/tasks.py` picks it
up and calls `app/services/ingestion/runner.py` , which tracks the job's
lifecycle in `ingestion_jobs` (idempotent via a unique `idempotency_key` —
re-triggering ingestion for the same provider + external ID reuses the same
job record rather than double-processing).
2026-09-05 02:31:43 -05:00
Failures actually get retried, not just labeled retryable: arq only
auto-retries a job on its own `arq.Retry` exception (or `CancelledError` ) —
never on an arbitrary exception a task raises — so `_run_ingestion` catches
the failure, records it, and if `ingestion_jobs.attempts` hasn't hit
`max_attempts` yet, raises `Retry(defer=...)` (capped exponential backoff)
so arq actually puts the job back on the queue instead of the DB saying
"retrying" while nothing retries it. Series ingestion cascading into
concurrent season jobs (`include_seasons` ) is exactly what surfaces
transient failures worth retrying this way: two seasons sharing a guest
star can race to create the same `Person` row (see
`app/repositories/reference_repository.py:_create_or_recover` , which
recovers from that specific race directly; the arq retry here is the
backstop for whatever it doesn't).
2026-09-05 03:26:28 -05:00
## Artwork self-hosting (section 10)
```
ingest_movie/ingest_series/ingest_person succeeds
-> tasks.py: get_primary_artwork_for_download(entity_type, [entity_id, ...])
-> one download_artwork_task enqueued per primary artwork id
-> download_artwork(): fetch original_url, verify it's an image, cap
size, sha256 it, storage.save(), set Artwork.local_path + checksum
```
Every image API responses reference is stored with its provider CDN URL
(`original_url` ) from the moment it's ingested; `local_path` starts empty
and is filled in by this pipeline running asynchronously afterward, so
responses transparently point at TMDb until a local copy exists, then
switch over. Only artwork marked `is_primary` (the one image TMDb itself
designates for an entity, not the dozens of alternates also ingested) gets
downloaded automatically — see `docs/database.md#artwork-storage-self-hosting` .
Storage is behind an `ArtworkStorage` interface
(`app/services/artwork/storage.py` ) with one implementation,
`LocalDiskStorage` — a shared Docker volume between the `api` and `worker`
containers (`docker-compose.yml` 's `aurora_media_data` ), which is enough
for this project's actual deployment shape (one Docker host, principle 9:
avoid premature complexity) but assumes exactly that. A deployment that
outgrew a single host would implement the same interface against S3/R2
instead, without the download pipeline or callers changing.
This is also why `app/repositories/polymorphic.py:sync_artwork` upserts
artwork by `provider_asset_id` instead of the delete-then-recreate pattern
every other satellite table uses (movie/series/person credits, videos,
genre/keyword/company links): a delete-then-recreate would generate a fresh
`Artwork.id` with `local_path=None` on every re-ingestion, silently
discarding every downloaded copy and forcing a full re-download each sync.
2026-09-04 17:34:03 -05:00
## Search (section 16)
`app/services/search/base.py:SearchProvider` is the abstraction; API
handlers and `SearchService` only depend on it, never on SQL directly.
`PostgreSQLSearchProvider` blends `ts_rank_cd` (full-text) with `pg_trgm`
`similarity()` (typo tolerance) across primary titles/names and alternate
titles. Swapping in `OpenSearchProvider` later means implementing the same
`SearchProvider.search()` contract — no caller changes.
## Caching (section 21)
`app/services/cache.py` wraps Redis with fail-open semantics: every read/
write function catches Redis errors and behaves as a cache miss / no-op
rather than raising, so Redis being down degrades performance, not
correctness. `app/api/deps.py:cached_response()` is the read-through helper
used by the movie/person detail endpoints (the read-heaviest, per section
21). Cache keys are versioned (`aurora:v1:...` ) so a response-shape change
(bumping `CACHE_VERSION` ) doesn't require a manual flush.
Cache invalidation on write: 0.1 has no write API yet (ingestion writes
directly to Postgres via workers, not through the cached read endpoints), so
there is nothing to invalidate today. The moment a write endpoint exists,
it must call `app/services/cache.py:cache_delete` for the affected keys —
tracked as a Known limitation.
## Security (section 22)
- CORS restricted to `CORS_ORIGINS` (comma-separated env var), not `*` .
- Every response carries `X-Content-Type-Options` , `X-Frame-Options` ,
`Referrer-Policy` (see `app/main.py` middleware).
- Request bodies over 2 MiB are rejected before parsing.
- Redis-backed fixed-window rate limiting (`app/core/security.py` ), fails
open if Redis is unreachable (same reasoning as caching).
- Admin/provider endpoints (`/api/v1/admin/*` ) require `X-API-Key` matching
`AURORA_ADMIN_API_KEYS` ; public read endpoints require no auth but are
rate-limited.
- No provider credentials are hard-coded; `TMDB_API_KEY` is read from the
environment and the TMDb adapter raises a clear
`ProviderNotConfiguredError` (not a fake success) if it's missing.
- Secrets never committed — `.env.example` holds placeholders only.
Not yet implemented (tracked): encryption-at-rest for provider credentials
beyond "not committed to Git" (there's currently exactly one credential,
`TMDB_API_KEY` , sourced from the environment — a secrets manager integration
is the natural next step once there are more); audit logging for admin
actions beyond structured request logs.
## Observability (section 23)
Structured logging via `structlog` , JSON in non-development environments.
Every request gets a correlation ID (`X-Request-ID` , generated if the client
doesn't send one) bound into the log context for that request's lifetime
(`app/core/logging.py` , `app/main.py:request_context_middleware` ).
`GET /metrics` exposes Prometheus counters/histograms (HTTP request count,
latency by route) — compatible with Prometheus/Grafana; nothing in local
development requires the observability stack to be running.
## Health checks (section 24)
`GET /health` — liveness only (process is up). `GET /ready` — checks
PostgreSQL connectivity (always required) and Redis (required only if
`REDIS_REQUIRED=true` , since Redis is disposable — see `docs/database.md` ).
Both are mounted twice: unprefixed (`/health` , `/ready` , for
Docker/Traefik health checks) and under `/api/v1` (per section 17's endpoint
list).
## Statelessness & horizontal scaling (section 1/2)
The API holds no in-process state across requests — no server-side
sessions, no in-memory caches keyed by request. Every dependency
(`get_db` , `get_redis` , `get_queue` ) is request-scoped and backed by a
shared pool. This is what makes `docker-compose.prod.yml` 's `api: replicas:
3` behind Traefik correct without sticky sessions. Workers
(`docker-compose.prod.yml` 's `worker: replicas: 2` ) scale independently of
the API — arq's Redis-backed queue ensures each job is claimed by exactly
one worker.
