NixReaperandClaude Sonnet 5
cd4dd98adb
CI / lint (push) Canceled after 0s
CI / typecheck (push) Canceled after 0s
CI / unit-tests (push) Canceled after 0s
CI / integration-tests (push) Canceled after 0s
CI / container-build (push) Canceled after 0s
CI / dependency-audit (push) Canceled after 0s
arq only auto-retries a job on its own Retry exception (or CancelledError) -- never on an arbitrary exception a task raises. _run_ingestion caught failures, recorded them via mark_job_failed (which sets ingestion_jobs.status to "retrying" when attempts hasn't hit max_attempts), and then just re-raised the original exception -- which arq treats as a terminal failure. The DB said "retrying" forever; nothing ever put the job back on the queue. Confirmed live: cascading Better Call Saul's 6 seasons hit the get_or_create race fixed in the previous commit on 4 of them, and the worker's own shutdown summary showed "4 failed, 0 retries" -- every one of those was permanently lost despite max_attempts=5 suggesting otherwise. _run_ingestion now raises arq.Retry (capped exponential backoff) when mark_job_failed left the job in RETRYING state, so arq actually requeues it; only once attempts are exhausted does the original exception propagate as a genuine terminal failure. Co-Authored-By: Claude Sonnet 5 <[email protected]>
169 lines
8.8 KiB
Markdown
169 lines
8.8 KiB
Markdown
# Aurora Core — Architecture
|
||
|
||
Aurora Core is a production-grade **modular monolith** (section 1/9 of
|
||
`docs/aurora-core-spec.md`) — one deployable API codebase with clear internal
|
||
module boundaries, not a set of microservices. It is the canonical metadata
|
||
layer for the Aurora ecosystem; Plex, Overseerr, Radarr, Sonarr, and future
|
||
Aurora applications are consumers of it, never dependencies of it.
|
||
|
||
```
|
||
Internet
|
||
|
|
||
Traefik
|
||
|
|
||
Aurora Core API replicas (stateless)
|
||
|
|
||
+--- PostgreSQL (authoritative)
|
||
|
|
||
+--- PgBouncer (production connection pooling)
|
||
|
|
||
+--- Redis (cache / rate limit / job queue — disposable)
|
||
|
|
||
+--- Background Workers (arq, scale independently)
|
||
|
|
||
+--- Metadata ingestion
|
||
+--- Search index maintenance (implicit: Postgres FTS/trigram indexes)
|
||
```
|
||
|
||
## Module boundaries (`app/`)
|
||
|
||
| Module | Responsibility |
|
||
|---|---|
|
||
| `app/core/` | Config, DB engine, Redis client, structured logging, security/rate-limiting, job queue connection. No business logic. |
|
||
| `app/models/` | SQLAlchemy 2.0 ORM models — the schema. See `docs/database.md`. |
|
||
| `app/schemas/` | Pydantic **response** models. API responses are always built from these, never from serialized ORM objects directly (section 18). |
|
||
| `app/repositories/` | Raw data access (SQLAlchemy queries) per aggregate, plus `polymorphic.py` for the shared credit/artwork/video/external-id tables. No response shaping. |
|
||
| `app/services/` | Business logic: assembling API responses from repositories (`movie_service.py`, ...), caching (`cache.py`), search (`search/`), and the ingestion pipeline (`metadata/`, `ingestion/`). |
|
||
| `app/providers/` | Provider adapters (`MetadataProvider` ABC + `TMDbProvider`). Fetch **raw** data only — never write to the database. |
|
||
| `app/api/v1/` | FastAPI routers. Thin: parse request, call a service, translate service exceptions to HTTP errors. |
|
||
| `app/workers/` | arq worker process + task functions, thin wrappers around `app/services/ingestion/runner.py`. |
|
||
|
||
Data flows in one direction: `providers` → `services/metadata` (normalize) →
|
||
`services/metadata/matcher` (match) → `repositories` (persist) →
|
||
`services/*_service` (assemble response) → `schemas` (validate/serialize) →
|
||
`api/v1` (HTTP). Nothing skips a layer; in particular, provider adapters
|
||
never touch the database, and routers never touch SQLAlchemy models
|
||
directly.
|
||
|
||
## Provider abstraction & ingestion pipeline (sections 13–15)
|
||
|
||
```
|
||
Provider.get_movie(external_id) # raw dict, provider-shaped
|
||
-> normalize_movie() # -> NormalizedMovie, provider-agnostic
|
||
-> match_entity() # exact external-ID match, or NEW
|
||
-> ingest_movie() # upsert Movie + satellite tables
|
||
-> upsert_provider_record() # provenance: raw payload + match confidence
|
||
```
|
||
|
||
`MetadataProvider` (`app/providers/base.py`) is the only thing that knows
|
||
TMDb-specific request shapes. A second provider (IMDb dataset export,
|
||
Wikidata, ...) means writing a new adapter that returns the same raw shape
|
||
TMDb's endpoints return, or a new `normalize_*` sibling if its shape differs
|
||
— nothing in `app/services/ingestion/` or downstream needs to change.
|
||
|
||
**Matching policy is deliberately conservative** (section 14): only an exact
|
||
external-ID match reuses an existing Aurora entity. Everything else creates
|
||
a new entity. Aurora never guesses via fuzzy title matching that two
|
||
providers' records are "the same movie" — that risk (wrongly merging two
|
||
different films) is worse than the cost of a duplicate until a human-
|
||
reviewable matching workflow exists (tracked in `docs/database.md#known-limitations`).
|
||
|
||
Ingestion always runs through workers, never inline in an HTTP request
|
||
(section 15). The admin API (`POST /api/v1/admin/ingest/movie`) only
|
||
enqueues an arq job and returns immediately; `app/workers/tasks.py` picks it
|
||
up and calls `app/services/ingestion/runner.py`, which tracks the job's
|
||
lifecycle in `ingestion_jobs` (idempotent via a unique `idempotency_key` —
|
||
re-triggering ingestion for the same provider + external ID reuses the same
|
||
job record rather than double-processing).
|
||
|
||
Failures actually get retried, not just labeled retryable: arq only
|
||
auto-retries a job on its own `arq.Retry` exception (or `CancelledError`) —
|
||
never on an arbitrary exception a task raises — so `_run_ingestion` catches
|
||
the failure, records it, and if `ingestion_jobs.attempts` hasn't hit
|
||
`max_attempts` yet, raises `Retry(defer=...)` (capped exponential backoff)
|
||
so arq actually puts the job back on the queue instead of the DB saying
|
||
"retrying" while nothing retries it. Series ingestion cascading into
|
||
concurrent season jobs (`include_seasons`) is exactly what surfaces
|
||
transient failures worth retrying this way: two seasons sharing a guest
|
||
star can race to create the same `Person` row (see
|
||
`app/repositories/reference_repository.py:_create_or_recover`, which
|
||
recovers from that specific race directly; the arq retry here is the
|
||
backstop for whatever it doesn't).
|
||
|
||
## Search (section 16)
|
||
|
||
`app/services/search/base.py:SearchProvider` is the abstraction; API
|
||
handlers and `SearchService` only depend on it, never on SQL directly.
|
||
`PostgreSQLSearchProvider` blends `ts_rank_cd` (full-text) with `pg_trgm`
|
||
`similarity()` (typo tolerance) across primary titles/names and alternate
|
||
titles. Swapping in `OpenSearchProvider` later means implementing the same
|
||
`SearchProvider.search()` contract — no caller changes.
|
||
|
||
## Caching (section 21)
|
||
|
||
`app/services/cache.py` wraps Redis with fail-open semantics: every read/
|
||
write function catches Redis errors and behaves as a cache miss / no-op
|
||
rather than raising, so Redis being down degrades performance, not
|
||
correctness. `app/api/deps.py:cached_response()` is the read-through helper
|
||
used by the movie/person detail endpoints (the read-heaviest, per section
|
||
21). Cache keys are versioned (`aurora:v1:...`) so a response-shape change
|
||
(bumping `CACHE_VERSION`) doesn't require a manual flush.
|
||
|
||
Cache invalidation on write: 0.1 has no write API yet (ingestion writes
|
||
directly to Postgres via workers, not through the cached read endpoints), so
|
||
there is nothing to invalidate today. The moment a write endpoint exists,
|
||
it must call `app/services/cache.py:cache_delete` for the affected keys —
|
||
tracked as a Known limitation.
|
||
|
||
## Security (section 22)
|
||
|
||
- CORS restricted to `CORS_ORIGINS` (comma-separated env var), not `*`.
|
||
- Every response carries `X-Content-Type-Options`, `X-Frame-Options`,
|
||
`Referrer-Policy` (see `app/main.py` middleware).
|
||
- Request bodies over 2 MiB are rejected before parsing.
|
||
- Redis-backed fixed-window rate limiting (`app/core/security.py`), fails
|
||
open if Redis is unreachable (same reasoning as caching).
|
||
- Admin/provider endpoints (`/api/v1/admin/*`) require `X-API-Key` matching
|
||
`AURORA_ADMIN_API_KEYS`; public read endpoints require no auth but are
|
||
rate-limited.
|
||
- No provider credentials are hard-coded; `TMDB_API_KEY` is read from the
|
||
environment and the TMDb adapter raises a clear
|
||
`ProviderNotConfiguredError` (not a fake success) if it's missing.
|
||
- Secrets never committed — `.env.example` holds placeholders only.
|
||
|
||
Not yet implemented (tracked): encryption-at-rest for provider credentials
|
||
beyond "not committed to Git" (there's currently exactly one credential,
|
||
`TMDB_API_KEY`, sourced from the environment — a secrets manager integration
|
||
is the natural next step once there are more); audit logging for admin
|
||
actions beyond structured request logs.
|
||
|
||
## Observability (section 23)
|
||
|
||
Structured logging via `structlog`, JSON in non-development environments.
|
||
Every request gets a correlation ID (`X-Request-ID`, generated if the client
|
||
doesn't send one) bound into the log context for that request's lifetime
|
||
(`app/core/logging.py`, `app/main.py:request_context_middleware`).
|
||
`GET /metrics` exposes Prometheus counters/histograms (HTTP request count,
|
||
latency by route) — compatible with Prometheus/Grafana; nothing in local
|
||
development requires the observability stack to be running.
|
||
|
||
## Health checks (section 24)
|
||
|
||
`GET /health` — liveness only (process is up). `GET /ready` — checks
|
||
PostgreSQL connectivity (always required) and Redis (required only if
|
||
`REDIS_REQUIRED=true`, since Redis is disposable — see `docs/database.md`).
|
||
Both are mounted twice: unprefixed (`/health`, `/ready`, for
|
||
Docker/Traefik health checks) and under `/api/v1` (per section 17's endpoint
|
||
list).
|
||
|
||
## Statelessness & horizontal scaling (section 1/2)
|
||
|
||
The API holds no in-process state across requests — no server-side
|
||
sessions, no in-memory caches keyed by request. Every dependency
|
||
(`get_db`, `get_redis`, `get_queue`) is request-scoped and backed by a
|
||
shared pool. This is what makes `docker-compose.prod.yml`'s `api: replicas:
|
||
3` behind Traefik correct without sticky sessions. Workers
|
||
(`docker-compose.prod.yml`'s `worker: replicas: 2`) scale independently of
|
||
the API — arq's Redis-backed queue ensures each job is claimed by exactly
|
||
one worker.
|