Aurora-Core/docs/architecture.md

Aurora Core — Architecture

Aurora Core is a production-grade modular monolith (section 1/9 of docs/aurora-core-spec.md) — one deployable API codebase with clear internal module boundaries, not a set of microservices. It is the canonical metadata layer for the Aurora ecosystem; Plex, Overseerr, Radarr, Sonarr, and future Aurora applications are consumers of it, never dependencies of it.

Internet
    |
Traefik
    |
Aurora Core API replicas (stateless)
    |
    +--- PostgreSQL (authoritative)
    |
    +--- PgBouncer (production connection pooling)
    |
    +--- Redis (cache / rate limit / job queue — disposable)
    |
    +--- Background Workers (arq, scale independently)
            |
            +--- Metadata ingestion
            +--- Search index maintenance (implicit: Postgres FTS/trigram indexes)

Module boundaries (app/)

Module	Responsibility
app/core/	Config, DB engine, Redis client, structured logging, security/rate-limiting, job queue connection. No business logic.
app/models/	SQLAlchemy 2.0 ORM models — the schema. See docs/database.md.
app/schemas/	Pydantic response models. API responses are always built from these, never from serialized ORM objects directly (section 18).
app/repositories/	Raw data access (SQLAlchemy queries) per aggregate, plus polymorphic.py for the shared credit/artwork/video/external-id tables. No response shaping.
app/services/	Business logic: assembling API responses from repositories (movie_service.py, ...), caching (cache.py), search (search/), and the ingestion pipeline (metadata/, ingestion/).
app/providers/	Provider adapters (MetadataProvider ABC + TMDbProvider). Fetch raw data only — never write to the database.
app/api/v1/	FastAPI routers. Thin: parse request, call a service, translate service exceptions to HTTP errors.
app/workers/	arq worker process + task functions, thin wrappers around app/services/ingestion/runner.py.

Data flows in one direction: providers → services/metadata (normalize) → services/metadata/matcher (match) → repositories (persist) → services/*_service (assemble response) → schemas (validate/serialize) → api/v1 (HTTP). Nothing skips a layer; in particular, provider adapters never touch the database, and routers never touch SQLAlchemy models directly.

Provider abstraction & ingestion pipeline (sections 13–15)

Provider.get_movie(external_id)   # raw dict, provider-shaped
    -> normalize_movie()          # -> NormalizedMovie, provider-agnostic
    -> match_entity()             # exact external-ID match, or NEW
    -> ingest_movie()             # upsert Movie + satellite tables
    -> upsert_provider_record()   # provenance: raw payload + match confidence

MetadataProvider (app/providers/base.py) is the only thing that knows TMDb-specific request shapes. A second provider (IMDb dataset export, Wikidata, ...) means writing a new adapter that returns the same raw shape TMDb's endpoints return, or a new normalize_* sibling if its shape differs — nothing in app/services/ingestion/ or downstream needs to change.

Matching policy is deliberately conservative (section 14): only an exact external-ID match reuses an existing Aurora entity. Everything else creates a new entity. Aurora never guesses via fuzzy title matching that two providers' records are "the same movie" — that risk (wrongly merging two different films) is worse than the cost of a duplicate until a human- reviewable matching workflow exists (tracked in docs/database.md#known-limitations).

Ingestion always runs through workers, never inline in an HTTP request (section 15). The admin API (POST /api/v1/admin/ingest/movie) only enqueues an arq job and returns immediately; app/workers/tasks.py picks it up and calls app/services/ingestion/runner.py, which tracks the job's lifecycle in ingestion_jobs (idempotent via a unique idempotency_key — re-triggering ingestion for the same provider + external ID reuses the same job record rather than double-processing).

Search (section 16)

app/services/search/base.py:SearchProvider is the abstraction; API handlers and SearchService only depend on it, never on SQL directly. PostgreSQLSearchProvider blends ts_rank_cd (full-text) with pg_trgm similarity() (typo tolerance) across primary titles/names and alternate titles. Swapping in OpenSearchProvider later means implementing the same SearchProvider.search() contract — no caller changes.

Caching (section 21)

app/services/cache.py wraps Redis with fail-open semantics: every read/ write function catches Redis errors and behaves as a cache miss / no-op rather than raising, so Redis being down degrades performance, not correctness. app/api/deps.py:cached_response() is the read-through helper used by the movie/person detail endpoints (the read-heaviest, per section 21). Cache keys are versioned (aurora:v1:...) so a response-shape change (bumping CACHE_VERSION) doesn't require a manual flush.

Cache invalidation on write: 0.1 has no write API yet (ingestion writes directly to Postgres via workers, not through the cached read endpoints), so there is nothing to invalidate today. The moment a write endpoint exists, it must call app/services/cache.py:cache_delete for the affected keys — tracked as a Known limitation.

Security (section 22)

• CORS restricted to CORS_ORIGINS (comma-separated env var), not *.
• Every response carries X-Content-Type-Options, X-Frame-Options, Referrer-Policy (see app/main.py middleware).
• Request bodies over 2 MiB are rejected before parsing.
• Redis-backed fixed-window rate limiting (app/core/security.py), fails open if Redis is unreachable (same reasoning as caching).
• Admin/provider endpoints (/api/v1/admin/*) require X-API-Key matching AURORA_ADMIN_API_KEYS; public read endpoints require no auth but are rate-limited.
• No provider credentials are hard-coded; TMDB_API_KEY is read from the environment and the TMDb adapter raises a clear ProviderNotConfiguredError (not a fake success) if it's missing.
• Secrets never committed — .env.example holds placeholders only.

Not yet implemented (tracked): encryption-at-rest for provider credentials beyond "not committed to Git" (there's currently exactly one credential, TMDB_API_KEY, sourced from the environment — a secrets manager integration is the natural next step once there are more); audit logging for admin actions beyond structured request logs.

Observability (section 23)

Structured logging via structlog, JSON in non-development environments. Every request gets a correlation ID (X-Request-ID, generated if the client doesn't send one) bound into the log context for that request's lifetime (app/core/logging.py, app/main.py:request_context_middleware). GET /metrics exposes Prometheus counters/histograms (HTTP request count, latency by route) — compatible with Prometheus/Grafana; nothing in local development requires the observability stack to be running.

Health checks (section 24)

GET /health — liveness only (process is up). GET /ready — checks PostgreSQL connectivity (always required) and Redis (required only if REDIS_REQUIRED=true, since Redis is disposable — see docs/database.md). Both are mounted twice: unprefixed (/health, /ready, for Docker/Traefik health checks) and under /api/v1 (per section 17's endpoint list).

Statelessness & horizontal scaling (section 1/2)

The API holds no in-process state across requests — no server-side sessions, no in-memory caches keyed by request. Every dependency (get_db, get_redis, get_queue) is request-scoped and backed by a shared pool. This is what makes docker-compose.prod.yml's api: replicas: 3 behind Traefik correct without sticky sessions. Workers (docker-compose.prod.yml's worker: replicas: 2) scale independently of the API — arq's Redis-backed queue ensures each job is claimed by exactly one worker.