Aurora-Core/docs/architecture.md

Aurora Core — Architecture

Aurora Core is a production-grade modular monolith (section 1/9 of docs/aurora-core-spec.md) — one deployable API codebase with clear internal module boundaries, not a set of microservices. It is the canonical metadata layer for the Aurora ecosystem; Plex, Overseerr, Radarr, Sonarr, and future Aurora applications are consumers of it, never dependencies of it.

Internet
    |
Traefik
    |
Aurora Core API replicas (stateless)
    |
    +--- PostgreSQL (authoritative)
    |
    +--- PgBouncer (production connection pooling)
    |
    +--- Redis (cache / rate limit / job queue — disposable)
    |
    +--- Background Workers (arq, scale independently)
            |
            +--- Metadata ingestion
            +--- Search index maintenance (implicit: Postgres FTS/trigram indexes)

Module boundaries (app/)

Module	Responsibility
app/core/	Config, DB engine, Redis client, structured logging, security/rate-limiting, job queue connection. No business logic.
app/models/	SQLAlchemy 2.0 ORM models — the schema. See docs/database.md.
app/schemas/	Pydantic response models. API responses are always built from these, never from serialized ORM objects directly (section 18).
app/repositories/	Raw data access (SQLAlchemy queries) per aggregate, plus polymorphic.py for the shared credit/artwork/video/external-id tables. No response shaping.
app/services/	Business logic: assembling API responses from repositories (movie_service.py, ...), caching (cache.py), search (search/), and the ingestion pipeline (metadata/, ingestion/).
app/providers/	Provider adapters (MetadataProvider ABC + TMDbProvider). Fetch raw data only — never write to the database.
app/api/v1/	FastAPI routers. Thin: parse request, call a service, translate service exceptions to HTTP errors.
app/workers/	arq worker process + task functions, thin wrappers around app/services/ingestion/runner.py.

Data flows in one direction: providers → services/metadata (normalize) → services/metadata/matcher (match) → repositories (persist) → services/*_service (assemble response) → schemas (validate/serialize) → api/v1 (HTTP). Nothing skips a layer; in particular, provider adapters never touch the database, and routers never touch SQLAlchemy models directly.

Provider abstraction & ingestion pipeline (sections 13–15)

Provider.get_movie(external_id)   # raw dict, provider-shaped
    -> normalize_movie()          # -> NormalizedMovie, provider-agnostic
    -> match_entity()             # exact external-ID match, or NEW
    -> ingest_movie()             # upsert Movie + satellite tables
    -> upsert_provider_record()   # provenance: raw payload + match confidence

MetadataProvider (app/providers/base.py) is the only thing that knows TMDb-specific request shapes. A second provider (IMDb dataset export, Wikidata, ...) means writing a new adapter that returns the same raw shape TMDb's endpoints return, or a new normalize_* sibling if its shape differs — nothing in app/services/ingestion/ or downstream needs to change.

Matching policy is deliberately conservative (section 14): only an exact external-ID match reuses an existing Aurora entity. Everything else creates a new entity. Aurora never guesses via fuzzy title matching that two providers' records are "the same movie" — that risk (wrongly merging two different films) is worse than the cost of a duplicate until a human- reviewable matching workflow exists (tracked in docs/database.md#known-limitations).

Ingestion always runs through workers, never inline in an HTTP request (section 15). The admin API (POST /api/v1/admin/ingest/movie) only enqueues an arq job and returns immediately; app/workers/tasks.py picks it up and calls app/services/ingestion/runner.py, which tracks the job's lifecycle in ingestion_jobs (idempotent via a unique idempotency_key — re-triggering ingestion for the same provider + external ID reuses the same job record rather than double-processing).

Failures actually get retried, not just labeled retryable: arq only auto-retries a job on its own arq.Retry exception (or CancelledError) — never on an arbitrary exception a task raises — so _run_ingestion catches the failure, records it, and if ingestion_jobs.attempts hasn't hit max_attempts yet, raises Retry(defer=...) (capped exponential backoff) so arq actually puts the job back on the queue instead of the DB saying "retrying" while nothing retries it. Series ingestion cascading into concurrent season jobs (include_seasons) is exactly what surfaces transient failures worth retrying this way: two seasons sharing a guest star can race to create the same Person row (see app/repositories/reference_repository.py:_create_or_recover, which recovers from that specific race directly; the arq retry here is the backstop for whatever it doesn't).

Artwork self-hosting (section 10)

ingest_movie/ingest_series/ingest_person succeeds
    -> tasks.py: get_primary_artwork_for_download(entity_type, [entity_id, ...])
    -> one download_artwork_task enqueued per primary artwork id
    -> download_artwork(): fetch original_url, verify it's an image, cap
       size, sha256 it, storage.save(), set Artwork.local_path + checksum

Every image API responses reference is stored with its provider CDN URL (original_url) from the moment it's ingested; local_path starts empty and is filled in by this pipeline running asynchronously afterward, so responses transparently point at TMDb until a local copy exists, then switch over. Only artwork marked is_primary (the one image TMDb itself designates for an entity, not the dozens of alternates also ingested) gets downloaded automatically — see docs/database.md#artwork-storage-self-hosting.

Storage is behind an ArtworkStorage interface (app/services/artwork/storage.py) with one implementation, LocalDiskStorage — a shared Docker volume between the api and worker containers (docker-compose.yml's aurora_media_data), which is enough for this project's actual deployment shape (one Docker host, principle 9: avoid premature complexity) but assumes exactly that. A deployment that outgrew a single host would implement the same interface against S3/R2 instead, without the download pipeline or callers changing.

This is also why app/repositories/polymorphic.py:sync_artwork upserts artwork by provider_asset_id instead of the delete-then-recreate pattern every other satellite table uses (movie/series/person credits, videos, genre/keyword/company links): a delete-then-recreate would generate a fresh Artwork.id with local_path=None on every re-ingestion, silently discarding every downloaded copy and forcing a full re-download each sync.

Search (section 16)

app/services/search/base.py:SearchProvider is the abstraction; API handlers and SearchService only depend on it, never on SQL directly. PostgreSQLSearchProvider blends ts_rank_cd (full-text) with pg_trgm similarity() (typo tolerance) across primary titles/names and alternate titles. Swapping in OpenSearchProvider later means implementing the same SearchProvider.search() contract — no caller changes.

Caching (section 21)

app/services/cache.py wraps Redis with fail-open semantics: every read/ write function catches Redis errors and behaves as a cache miss / no-op rather than raising, so Redis being down degrades performance, not correctness. app/api/deps.py:cached_response() is the read-through helper used by the movie/series/person detail endpoints (the read-heaviest, per section 21). Cache keys are versioned (aurora:v1:...) so a response-shape change (bumping CACHE_VERSION) doesn't require a manual flush.

Cache invalidation on write: ingestion is the only write path that touches cached reads (there's no separate write API in 0.1), and it invalidates — app/services/ingestion/runner.py:_run_ingestion takes a cache_entity_type ("movie"/"series"/"person") and, on success, deletes that entity's detail cache key so the next GET reflects the just-completed ingestion instead of serving a pre-ingestion snapshot for up to CACHE_DEFAULT_TTL. This was found live: re-ingesting a movie to pick up a new company-logo field kept returning the old response because nothing invalidated the cache. Seasons and episodes aren't cached, so run_season_ingestion passes no cache_entity_type.

Security (section 22)

• CORS restricted to CORS_ORIGINS (comma-separated env var), not *.
• Every response carries X-Content-Type-Options, X-Frame-Options, Referrer-Policy (see app/main.py middleware).
• Request bodies over 2 MiB are rejected before parsing.
• Redis-backed fixed-window rate limiting (app/core/security.py), fails open if Redis is unreachable (same reasoning as caching).
• Admin/provider endpoints (/api/v1/admin/*) require X-API-Key matching AURORA_ADMIN_API_KEYS; public read endpoints require no auth but are rate-limited.
• No provider credentials are hard-coded; TMDB_API_KEY is read from the environment and the TMDb adapter raises a clear ProviderNotConfiguredError (not a fake success) if it's missing.
• Secrets never committed — .env.example holds placeholders only.

Not yet implemented (tracked): encryption-at-rest for provider credentials beyond "not committed to Git" (there's currently exactly one credential, TMDB_API_KEY, sourced from the environment — a secrets manager integration is the natural next step once there are more); audit logging for admin actions beyond structured request logs.

Observability (section 23)

Structured logging via structlog, JSON in non-development environments. Every request gets a correlation ID (X-Request-ID, generated if the client doesn't send one) bound into the log context for that request's lifetime (app/core/logging.py, app/main.py:request_context_middleware). GET /metrics exposes Prometheus counters/histograms (HTTP request count, latency by route) — compatible with Prometheus/Grafana; nothing in local development requires the observability stack to be running.

Health checks (section 24)

GET /health — liveness only (process is up). GET /ready — checks PostgreSQL connectivity (always required) and Redis (required only if REDIS_REQUIRED=true, since Redis is disposable — see docs/database.md). Both are mounted twice: unprefixed (/health, /ready, for Docker/Traefik health checks) and under /api/v1 (per section 17's endpoint list).

Statelessness & horizontal scaling (section 1/2)

The API holds no in-process state across requests — no server-side sessions, no in-memory caches keyed by request. Every dependency (get_db, get_redis, get_queue) is request-scoped and backed by a shared pool. This is what makes docker-compose.prod.yml's api: replicas: 3 behind Traefik correct without sticky sessions. Workers (docker-compose.prod.yml's worker: replicas: 2) scale independently of the API — arq's Redis-backed queue ensures each job is claimed by exactly one worker.