Aurora-Core/README.md

Aurora Core

Aurora Core is Aurora Dragon Studio's production-grade media metadata, discovery, and intelligence backend — a TMDb/IMDb-style canonical data layer for movies, TV, people, and everything that connects them. It is not a Plex application: Plex, Overseerr, Radarr, Sonarr, and future Aurora software are consumers built around Core, never dependencies of it.

See docs/aurora-core-spec.md for the full build specification this repository implements, and docs/architecture.md for how it's put together.

Architecture at a glance

A modular monolith (not microservices): a stateless FastAPI API behind Traefik, PostgreSQL as the sole source of truth, Redis for disposable caching/rate-limiting/queueing, and arq workers for async metadata ingestion. Full diagram and reasoning in docs/architecture.md.

• API: FastAPI + Pydantic v2, stateless, horizontally scalable
• Database: PostgreSQL via SQLAlchemy 2.0 (async) + Alembic migrations
• Aurora IDs: UUIDv7 primary keys, never a provider's ID — see docs/database.md
• Cache/queue: Redis + arq
• Search: PostgreSQL full-text + trigram, behind a swappable SearchProvider abstraction
• Providers: TMDb today (app/providers/tmdb.py), pluggable

Prerequisites

• Docker + Docker Compose (recommended path)
• or, for running outside Docker: Python 3.12, a local PostgreSQL 17 with pg_trgm available, and Redis 7

Local setup (Docker — recommended)

cp .env.example .env
# edit .env: set AURORA_SECRET_KEY, AURORA_ADMIN_API_KEYS, and (optionally)
# TMDB_API_KEY — see https://www.themoviedb.org/settings/api

docker compose up --build

This starts PostgreSQL, Redis, runs alembic upgrade head (the migrate service), then the API (http://localhost:8000) and a worker. API docs: http://localhost:8000/docs.

Local setup (without Docker)

python -m venv .venv
.venv/Scripts/activate        # Windows; use `source .venv/bin/activate` on macOS/Linux
pip install -e ".[dev]"

cp .env.example .env           # point DATABASE_URL / DATABASE_URL_SYNC / REDIS_URL at your own instances

alembic upgrade head
uvicorn app.main:app --reload

Run a worker in a second terminal:

arq app.workers.worker.WorkerSettings

Environment configuration

All configuration is environment-based (app/core/config.py); see .env.example for every variable and its meaning. Never commit a real .env — only .env.example (placeholders) belongs in Git.

Database migrations

alembic upgrade head                            # apply all migrations
alembic revision --autogenerate -m "add xyz"     # generate the next one after changing models
alembic downgrade -1                             # roll back one revision

The application never creates tables at startup — see docs/database.md.

Ingesting data

Ingestion always runs asynchronously through a worker, never inline in a request (see docs/architecture.md). With TMDB_API_KEY set and a worker running:

curl -X POST http://localhost:8000/api/v1/admin/ingest/movie \
  -H "X-API-Key: $AURORA_ADMIN_API_KEYS" \
  -H "Content-Type: application/json" \
  -d '{"tmdb_id": "603"}'
# -> {"job_id": "...", "arq_job_id": "...", "status": "enqueued"}

curl http://localhost:8000/api/v1/admin/ingest/jobs/<job_id> \
  -H "X-API-Key: $AURORA_ADMIN_API_KEYS"
# -> poll until "status": "succeeded"

curl "http://localhost:8000/api/v1/search?q=the%20matrix"

Ingesting a series also queues its posters/backdrops/profile images for local download automatically (only the provider's primary image per entity, not every alternate) — see the next section.

Artwork

Images are referenced by their provider CDN URL (TMDb) from the moment they're ingested, and cached locally in the background afterward — API responses point at the provider until a local copy exists, then switch over automatically, with no client-visible change. Cached files live under MEDIA_ROOT (a Docker volume shared between api and worker, see docker-compose.yml) and are served at MEDIA_URL_PREFIX (default /media). See docs/architecture.md and docs/database.md for how it works and its current single-host-deployment assumption.

Tests

pytest tests/unit -v                 # no external dependencies

# Integration/API tests need a real Postgres + Redis:
docker compose up -d postgres redis
createdb -h localhost -U aurora aurora_core_test   # or: docker exec -it aurora-core-postgres-1 createdb -U aurora aurora_core_test
export TEST_DATABASE_URL=postgresql+asyncpg://aurora:aurora@localhost:5432/aurora_core_test
export TEST_REDIS_URL=redis://localhost:6379/15
pytest tests/integration -v

Lint and type-check:

ruff check app migrations tests
mypy app

Load testing (targets: 500 concurrent users, search p95 < 500ms, other endpoints p95 < 250ms, 5xx rate < 0.1% — see docs/aurora-core-spec.md section 2):

pip install -e ".[load]"
locust -f tests/load/locustfile.py --host http://localhost:8000
# open http://localhost:8089, seed some data first (see tests/load/locustfile.py)

API documentation

• Interactive (OpenAPI/Swagger): /docs, controlled by AURORA_API_DOCS_ENABLED (set false in production if you don't want it publicly exposed)
• Reference: docs/api.md
• Schema/ERD: docs/database.md, docs/erd.md

Worker operation

Workers (arq, app/workers/worker.py) scale independently of the API — docker-compose.prod.yml runs 2 worker replicas and 3 API replicas as an example. Concurrency per worker process is controlled by WORKER_CONCURRENCY. Jobs are idempotent (keyed by tmdb:<entity_type>:<external_id>), retryable (max_tries in app/workers/worker.py), and observable via GET /api/v1/admin/ingest/jobs/{job_id}.

Production deployment

docker compose -f docker-compose.prod.yml up --build -d

Traefik-ready (labels on the api service), PgBouncer in front of PostgreSQL (transaction pooling — DATABASE_BEHIND_TRANSACTION_POOLER=true disables asyncpg's server-side prepared statements, which don't survive transaction-mode pooling), horizontally-scaled API and worker replicas. See docs/architecture.md.

Known limitations

See docs/database.md#known-limitations and docs/api.md#known-limitations for the current, tracked list (no curated language/country seed data, no fuzzy entity matching, no field-level provenance, no pagination beyond search, person credits' episode-level guest appearances rolled up to a per-series count rather than broken out individually). None of these block the Core 0.1 milestone (section 34): search → movie/series/person detail works end-to-end against real TMDb data, verified on a real PostgreSQL + Redis + Docker deployment (not just this repo's own dev machine).