Aurora-Core/README.md

Aurora Core

Aurora Core is Aurora Dragon Studio's production-grade media metadata, discovery, and intelligence backend — a TMDb/IMDb-style canonical data layer for movies, TV, people, and everything that connects them. It is not a Plex application: Plex, Overseerr, Radarr, Sonarr, and future Aurora software are consumers built around Core, never dependencies of it.

See docs/aurora-core-spec.md for the full build specification this repository implements, and docs/architecture.md for how it's put together.

Architecture at a glance

A modular monolith (not microservices): a stateless FastAPI API behind a reverse proxy, PostgreSQL as the sole source of truth, Redis for disposable caching/rate-limiting/queueing, and arq workers for async metadata ingestion. Full diagram and reasoning in docs/architecture.md.

• API: FastAPI + Pydantic v2, stateless, horizontally scalable
• Database: PostgreSQL via SQLAlchemy 2.0 (async) + Alembic migrations
• Aurora IDs: UUIDv7 primary keys, never a provider's ID — see docs/database.md
• Cache/queue: Redis + arq
• Search: PostgreSQL full-text + trigram, behind a swappable SearchProvider abstraction
• Providers: TMDb today (app/providers/tmdb.py), pluggable

Prerequisites

• Docker + Docker Compose (recommended path)
• or, for running outside Docker: Python 3.12, a local PostgreSQL 17 with pg_trgm available, and Redis 7

Local setup (Docker — recommended)

cp .env.example .env
# edit .env: set AURORA_SECRET_KEY, AURORA_ADMIN_API_KEYS, and (optionally)
# TMDB_API_KEY — see https://www.themoviedb.org/settings/api

docker compose up --build

This starts PostgreSQL, Redis, runs alembic upgrade head (the migrate service), then the API (http://localhost:8000) and a worker. API docs: http://localhost:8000/docs.

Local setup (without Docker)

python -m venv .venv
.venv/Scripts/activate        # Windows; use `source .venv/bin/activate` on macOS/Linux
pip install -e ".[dev]"

cp .env.example .env           # point DATABASE_URL / DATABASE_URL_SYNC / REDIS_URL at your own instances

alembic upgrade head
uvicorn app.main:app --reload

Run a worker in a second terminal:

arq app.workers.worker.WorkerSettings

Environment configuration

All configuration is environment-based (app/core/config.py); see .env.example for every variable and its meaning. Never commit a real .env — only .env.example (placeholders) belongs in Git.

Database migrations

alembic upgrade head                            # apply all migrations
alembic revision --autogenerate -m "add xyz"     # generate the next one after changing models
alembic downgrade -1                             # roll back one revision

The application never creates tables at startup — see docs/database.md.

Ingesting data

Ingestion always runs asynchronously through a worker, never inline in a request (see docs/architecture.md). With TMDB_API_KEY set and a worker running:

curl -X POST http://localhost:8000/api/v1/admin/ingest/movie \
  -H "X-API-Key: $AURORA_ADMIN_API_KEYS" \
  -H "Content-Type: application/json" \
  -d '{"tmdb_id": "603"}'
# -> {"job_id": "...", "arq_job_id": "...", "status": "enqueued"}

curl http://localhost:8000/api/v1/admin/ingest/jobs/<job_id> \
  -H "X-API-Key: $AURORA_ADMIN_API_KEYS"
# -> poll until "status": "succeeded"

curl "http://localhost:8000/api/v1/search?q=the%20matrix"

Ingesting a series also queues its posters/backdrops/profile images for local download automatically (only the provider's primary image per entity, not every alternate) — see the next section.

Bulk-importing TMDb's entire catalog

app/scripts/bulk_import.py enqueues an ingestion job for every id in TMDb's daily ID export files (published once a day specifically for full-catalog sync, unlike /discover, which caps at 10,000 results per query) — the same one-id-at-a-time pipeline the admin endpoints above use, just enqueued for every id TMDb has rather than one at a time by hand:

docker compose -f docker-compose.prod.yml exec worker \
  python -m app.scripts.bulk_import movie series person collection

As of 2026-09, TMDb's real catalog is roughly 1.24M movies, 231K TV series, 4.9M people, and 9.8K collections — ~6.4M entities total. At this project's default worker concurrency (WORKER_CONCURRENCY=4 × 2 worker replicas), expect this to run for on the order of weeks, not hours — it does not rate-limit itself against TMDb; TMDb's request rate is governed entirely by how many workers are consuming the queue concurrently (raise WORKER_CONCURRENCY / scale up workers to go faster, mindful of TMDb's own rate limits — app/providers/tmdb.py backs off on a 429 using TMDb's own Retry-After hint, but sustained high concurrency will still see more of them). Run it detached (docker compose ... exec -d worker ..., or under nohup/tmux/screen) so it survives a disconnect, and use --dry-run first to see counts without enqueueing anything. Re-running (e.g. after an interruption) is safe — every id is idempotency-keyed, so already-created jobs are reused, never duplicated.

Expect substantial disk usage from artwork self-hosting cascading on every successful ingestion (see the next section) at this scale — budget accordingly before running it against the full catalog.

Artwork

Images are referenced by their provider CDN URL (TMDb) from the moment they're ingested, and cached locally in the background afterward — API responses point at the provider until a local copy exists, then switch over automatically, with no client-visible change. Cached files live under MEDIA_ROOT (a Docker volume shared between api and worker, see docker-compose.yml) and are served at MEDIA_URL_PREFIX (default /media). See docs/architecture.md and docs/database.md for how it works and its current single-host-deployment assumption.

Tests

pytest tests/unit -v                 # no external dependencies

# Integration/API tests need a real Postgres + Redis:
docker compose up -d postgres redis
createdb -h localhost -U aurora aurora_core_test   # or: docker exec -it aurora-core-postgres-1 createdb -U aurora aurora_core_test
export TEST_DATABASE_URL=postgresql+asyncpg://aurora:aurora@localhost:5432/aurora_core_test
export TEST_REDIS_URL=redis://localhost:6379/15
pytest tests/integration -v

Lint and type-check:

ruff check app migrations tests
mypy app

Load testing (targets: 500 concurrent users, search p95 < 500ms, other endpoints p95 < 250ms, 5xx rate < 0.1% — see docs/aurora-core-spec.md section 2):

pip install -e ".[load]"
locust -f tests/load/locustfile.py --host http://localhost:8000
# open http://localhost:8089, seed some data first (see tests/load/locustfile.py)

API documentation

• Interactive (OpenAPI/Swagger): /docs, controlled by AURORA_API_DOCS_ENABLED (set false in production if you don't want it publicly exposed)
• Reference: docs/api.md
• Schema/ERD: docs/database.md, docs/erd.md

Worker operation

Workers (arq, app/workers/worker.py) scale independently of the API — docker-compose.prod.yml runs 2 worker replicas and 3 API replicas as an example. Concurrency per worker process is controlled by WORKER_CONCURRENCY. Jobs are idempotent (keyed by tmdb:<entity_type>:<external_id>), retryable (max_tries in app/workers/worker.py), and observable via GET /api/v1/admin/ingest/jobs/{job_id}.

Production deployment

The production host runs cPanel, which already owns ports 80/443 for other sites on the box — there's no Traefik here. docker-compose.prod.yml binds the API container to 127.0.0.1:8000 only; the host's own webserver (Apache or LiteSpeed, via cPanel) reverse-proxies api.amdbs.net to it and terminates TLS itself (cPanel's AutoSSL can issue the Let's Encrypt cert once the subdomain exists and DNS resolves).

1. DNS (Cloudflare, where amdbs.net is managed): add an A record for api.amdbs.net pointing at the server's IP.

2. cPanel: create the api.amdbs.net subdomain so AutoSSL picks it up and issues a cert; then reverse-proxy that vhost to 127.0.0.1:8000 — either cPanel/WHM's Application Manager if available, or a custom Apache/LiteSpeed vhost include (ProxyPass/ProxyPassReverse for Apache; LiteSpeed's External App + rewrite rule for LiteSpeed).

3. .env on the server (never committed) — fill in every value in .env.example, including the "Production only" section at the bottom (POSTGRES_USER/PASSWORD/DB), plus:

   CORS_ORIGINS=https://www.amdbs.net  # must include the frontend's real origin
   

4. Deploy:

   docker compose -f docker-compose.prod.yml up --build -d
   

PgBouncer sits in front of PostgreSQL (transaction pooling — DATABASE_BEHIND_TRANSACTION_POOLER=true disables asyncpg's server-side prepared statements, which don't survive transaction-mode pooling). See docs/architecture.md.

Aurora Core's production domain split is api.amdbs.net (this backend) / www.amdbs.net (frontend, a separate project).

Known limitations

See docs/database.md#known-limitations and docs/api.md#known-limitations for the current, tracked list (no curated language/country seed data, no fuzzy entity matching, no field-level provenance, list-shaped responses beyond search and browse/discover return everything with no pagination). None of these block the Core 0.1 milestone (section 34): search → movie/series/person detail works end-to-end against real TMDb data, verified on a real PostgreSQL + Redis + Docker deployment (not just this repo's own dev machine).