Aurora-Core/README.md

# Aurora Core

Aurora Core is Aurora Dragon Studio's production-grade media metadata,
discovery, and intelligence backend — a TMDb/IMDb-style canonical data layer
for movies, TV, people, and everything that connects them. It is **not** a
Plex application: Plex, Overseerr, Radarr, Sonarr, and future Aurora
software are consumers built around Core, never dependencies of it.

See [`docs/aurora-core-spec.md`](docs/aurora-core-spec.md) for the full
build specification this repository implements, and
[`docs/architecture.md`](docs/architecture.md) for how it's put together.

## Architecture at a glance

A modular monolith (not microservices): a stateless FastAPI API behind a
reverse proxy, PostgreSQL as the sole source of truth, Redis for disposable
caching/rate-limiting/queueing, and arq workers for async metadata
ingestion. Full diagram and reasoning in
[`docs/architecture.md`](docs/architecture.md).

- **API**: FastAPI + Pydantic v2, stateless, horizontally scalable
- **Database**: PostgreSQL via SQLAlchemy 2.0 (async) + Alembic migrations
- **Aurora IDs**: UUIDv7 primary keys, never a provider's ID — see
  [`docs/database.md`](docs/database.md)
- **Cache/queue**: Redis + arq
- **Search**: PostgreSQL full-text + trigram, behind a swappable
  `SearchProvider` abstraction
- **Providers**: TMDb today (`app/providers/tmdb.py`), pluggable

## Prerequisites

- Docker + Docker Compose (recommended path)
- *or*, for running outside Docker: Python 3.12, a local PostgreSQL 17 with
  `pg_trgm` available, and Redis 7

## Local setup (Docker — recommended)

```bash
cp .env.example .env
# edit .env: set AURORA_SECRET_KEY, AURORA_ADMIN_API_KEYS, and (optionally)
# TMDB_API_KEY — see https://www.themoviedb.org/settings/api

docker compose up --build
```

This starts PostgreSQL, Redis, runs `alembic upgrade head` (the `migrate`
service), then the API (`http://localhost:8000`) and a worker. API docs:
`http://localhost:8000/docs`.

## Local setup (without Docker)

```bash
python -m venv .venv
.venv/Scripts/activate        # Windows; use `source .venv/bin/activate` on macOS/Linux
pip install -e ".[dev]"

cp .env.example .env           # point DATABASE_URL / DATABASE_URL_SYNC / REDIS_URL at your own instances

alembic upgrade head
uvicorn app.main:app --reload
```

Run a worker in a second terminal:

```bash
arq app.workers.worker.WorkerSettings
```

## Environment configuration

All configuration is environment-based (`app/core/config.py`); see
[`.env.example`](.env.example) for every variable and its meaning. Never
commit a real `.env` — only `.env.example` (placeholders) belongs in Git.

## Database migrations

```bash
alembic upgrade head                            # apply all migrations
alembic revision --autogenerate -m "add xyz"     # generate the next one after changing models
alembic downgrade -1                             # roll back one revision
```

The application never creates tables at startup — see
[`docs/database.md`](docs/database.md#migrations).

## Ingesting data

Ingestion always runs asynchronously through a worker, never inline in a
request (see [`docs/architecture.md`](docs/architecture.md)). With
`TMDB_API_KEY` set and a worker running:

```bash
curl -X POST http://localhost:8000/api/v1/admin/ingest/movie \
  -H "X-API-Key: $AURORA_ADMIN_API_KEYS" \
  -H "Content-Type: application/json" \
  -d '{"tmdb_id": "603"}'
# -> {"job_id": "...", "arq_job_id": "...", "status": "enqueued"}

curl http://localhost:8000/api/v1/admin/ingest/jobs/<job_id> \
  -H "X-API-Key: $AURORA_ADMIN_API_KEYS"
# -> poll until "status": "succeeded"

curl "http://localhost:8000/api/v1/search?q=the%20matrix"
```

Ingesting a series also queues its posters/backdrops/profile images for
local download automatically (only the provider's *primary* image per
entity, not every alternate) — see the next section.

### Bulk-importing TMDb's entire catalog

`app/scripts/bulk_import.py` enqueues an ingestion job for every id in
TMDb's [daily ID export files](https://developer.themoviedb.org/docs/daily-id-exports)
(published once a day specifically for full-catalog sync, unlike
`/discover`, which caps at 10,000 results per query) — the same
one-id-at-a-time pipeline the admin endpoints above use, just enqueued for
every id TMDb has rather than one at a time by hand:

```bash
docker compose -f docker-compose.prod.yml exec worker \
  python -m app.scripts.bulk_import movie series person collection
```

As of 2026-09, TMDb's real catalog is roughly 1.24M movies, 231K TV
series, 4.9M people, and 9.8K collections — **~6.4M entities total**. At
this project's default worker concurrency (`WORKER_CONCURRENCY=4` × 2
worker replicas), expect this to run for **on the order of weeks**, not
hours — it does not rate-limit itself against TMDb; TMDb's request rate is
governed entirely by how many workers are consuming the queue
concurrently (raise `WORKER_CONCURRENCY` / scale up workers to go faster,
mindful of TMDb's own rate limits — `app/providers/tmdb.py` backs off on a
429 using TMDb's own `Retry-After` hint, but sustained high concurrency
will still see more of them). Run it detached
(`docker compose ... exec -d worker ...`, or under `nohup`/`tmux`/`screen`)
so it survives a disconnect, and use `--dry-run` first to see counts
without enqueueing anything. Re-running (e.g. after an interruption) is
safe — every id is idempotency-keyed, so already-created jobs are reused,
never duplicated.

Expect substantial disk usage from artwork self-hosting cascading on
every successful ingestion (see the next section) at this scale — budget
accordingly before running it against the full catalog.

## Artwork

Images are referenced by their provider CDN URL (TMDb) from the moment
they're ingested, and cached locally in the background afterward — API
responses point at the provider until a local copy exists, then switch over
automatically, with no client-visible change. Cached files live under
`MEDIA_ROOT` (a Docker volume shared between `api` and `worker`, see
`docker-compose.yml`) and are served at `MEDIA_URL_PREFIX` (default
`/media`). See [`docs/architecture.md`](docs/architecture.md#artwork-self-hosting-section-10)
and [`docs/database.md`](docs/database.md#artwork-storage-self-hosting) for
how it works and its current single-host-deployment assumption.

## Tests

```bash
pytest tests/unit -v                 # no external dependencies

# Integration/API tests need a real Postgres + Redis:
docker compose up -d postgres redis
createdb -h localhost -U aurora aurora_core_test   # or: docker exec -it aurora-core-postgres-1 createdb -U aurora aurora_core_test
export TEST_DATABASE_URL=postgresql+asyncpg://aurora:aurora@localhost:5432/aurora_core_test
export TEST_REDIS_URL=redis://localhost:6379/15
pytest tests/integration -v
```

Lint and type-check:

```bash
ruff check app migrations tests
mypy app
```

Load testing (targets: 500 concurrent users, search p95 < 500ms, other
endpoints p95 < 250ms, 5xx rate < 0.1% — see
[`docs/aurora-core-spec.md`](docs/aurora-core-spec.md) section 2):

```bash
pip install -e ".[load]"
locust -f tests/load/locustfile.py --host http://localhost:8000
# open http://localhost:8089, seed some data first (see tests/load/locustfile.py)
```

## API documentation

- Interactive (OpenAPI/Swagger): `/docs`, controlled by
  `AURORA_API_DOCS_ENABLED` (set `false` in production if you don't want it
  publicly exposed)
- Reference: [`docs/api.md`](docs/api.md)
- Schema/ERD: [`docs/database.md`](docs/database.md),
  [`docs/erd.md`](docs/erd.md)

## Worker operation

Workers (arq, `app/workers/worker.py`) scale independently of the API —
`docker-compose.prod.yml` runs 2 worker replicas and 3 API replicas as an
example. Concurrency per worker process is controlled by
`WORKER_CONCURRENCY`. Jobs are idempotent (keyed by
`tmdb:<entity_type>:<external_id>`), retryable (`max_tries` in
`app/workers/worker.py`), and observable via
`GET /api/v1/admin/ingest/jobs/{job_id}`.

## Production deployment

The production host runs cPanel, which already owns ports 80/443 for other
sites on the box — there's no Traefik here. `docker-compose.prod.yml` binds
the API container to `127.0.0.1:8000` only; the host's own webserver
(Apache or LiteSpeed, via cPanel) reverse-proxies `api.amdbs.net` to it and
terminates TLS itself (cPanel's AutoSSL can issue the Let's Encrypt cert
once the subdomain exists and DNS resolves).

1. **DNS** (Cloudflare, where `amdbs.net` is managed): add an A record for
   `api.amdbs.net` pointing at the server's IP.
2. **cPanel**: create the `api.amdbs.net` subdomain so AutoSSL picks it up
   and issues a cert; then reverse-proxy that vhost to `127.0.0.1:8000` —
   either cPanel/WHM's Application Manager if available, or a custom
   Apache/LiteSpeed vhost include (`ProxyPass`/`ProxyPassReverse` for
   Apache; LiteSpeed's External App + rewrite rule for LiteSpeed).
3. **`.env`** on the server (never committed) — fill in every value in
   `.env.example`, including the "Production only" section at the bottom
   (`POSTGRES_USER`/`PASSWORD`/`DB`), plus:

   ```bash
   CORS_ORIGINS=https://www.amdbs.net  # must include the frontend's real origin
   ```

4. **Deploy**:

   ```bash
   docker compose -f docker-compose.prod.yml up --build -d
   ```

PgBouncer sits in front of PostgreSQL (transaction pooling —
`DATABASE_BEHIND_TRANSACTION_POOLER=true` disables asyncpg's server-side
prepared statements, which don't survive transaction-mode pooling). See
[`docs/architecture.md`](docs/architecture.md).

Aurora Core's production domain split is `api.amdbs.net` (this backend) /
`www.amdbs.net` (frontend, a separate project).

## Known limitations

See [`docs/database.md#known-limitations`](docs/database.md#known-limitations)
and [`docs/api.md#known-limitations`](docs/api.md#known-limitations) for the
current, tracked list (no curated language/country seed data, no fuzzy
entity matching, no field-level provenance, list-shaped responses beyond
search and browse/discover return everything with no pagination). None of
these block the Core 0.1 milestone (section 34): search → movie/series/person
detail works end-to-end against real TMDb data, verified on a real
PostgreSQL + Redis +
Docker deployment (not just this repo's own dev machine).