Aurora-Core/README.md

# Aurora Core

Aurora Core is Aurora Dragon Studio's production-grade media metadata,
discovery, and intelligence backend — a TMDb/IMDb-style canonical data layer
for movies, TV, people, and everything that connects them. It is **not** a
Plex application: Plex, Overseerr, Radarr, Sonarr, and future Aurora
software are consumers built around Core, never dependencies of it.

See [`docs/aurora-core-spec.md`](docs/aurora-core-spec.md) for the full
build specification this repository implements, and
[`docs/architecture.md`](docs/architecture.md) for how it's put together.

## Architecture at a glance

A modular monolith (not microservices): a stateless FastAPI API behind
Traefik, PostgreSQL as the sole source of truth, Redis for disposable
caching/rate-limiting/queueing, and arq workers for async metadata
ingestion. Full diagram and reasoning in
[`docs/architecture.md`](docs/architecture.md).

- **API**: FastAPI + Pydantic v2, stateless, horizontally scalable
- **Database**: PostgreSQL via SQLAlchemy 2.0 (async) + Alembic migrations
- **Aurora IDs**: UUIDv7 primary keys, never a provider's ID — see
  [`docs/database.md`](docs/database.md)
- **Cache/queue**: Redis + arq
- **Search**: PostgreSQL full-text + trigram, behind a swappable
  `SearchProvider` abstraction
- **Providers**: TMDb today (`app/providers/tmdb.py`), pluggable

## Prerequisites

- Docker + Docker Compose (recommended path)
- *or*, for running outside Docker: Python 3.12, a local PostgreSQL 17 with
  `pg_trgm` available, and Redis 7

## Local setup (Docker — recommended)

```bash
cp .env.example .env
# edit .env: set AURORA_SECRET_KEY, AURORA_ADMIN_API_KEYS, and (optionally)
# TMDB_API_KEY — see https://www.themoviedb.org/settings/api

docker compose up --build
```

This starts PostgreSQL, Redis, runs `alembic upgrade head` (the `migrate`
service), then the API (`http://localhost:8000`) and a worker. API docs:
`http://localhost:8000/docs`.

## Local setup (without Docker)

```bash
python -m venv .venv
.venv/Scripts/activate        # Windows; use `source .venv/bin/activate` on macOS/Linux
pip install -e ".[dev]"

cp .env.example .env           # point DATABASE_URL / DATABASE_URL_SYNC / REDIS_URL at your own instances

alembic upgrade head
uvicorn app.main:app --reload
```

Run a worker in a second terminal:

```bash
arq app.workers.worker.WorkerSettings
```

## Environment configuration

All configuration is environment-based (`app/core/config.py`); see
[`.env.example`](.env.example) for every variable and its meaning. Never
commit a real `.env` — only `.env.example` (placeholders) belongs in Git.

## Database migrations

```bash
alembic upgrade head                            # apply all migrations
alembic revision --autogenerate -m "add xyz"     # generate the next one after changing models
alembic downgrade -1                             # roll back one revision
```

The application never creates tables at startup — see
[`docs/database.md`](docs/database.md#migrations).

## Ingesting data

Ingestion always runs asynchronously through a worker, never inline in a
request (see [`docs/architecture.md`](docs/architecture.md)). With
`TMDB_API_KEY` set and a worker running:

```bash
curl -X POST http://localhost:8000/api/v1/admin/ingest/movie \
  -H "X-API-Key: $AURORA_ADMIN_API_KEYS" \
  -H "Content-Type: application/json" \
  -d '{"tmdb_id": "603"}'
# -> {"job_id": "...", "arq_job_id": "...", "status": "enqueued"}

curl http://localhost:8000/api/v1/admin/ingest/jobs/<job_id> \
  -H "X-API-Key: $AURORA_ADMIN_API_KEYS"
# -> poll until "status": "succeeded"

curl "http://localhost:8000/api/v1/search?q=the%20matrix"
```

## Tests

```bash
pytest tests/unit -v                 # no external dependencies

# Integration/API tests need a real Postgres + Redis:
docker compose up -d postgres redis
createdb -h localhost -U aurora aurora_core_test   # or: docker exec -it aurora-core-postgres-1 createdb -U aurora aurora_core_test
export TEST_DATABASE_URL=postgresql+asyncpg://aurora:aurora@localhost:5432/aurora_core_test
export TEST_REDIS_URL=redis://localhost:6379/15
pytest tests/integration -v
```

Lint and type-check:

```bash
ruff check app migrations tests
mypy app
```

Load testing (targets: 500 concurrent users, search p95 < 500ms, other
endpoints p95 < 250ms, 5xx rate < 0.1% — see
[`docs/aurora-core-spec.md`](docs/aurora-core-spec.md) section 2):

```bash
pip install -e ".[load]"
locust -f tests/load/locustfile.py --host http://localhost:8000
# open http://localhost:8089, seed some data first (see tests/load/locustfile.py)
```

## API documentation

- Interactive (OpenAPI/Swagger): `/docs`, controlled by
  `AURORA_API_DOCS_ENABLED` (set `false` in production if you don't want it
  publicly exposed)
- Reference: [`docs/api.md`](docs/api.md)
- Schema/ERD: [`docs/database.md`](docs/database.md),
  [`docs/erd.md`](docs/erd.md)

## Worker operation

Workers (arq, `app/workers/worker.py`) scale independently of the API —
`docker-compose.prod.yml` runs 2 worker replicas and 3 API replicas as an
example. Concurrency per worker process is controlled by
`WORKER_CONCURRENCY`. Jobs are idempotent (keyed by
`tmdb:<entity_type>:<external_id>`), retryable (`max_tries` in
`app/workers/worker.py`), and observable via
`GET /api/v1/admin/ingest/jobs/{job_id}`.

## Production deployment

```bash
docker compose -f docker-compose.prod.yml up --build -d
```

Traefik-ready (labels on the `api` service), PgBouncer in front of
PostgreSQL (transaction pooling — `DATABASE_BEHIND_TRANSACTION_POOLER=true`
disables asyncpg's server-side prepared statements, which don't survive
transaction-mode pooling), horizontally-scaled API and worker replicas. See
[`docs/architecture.md`](docs/architecture.md).

## Known limitations

See [`docs/database.md#known-limitations`](docs/database.md#known-limitations)
and [`docs/api.md#known-limitations`](docs/api.md#known-limitations) for the
current, tracked list (no curated language/country seed data, no fuzzy
entity matching, no field-level provenance, no pagination beyond search,
person credits limited to movies). None of these block the Core 0.1
milestone (section 34): search → movie/person detail works end-to-end.

This repository was built in an environment without Docker or a
pre-existing PostgreSQL instance; the initial Alembic migration was
hand-written from the models (not autogenerated against a live database)
and validated with `alembic history`, but has not yet been executed against
a running PostgreSQL instance. **Run `docker compose up --build` and confirm
`alembic upgrade head` succeeds against a genuinely empty database before
treating the schema as final** — see
[`docs/database.md#migrations`](docs/database.md#migrations).