Aurora-Core/docs/aurora-core-spec.md

# Aurora Core — Codex Build Specification

## Project

**Name:** Aurora Core
**Company:** Aurora Dragon Studio
**Purpose:** Production-grade media metadata, discovery, and intelligence backend.

Aurora Core will become the canonical data layer for the Aurora media ecosystem.

It should function conceptually like our own TMDb/IMDb-style platform, providing normalized information about:

- Movies
- Television series
- Seasons
- Episodes
- People
- Cast and crew
- Companies
- Collections/franchises
- Genres
- Keywords
- Artwork
- Videos/trailers
- Release information
- Ratings/certifications
- External provider IDs
- Relationships between all of the above

Aurora Core is NOT a Plex application.

Plex, Overseerr, Radarr, Sonarr, Jellyfin, Emby, and future Aurora streaming software will be consumers or integrations built around Core.

Do NOT make Aurora Core dependent on Plex.

---

# 1. Primary Architecture

Build Aurora Core as a production-grade **modular monolith**.

Do NOT prematurely break the application into microservices.

Architecture:

Internet
    |
Traefik
    |
Aurora Core API replicas
    |
    +--- PostgreSQL
    |
    +--- PgBouncer
    |
    +--- Redis
    |
    +--- Background Workers
            |
            +--- Metadata ingestion
            +--- Artwork processing
            +--- Search indexing
            +--- Provider synchronization

Primary technology stack:

- Python
- FastAPI
- PostgreSQL
- SQLAlchemy 2.x
- Alembic
- Pydantic
- Redis
- PgBouncer
- Background job workers
- Docker
- Docker Compose
- Traefik-compatible deployment

The API must remain stateless so multiple replicas can run simultaneously.

Example:

Traefik
    |
    +--- aurora-core-api-1
    +--- aurora-core-api-2
    +--- aurora-core-api-3

Workers must also scale independently.

---

# 2. Production Requirements

This is NOT a prototype.

Design Aurora Core for production use from the beginning.

Initial engineering target:

- 500 concurrent interactive users
- Horizontal API scaling
- Horizontal worker scaling
- Stateless API containers
- Database connection pooling
- Redis caching
- Background processing
- Idempotent jobs
- Structured logging
- Health checks
- Readiness checks
- Metrics
- Rate limiting
- Graceful shutdown
- Proper database migrations
- Automated tests

Initial performance goals:

| Requirement | Target |
|---|---|
| Concurrent interactive users | 500 |
| Normal API p95 | < 250 ms |
| Search p95 | < 500 ms |
| Core 5xx rate | < 0.1% |
| API horizontal scaling | Required |
| Worker horizontal scaling | Required |
| Lost jobs | 0 |
| Duplicate ingestion records | 0 |

These targets must ultimately be verified using load testing.

---

# 3. Aurora-Owned IDs

Aurora Core MUST own its identifiers.

Do NOT use TMDb IDs, IMDb IDs, TVDb IDs, or other provider IDs as primary keys.

Example:

aur_movie_01K5A8XXXXXXXXXXXX

or another suitable sortable globally unique ID scheme such as UUIDv7.

Internal database primary keys may use UUIDv7.

External IDs must be separate records.

Example:

Aurora Movie:
    Aurora ID: aur_movie_xxxxx

External IDs:
    TMDb: 550
    IMDb: tt0137523
    TVDb: ...

This allows Aurora Core to survive provider changes.

External providers are DATA SOURCES.

They are not Aurora Core's identity system.

---

# 4. Core Media Model

## Movies

Movie records should support:

- Aurora ID
- title
- original title
- alternate titles
- overview
- tagline
- release dates
- runtime
- status
- original language
- spoken languages
- countries
- genres
- keywords
- certifications
- budget
- revenue
- popularity data
- cast
- crew
- directors
- writers
- producers
- production companies
- distributors
- collections/franchises
- posters
- backdrops
- logos
- artwork
- trailers
- videos
- external IDs
- recommendations
- similar titles
- source/provenance information
- created timestamp
- updated timestamp

---

# 5. Television Model

Support:

Series
    |
    +--- Seasons
            |
            +--- Episodes

Series should support:

- title
- original title
- alternate titles
- overview
- tagline
- first air date
- last air date
- status
- networks
- production companies
- creators
- cast
- crew
- genres
- keywords
- languages
- countries
- artwork
- videos
- external IDs

Season should support:

- season number
- title
- overview
- air date
- episode count
- artwork
- external IDs

Episode should support:

- episode number
- season number
- title
- overview
- air date
- runtime
- cast
- guest stars
- crew
- directors
- writers
- still images
- external IDs

---

# 6. People

People are first-class Aurora entities.

Person records should support:

- Aurora ID
- name
- aliases
- biography
- birthday
- death date
- birthplace
- gender where legitimately provided by metadata sources
- profile images
- external IDs
- known-for department
- acting credits
- directing credits
- writing credits
- production credits
- other crew credits

Credits should NOT be stored as duplicated text blobs.

Use normalized relationship tables.

---

# 7. Credits

Create a flexible credit system.

Examples:

Person
    |
    +--- Movie
    |      role = actor
    |      character = "..."
    |
    +--- Movie
    |      role = director
    |
    +--- Episode
           role = guest_star

Credits should support:

- person
- media entity
- department
- job
- character
- billing order
- episode count where appropriate
- source information

---

# 8. Companies

Support entities such as:

- production companies
- networks
- studios
- distributors

Fields should include:

- Aurora ID
- company name
- description
- country
- logo
- external IDs
- parent company where applicable

---

# 9. Collections / Franchises

Collections should be first-class entities.

Examples:

Alien
Star Wars
Fast & Furious
Marvel Cinematic Universe

Support:

Collection
    |
    +--- Movies
    +--- Series
    +--- Related collections

Do not assume every provider's concept of "collection" is sufficient for Aurora's future requirements.

---

# 10. Artwork System

Do NOT simply place poster_url columns on movie tables.

Artwork must be its own entity.

Artwork fields:

- Aurora artwork ID
- associated entity
- artwork type
- language
- width
- height
- aspect ratio
- provider
- provider asset ID
- original URL
- cached/local location where permitted
- checksum
- attribution
- licensing information
- source/provenance
- created timestamp

Artwork types:

- poster
- backdrop
- logo
- profile
- still
- banner
- thumbnail

Aurora must retain sufficient provider/licensing information to know where artwork originated.

Do not assume externally sourced artwork can automatically be redistributed.

---

# 11. Videos

Videos should be normalized entities.

Support:

- trailer
- teaser
- clip
- featurette
- behind-the-scenes
- interview
- other

Store:

- provider
- provider video ID
- URL/reference
- language
- country
- official flag
- publication date
- source/provenance

Do NOT download copyrighted video media unless explicitly supported by the provider's licensing/terms.

---

# 12. External IDs

Create a generalized external identifier system.

Example providers:

- TMDb
- IMDb
- TVDb
- Wikidata
- other future sources

External IDs should support multiple Aurora entity types.

Enforce uniqueness where appropriate.

---

# 13. Provider System

Build a provider abstraction.

Example:

MetadataProvider

    TMDbProvider
    IMDbDatasetProvider
    FutureProvider

Providers should NOT write directly into arbitrary database tables.

Flow:

Provider
    |
Raw Provider Data
    |
Normalizer
    |
Entity Matching
    |
Deduplication
    |
Aurora Entity
    |
PostgreSQL

Provider adapters should be replaceable.

---

# 14. Provenance

Aurora Core needs to know where metadata originated.

Track:

- provider
- provider entity ID
- ingestion date
- last synchronization
- source version if available
- field provenance where practical
- ingestion job
- confidence/match information

Never silently merge questionable entities.

Ambiguous matches should be detectable and reviewable.

---

# 15. Ingestion

Metadata ingestion MUST run asynchronously.

Do NOT perform large provider synchronization jobs inside HTTP requests.

Use workers.

Example:

Provider
    |
Ingestion Job
    |
Raw Record
    |
Normalizer
    |
Entity Matcher
    |
Deduplication
    |
Database
    |
Search Index Update

Jobs must be:

- retryable
- idempotent
- observable
- resumable where practical
- protected against duplicate processing

---

# 16. Search

Search is a major Aurora Core feature.

Initial search endpoint:

GET /api/v1/search?q=alien

Search across:

- movies
- TV series
- episodes where appropriate
- people
- collections
- companies

Initial implementation may use PostgreSQL:

- full-text search
- trigram indexes
- normalized title search
- alternate titles

Design a search abstraction.

Example:

SearchProvider

    PostgreSQLSearchProvider

Future:

    OpenSearchProvider

Do NOT make API clients aware of the underlying search engine.

---

# 17. Core 0.1 API

Initial endpoints should include at minimum:

GET /api/v1/health
GET /api/v1/ready

GET /api/v1/search

GET /api/v1/movies/{aurora_id}
GET /api/v1/movies/{aurora_id}/credits
GET /api/v1/movies/{aurora_id}/images
GET /api/v1/movies/{aurora_id}/videos

GET /api/v1/series/{aurora_id}
GET /api/v1/series/{aurora_id}/seasons

GET /api/v1/seasons/{aurora_id}

GET /api/v1/episodes/{aurora_id}

GET /api/v1/people/{aurora_id}
GET /api/v1/people/{aurora_id}/credits

GET /api/v1/collections/{aurora_id}

GET /api/v1/companies/{aurora_id}

Administrative/provider endpoints should be separately protected.

---

# 18. API Response Design

Responses should be versioned and consistent.

Example:

GET /api/v1/movies/{id}

Response concept:

{
    "id": "aur_movie_...",
    "type": "movie",
    "title": "Example Movie",
    "original_title": "Example Movie",
    "overview": "...",
    "release_date": "2026-01-01",
    "runtime": 120,
    "genres": [],
    "cast": [],
    "crew": [],
    "companies": [],
    "collection": null,
    "images": {},
    "videos": [],
    "external_ids": {},
    "created_at": "...",
    "updated_at": "..."
}

Use proper Pydantic response schemas.

Do NOT serialize SQLAlchemy models directly.

---

# 19. PostgreSQL

PostgreSQL is authoritative.

Important application data must NEVER exist only in Redis.

Use:

- foreign keys
- unique constraints
- check constraints
- appropriate indexes
- partial indexes where beneficial
- transaction boundaries
- PostgreSQL full-text capabilities

Use PgBouncer for production connection pooling.

Avoid N+1 queries.

Use eager/select-in loading appropriately.

API endpoints should be profiled for query count.

---

# 20. Redis

Redis may provide:

- API caching
- search-result caching
- rate limiting
- distributed locks
- temporary state
- worker coordination
- job queues

Redis must NOT become the permanent source of truth.

Aurora Core should recover gracefully from Redis loss.

---

# 21. Caching

Create a cache abstraction.

Cache suitable read-heavy endpoints such as:

- movie details
- person details
- series details
- common search queries

Use sensible TTLs.

Implement cache invalidation when records change.

Do not cache authorization-sensitive responses without appropriate cache keys.

---

# 22. Security

Production security requirements:

- secrets never committed to Git
- `.env.example` only contains examples
- validate all API inputs
- secure database credentials
- secure Redis configuration
- restricted CORS
- security headers
- request-size limits
- rate limiting
- administrative authorization
- API-key support where required
- audit logging for administrative actions
- dependency vulnerability scanning
- container security considerations
- no hard-coded provider tokens

Sensitive provider credentials should be encrypted at rest where appropriate.

---

# 23. Observability

Implement structured JSON-compatible logging.

Every request should have a correlation/request ID.

Log useful context without logging secrets.

Prepare metrics for:

- HTTP request count
- response latency
- HTTP errors
- database latency
- Redis latency
- queue depth
- worker jobs
- worker failures
- provider failures
- ingestion throughput

Architecture should be compatible with:

- Prometheus
- Grafana
- Loki
- OpenTelemetry

Do not require the full observability stack merely to run local development.

---

# 24. Health Checks

Implement:

GET /health

Used for liveness.

GET /ready

Readiness should validate required dependencies such as PostgreSQL.

Redis readiness behavior should reflect whether Redis is mandatory for the configured deployment.

Docker services must have health checks.

---

# 25. Repository Structure

Create approximately:

aurora-core/
├── app/
│   ├── api/
│   │   └── v1/
│   │       ├── movies.py
│   │       ├── series.py
│   │       ├── seasons.py
│   │       ├── episodes.py
│   │       ├── people.py
│   │       ├── collections.py
│   │       ├── companies.py
│   │       ├── search.py
│   │       └── health.py
│   │
│   ├── core/
│   │       ├── config.py
│   │       ├── database.py
│   │       ├── redis.py
│   │       ├── logging.py
│   │       └── security.py
│   │
│   ├── models/
│   ├── schemas/
│   ├── repositories/
│   ├── services/
│   │       ├── search/
│   │       ├── metadata/
│   │       └── ingestion/
│   │
│   ├── providers/
│   ├── workers/
│   └── main.py
│
├── migrations/
├── tests/
│   ├── unit/
│   ├── integration/
│   └── load/
│
├── docs/
│   ├── architecture.md
│   ├── database.md
│   ├── api.md
│   └── erd.md
│
├── docker/
│   ├── api.Dockerfile
│   └── worker.Dockerfile
│
├── .github/
│   └── workflows/
│
├── docker-compose.yml
├── docker-compose.prod.yml
├── pyproject.toml
├── alembic.ini
├── .env.example
├── .gitignore
└── README.md

Codex may improve this layout where technically justified.

Keep module boundaries clear.

---

# 26. Database ERD

Create a documented ERD before considering Core 0.1 complete.

Include relationships between:

- movies
- series
- seasons
- episodes
- people
- credits
- companies
- genres
- keywords
- collections
- languages
- countries
- certifications
- release dates
- alternate titles
- artwork
- videos
- providers
- provider records
- external IDs
- ingestion jobs
- provenance records

Avoid unnecessary JSON blobs when normalized relational data is appropriate.

JSONB may be used for provider-specific raw data and genuinely flexible metadata.

---

# 27. Migrations

Use Alembic.

The application must NOT create production database tables automatically on startup.

Database changes require migrations.

CI should test:

fresh database
    |
run migrations
    |
start application
    |
run tests

Also test migration upgrades.

---

# 28. Docker

Provide:

docker-compose.yml

for development.

Provide:

docker-compose.prod.yml

for production-oriented deployment.

Services should include as appropriate:

- Aurora Core API
- Aurora worker
- PostgreSQL
- PgBouncer
- Redis

Traefik integration should be supported.

Do not bundle unrelated applications into the Core repository.

---

# 29. Configuration

Use environment-based configuration.

Provide `.env.example`.

Possible variables:

AURORA_ENV
AURORA_LOG_LEVEL

DATABASE_URL

REDIS_URL

AURORA_SECRET_KEY

TMDB_API_KEY

CORS_ORIGINS

WORKER_CONCURRENCY

CACHE_DEFAULT_TTL

RATE_LIMIT_*

Do NOT place actual credentials in examples.

---

# 30. Testing

Testing is mandatory.

Create:

## Unit tests

Test:

- normalization
- matching
- ID generation
- services
- schema validation
- provider adapters
- search logic

## Integration tests

Test:

- PostgreSQL
- Redis
- API endpoints
- migrations
- ingestion
- entity relationships

## API tests

Test:

- success responses
- invalid IDs
- missing entities
- pagination
- filtering
- validation
- error format

## Load tests

Create k6 or Locust tests.

Simulate at least:

- searching
- opening movies
- opening TV series
- opening people
- pagination

Target:

500 concurrent interactive users.

Document how to run the load test.

---

# 31. CI

Create CI workflow.

Pipeline should include:

lint
    |
type checking
    |
unit tests
    |
integration tests
    |
migration test
    |
container build
    |
security/dependency checks

Use appropriate Python tooling.

Suggested tools may include:

- Ruff
- mypy or Pyright
- pytest

Codex may select equivalent mature tooling if justified.

---

# 32. Documentation

Create:

README.md

docs/architecture.md
docs/database.md
docs/api.md
docs/erd.md

README should explain:

- project purpose
- architecture
- prerequisites
- local setup
- environment configuration
- Docker startup
- migrations
- tests
- API documentation
- worker operation

FastAPI OpenAPI documentation should remain available in development.

Production exposure of API documentation should be configurable.

---

# 33. Core 0.1 Scope

DO build:

- database foundation
- Aurora IDs
- movies
- television
- seasons
- episodes
- people
- credits
- companies
- genres
- keywords
- collections
- artwork metadata
- videos
- external IDs
- provider abstraction
- ingestion pipeline
- normalization
- deduplication foundations
- search
- REST API
- caching
- workers
- production Docker architecture
- tests
- documentation

DO NOT build into Core 0.1:

- Plex integration
- Overseerr integration
- Radarr integration
- Sonarr integration
- video streaming
- transcoding
- CouchSurfer UI
- Aurora Media website UI

Those are separate applications/integrations.

---

# 34. Initial Milestone

The first meaningful milestone is:

A client can search Aurora Core for a movie, TV series, or person and retrieve a complete normalized Aurora-owned record.

Example:

GET /api/v1/search?q=the%20matrix

Aurora returns matching Aurora entities.

The client can then call:

GET /api/v1/movies/{aurora_id}

and receive:

- title
- overview
- release information
- runtime
- genres
- cast
- crew
- directors
- writers
- companies
- collection
- artwork metadata
- videos
- external IDs

Likewise:

GET /api/v1/people/{aurora_id}

returns:

- person information
- images
- external IDs
- known credits/filmography

---

# 35. Future Aurora Ecosystem

Aurora Core should eventually support:

Aurora Core
    |
    +--- Aurora Media
    |       IMDb-style web experience
    |
    +--- Aurora Curator
    |       Media identification/organization
    |
    +--- CouchSurfer
    |       Personal media front end
    |
    +--- Future Aurora streaming server
    |
    +--- Mobile applications
    |
    +--- Aurora AI/intelligence

Applications should consume Core APIs.

Do not tightly couple Core to any one application.

---

# 36. Important Engineering Principles

1. Production first.

2. Aurora owns its entity identities.

3. External metadata services are providers, not Aurora's database.

4. PostgreSQL is authoritative.

5. Redis is disposable acceleration/coordination.

6. Expensive work runs asynchronously.

7. APIs remain stateless and horizontally scalable.

8. Maintain clear module boundaries.

9. Avoid premature microservices.

10. Every schema change uses a migration.

11. Everything important is testable.

12. Provider provenance must be retained.

13. Do not duplicate entities because two providers describe them differently.

14. Do not silently merge uncertain matches.

15. Design interfaces so external providers can be replaced.

16. Core must function independently of Plex.

---

# 37. Codex Instructions

Begin implementing the repository.

Do not merely produce another architecture proposal.

Create the actual files and working code.

Implementation order:

1. Create repository structure.
2. Create Python project configuration.
3. Create application configuration.
4. Create database connection layer.
5. Create Redis layer.
6. Create Aurora ID system.
7. Implement SQLAlchemy models.
8. Create initial Alembic migration.
9. Create Pydantic schemas.
10. Create repositories.
11. Create service layer.
12. Create provider abstraction.
13. Implement initial metadata provider adapter.
14. Create ingestion/normalization pipeline.
15. Implement PostgreSQL search.
16. Implement REST API.
17. Implement workers.
18. Add caching.
19. Add logging/observability foundations.
20. Add Docker environment.
21. Add tests.
22. Add load tests.
23. Add CI.
24. Complete documentation.

After each major stage:

- run tests
- run linting
- run type checking
- fix failures before proceeding

Do not leave fake implementations silently returning successful responses.

If an external credential is required, create the integration interface and clearly document the required environment variable.

Use TODO comments only for genuinely deferred Core features.

---

# 38. Definition of Done — Aurora Core 0.1

Core 0.1 is complete when:

- Docker Compose can start the complete development environment.
- PostgreSQL migrations run successfully from an empty database.
- Aurora IDs are generated independently of providers.
- Movies can be ingested and normalized.
- TV entities can be ingested and normalized.
- People can be ingested and normalized.
- External IDs are linked without becoming primary IDs.
- Credits and relationships work.
- Artwork provenance is retained.
- Search returns movies, TV series, and people.
- Detail APIs return normalized records.
- Redis caching works.
- Background jobs work.
- Duplicate ingestion is protected.
- API instances can run horizontally.
- Automated tests pass.
- CI passes.
- Health/readiness checks work.
- Load-test scripts exist and are documented.
- The architecture and ERD are documented.
- No Plex dependency exists.
- No real secrets exist in Git.
- README provides enough information for another developer to launch the project.

At completion, output:

1. Complete repository tree.
2. Database/ERD summary.
3. API endpoint list.
4. Docker service list.
5. Test results.
6. Known limitations.
7. Commands required to launch Aurora Core.
8. Recommended next milestone.

Do not begin CouchSurfer or other Aurora applications until the Core 0.1 foundation is operational.