R3ap3r/Aurora-Core
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
0be052bc1fbeef63ed24898542a39e2ee4863cae
Aurora-Core/docs/aurora-core-spec.md
T

1360 lines
22 KiB
Markdown
Raw Normal View History

Initial Aurora Core 0.1 implementation
2026-09-04 17:34:03 -05:00
# Aurora Core — Codex Build Specification
## Project
**Name:** Aurora Core
**Company:** Aurora Dragon Studio
**Purpose:** Production-grade media metadata, discovery, and intelligence backend.
Aurora Core will become the canonical data layer for the Aurora media ecosystem.
It should function conceptually like our own TMDb/IMDb-style platform, providing normalized information about:
- Movies
- Television series
- Seasons
- Episodes
- People
- Cast and crew
- Companies
- Collections/franchises
- Genres
- Keywords
- Artwork
- Videos/trailers
- Release information
- Ratings/certifications
- External provider IDs
- Relationships between all of the above
Aurora Core is NOT a Plex application.
Plex, Overseerr, Radarr, Sonarr, Jellyfin, Emby, and future Aurora streaming software will be consumers or integrations built around Core.
Do NOT make Aurora Core dependent on Plex.
---
# 1. Primary Architecture
Build Aurora Core as a production-grade **modular monolith**.
Do NOT prematurely break the application into microservices.
Architecture:
Internet
|
Traefik
|
Aurora Core API replicas
|
+--- PostgreSQL
|
+--- PgBouncer
|
+--- Redis
|
+--- Background Workers
|
+--- Metadata ingestion
+--- Artwork processing
+--- Search indexing
+--- Provider synchronization
Primary technology stack:
- Python
- FastAPI
- PostgreSQL
- SQLAlchemy 2.x
- Alembic
- Pydantic
- Redis
- PgBouncer
- Background job workers
- Docker
- Docker Compose
- Traefik-compatible deployment
The API must remain stateless so multiple replicas can run simultaneously.
Example:
Traefik
|
+--- aurora-core-api-1
+--- aurora-core-api-2
+--- aurora-core-api-3
Workers must also scale independently.
---
# 2. Production Requirements
This is NOT a prototype.
Design Aurora Core for production use from the beginning.
Initial engineering target:
- 500 concurrent interactive users
- Horizontal API scaling
- Horizontal worker scaling
- Stateless API containers
- Database connection pooling
- Redis caching
- Background processing
- Idempotent jobs
- Structured logging
- Health checks
- Readiness checks
- Metrics
- Rate limiting
- Graceful shutdown
- Proper database migrations
- Automated tests
Initial performance goals:
| Requirement | Target |
|---|---|
| Concurrent interactive users | 500 |
| Normal API p95 | < 250 ms |
| Search p95 | < 500 ms |
| Core 5xx rate | < 0.1% |
| API horizontal scaling | Required |
| Worker horizontal scaling | Required |
| Lost jobs | 0 |
| Duplicate ingestion records | 0 |
These targets must ultimately be verified using load testing.
---
# 3. Aurora-Owned IDs
Aurora Core MUST own its identifiers.
Do NOT use TMDb IDs, IMDb IDs, TVDb IDs, or other provider IDs as primary keys.
Example:
aur_movie_01K5A8XXXXXXXXXXXX
or another suitable sortable globally unique ID scheme such as UUIDv7.
Internal database primary keys may use UUIDv7.
External IDs must be separate records.
Example:
Aurora Movie:
Aurora ID: aur_movie_xxxxx
External IDs:
TMDb: 550
IMDb: tt0137523
TVDb: ...
This allows Aurora Core to survive provider changes.
External providers are DATA SOURCES.
They are not Aurora Core's identity system.
---
# 4. Core Media Model
## Movies
Movie records should support:
- Aurora ID
- title
- original title
- alternate titles
- overview
- tagline
- release dates
- runtime
- status
- original language
- spoken languages
- countries
- genres
- keywords
- certifications
- budget
- revenue
- popularity data
- cast
- crew
- directors
- writers
- producers
- production companies
- distributors
- collections/franchises
- posters
- backdrops
- logos
- artwork
- trailers
- videos
- external IDs
- recommendations
- similar titles
- source/provenance information
- created timestamp
- updated timestamp
---
# 5. Television Model
Support:
Series
|
+--- Seasons
|
+--- Episodes
Series should support:
- title
- original title
- alternate titles
- overview
- tagline
- first air date
- last air date
- status
- networks
- production companies
- creators
- cast
- crew
- genres
- keywords
- languages
- countries
- artwork
- videos
- external IDs
Season should support:
- season number
- title
- overview
- air date
- episode count
- artwork
- external IDs
Episode should support:
- episode number
- season number
- title
- overview
- air date
- runtime
- cast
- guest stars
- crew
- directors
- writers
- still images
- external IDs
---
# 6. People
People are first-class Aurora entities.
Person records should support:
- Aurora ID
- name
- aliases
- biography
- birthday
- death date
- birthplace
- gender where legitimately provided by metadata sources
- profile images
- external IDs
- known-for department
- acting credits
- directing credits
- writing credits
- production credits
- other crew credits
Credits should NOT be stored as duplicated text blobs.
Use normalized relationship tables.
---
# 7. Credits
Create a flexible credit system.
Examples:
Person
|
+--- Movie
| role = actor
| character = "..."
|
+--- Movie
| role = director
|
+--- Episode
role = guest_star
Credits should support:
- person
- media entity
- department
- job
- character
- billing order
- episode count where appropriate
- source information
---
# 8. Companies
Support entities such as:
- production companies
- networks
- studios
- distributors
Fields should include:
- Aurora ID
- company name
- description
- country
- logo
- external IDs
- parent company where applicable
---
# 9. Collections / Franchises
Collections should be first-class entities.
Examples:
Alien
Star Wars
Fast & Furious
Marvel Cinematic Universe
Support:
Collection
|
+--- Movies
+--- Series
+--- Related collections
Do not assume every provider's concept of "collection" is sufficient for Aurora's future requirements.
---
# 10. Artwork System
Do NOT simply place poster_url columns on movie tables.
Artwork must be its own entity.
Artwork fields:
- Aurora artwork ID
- associated entity
- artwork type
- language
- width
- height
- aspect ratio
- provider
- provider asset ID
- original URL
- cached/local location where permitted
- checksum
- attribution
- licensing information
- source/provenance
- created timestamp
Artwork types:
- poster
- backdrop
- logo
- profile
- still
- banner
- thumbnail
Aurora must retain sufficient provider/licensing information to know where artwork originated.
Do not assume externally sourced artwork can automatically be redistributed.
---
# 11. Videos
Videos should be normalized entities.
Support:
- trailer
- teaser
- clip
- featurette
- behind-the-scenes
- interview
- other
Store:
- provider
- provider video ID
- URL/reference
- language
- country
- official flag
- publication date
- source/provenance
Do NOT download copyrighted video media unless explicitly supported by the provider's licensing/terms.
---
# 12. External IDs
Create a generalized external identifier system.
Example providers:
- TMDb
- IMDb
- TVDb
- Wikidata
- other future sources
External IDs should support multiple Aurora entity types.
Enforce uniqueness where appropriate.
---
# 13. Provider System
Build a provider abstraction.
Example:
MetadataProvider
TMDbProvider
IMDbDatasetProvider
FutureProvider
Providers should NOT write directly into arbitrary database tables.
Flow:
Provider
|
Raw Provider Data
|
Normalizer
|
Entity Matching
|
Deduplication
|
Aurora Entity
|
PostgreSQL
Provider adapters should be replaceable.
---
# 14. Provenance
Aurora Core needs to know where metadata originated.
Track:
- provider
- provider entity ID
- ingestion date
- last synchronization
- source version if available
- field provenance where practical
- ingestion job
- confidence/match information
Never silently merge questionable entities.
Ambiguous matches should be detectable and reviewable.
---
# 15. Ingestion
Metadata ingestion MUST run asynchronously.
Do NOT perform large provider synchronization jobs inside HTTP requests.
Use workers.
Example:
Provider
|
Ingestion Job
|
Raw Record
|
Normalizer
|
Entity Matcher
|
Deduplication
|
Database
|
Search Index Update
Jobs must be:
- retryable
- idempotent
- observable
- resumable where practical
- protected against duplicate processing
---
# 16. Search
Search is a major Aurora Core feature.
Initial search endpoint:
GET /api/v1/search?q=alien
Search across:
- movies
- TV series
- episodes where appropriate
- people
- collections
- companies
Initial implementation may use PostgreSQL:
- full-text search
- trigram indexes
- normalized title search
- alternate titles
Design a search abstraction.
Example:
SearchProvider
PostgreSQLSearchProvider
Future:
OpenSearchProvider
Do NOT make API clients aware of the underlying search engine.
---
# 17. Core 0.1 API
Initial endpoints should include at minimum:
GET /api/v1/health
GET /api/v1/ready
GET /api/v1/search
GET /api/v1/movies/{aurora_id}
GET /api/v1/movies/{aurora_id}/credits
GET /api/v1/movies/{aurora_id}/images
GET /api/v1/movies/{aurora_id}/videos
GET /api/v1/series/{aurora_id}
GET /api/v1/series/{aurora_id}/seasons
GET /api/v1/seasons/{aurora_id}
GET /api/v1/episodes/{aurora_id}
GET /api/v1/people/{aurora_id}
GET /api/v1/people/{aurora_id}/credits
GET /api/v1/collections/{aurora_id}
GET /api/v1/companies/{aurora_id}
Administrative/provider endpoints should be separately protected.
---
# 18. API Response Design
Responses should be versioned and consistent.
Example:
GET /api/v1/movies/{id}
Response concept:
{
"id": "aur_movie_...",
"type": "movie",
"title": "Example Movie",
"original_title": "Example Movie",
"overview": "...",
"release_date": "2026-01-01",
"runtime": 120,
"genres": [],
"cast": [],
"crew": [],
"companies": [],
"collection": null,
"images": {},
"videos": [],
"external_ids": {},
"created_at": "...",
"updated_at": "..."
}
Use proper Pydantic response schemas.
Do NOT serialize SQLAlchemy models directly.
---
# 19. PostgreSQL
PostgreSQL is authoritative.
Important application data must NEVER exist only in Redis.
Use:
- foreign keys
- unique constraints
- check constraints
- appropriate indexes
- partial indexes where beneficial
- transaction boundaries
- PostgreSQL full-text capabilities
Use PgBouncer for production connection pooling.
Avoid N+1 queries.
Use eager/select-in loading appropriately.
API endpoints should be profiled for query count.
---
# 20. Redis
Redis may provide:
- API caching
- search-result caching
- rate limiting
- distributed locks
- temporary state
- worker coordination
- job queues
Redis must NOT become the permanent source of truth.
Aurora Core should recover gracefully from Redis loss.
---
# 21. Caching
Create a cache abstraction.
Cache suitable read-heavy endpoints such as:
- movie details
- person details
- series details
- common search queries
Use sensible TTLs.
Implement cache invalidation when records change.
Do not cache authorization-sensitive responses without appropriate cache keys.
---
# 22. Security
Production security requirements:
- secrets never committed to Git
- `.env.example` only contains examples
- validate all API inputs
- secure database credentials
- secure Redis configuration
- restricted CORS
- security headers
- request-size limits
- rate limiting
- administrative authorization
- API-key support where required
- audit logging for administrative actions
- dependency vulnerability scanning
- container security considerations
- no hard-coded provider tokens
Sensitive provider credentials should be encrypted at rest where appropriate.
---
# 23. Observability
Implement structured JSON-compatible logging.
Every request should have a correlation/request ID.
Log useful context without logging secrets.
Prepare metrics for:
- HTTP request count
- response latency
- HTTP errors
- database latency
- Redis latency
- queue depth
- worker jobs
- worker failures
- provider failures
- ingestion throughput
Architecture should be compatible with:
- Prometheus
- Grafana
- Loki
- OpenTelemetry
Do not require the full observability stack merely to run local development.
---
# 24. Health Checks
Implement:
GET /health
Used for liveness.
GET /ready
Readiness should validate required dependencies such as PostgreSQL.
Redis readiness behavior should reflect whether Redis is mandatory for the configured deployment.
Docker services must have health checks.
---
# 25. Repository Structure
Create approximately:
aurora-core/
├── app/
│ ├── api/
│ │ └── v1/
│ │ ├── movies.py
│ │ ├── series.py
│ │ ├── seasons.py
│ │ ├── episodes.py
│ │ ├── people.py
│ │ ├── collections.py
│ │ ├── companies.py
│ │ ├── search.py
│ │ └── health.py
│ │
│ ├── core/
│ │ ├── config.py
│ │ ├── database.py
│ │ ├── redis.py
│ │ ├── logging.py
│ │ └── security.py
│ │
│ ├── models/
│ ├── schemas/
│ ├── repositories/
│ ├── services/
│ │ ├── search/
│ │ ├── metadata/
│ │ └── ingestion/
│ │
│ ├── providers/
│ ├── workers/
│ └── main.py
│
├── migrations/
├── tests/
│ ├── unit/
│ ├── integration/
│ └── load/
│
├── docs/
│ ├── architecture.md
│ ├── database.md
│ ├── api.md
│ └── erd.md
│
├── docker/
│ ├── api.Dockerfile
│ └── worker.Dockerfile
│
├── .github/
│ └── workflows/
│
├── docker-compose.yml
├── docker-compose.prod.yml
├── pyproject.toml
├── alembic.ini
├── .env.example
├── .gitignore
└── README.md
Codex may improve this layout where technically justified.
Keep module boundaries clear.
---
# 26. Database ERD
Create a documented ERD before considering Core 0.1 complete.
Include relationships between:
- movies
- series
- seasons
- episodes
- people
- credits
- companies
- genres
- keywords
- collections
- languages
- countries
- certifications
- release dates
- alternate titles
- artwork
- videos
- providers
- provider records
- external IDs
- ingestion jobs
- provenance records
Avoid unnecessary JSON blobs when normalized relational data is appropriate.
JSONB may be used for provider-specific raw data and genuinely flexible metadata.
---
# 27. Migrations
Use Alembic.
The application must NOT create production database tables automatically on startup.
Database changes require migrations.
CI should test:
fresh database
|
run migrations
|
start application
|
run tests
Also test migration upgrades.
---
# 28. Docker
Provide:
docker-compose.yml
for development.
Provide:
docker-compose.prod.yml
for production-oriented deployment.
Services should include as appropriate:
- Aurora Core API
- Aurora worker
- PostgreSQL
- PgBouncer
- Redis
Traefik integration should be supported.
Do not bundle unrelated applications into the Core repository.
---
# 29. Configuration
Use environment-based configuration.
Provide `.env.example`.
Possible variables:
AURORA_ENV
AURORA_LOG_LEVEL
DATABASE_URL
REDIS_URL
AURORA_SECRET_KEY
TMDB_API_KEY
CORS_ORIGINS
WORKER_CONCURRENCY
CACHE_DEFAULT_TTL
RATE_LIMIT_*
Do NOT place actual credentials in examples.
---
# 30. Testing
Testing is mandatory.
Create:
## Unit tests
Test:
- normalization
- matching
- ID generation
- services
- schema validation
- provider adapters
- search logic
## Integration tests
Test:
- PostgreSQL
- Redis
- API endpoints
- migrations
- ingestion
- entity relationships
## API tests
Test:
- success responses
- invalid IDs
- missing entities
- pagination
- filtering
- validation
- error format
## Load tests
Create k6 or Locust tests.
Simulate at least:
- searching
- opening movies
- opening TV series
- opening people
- pagination
Target:
500 concurrent interactive users.
Document how to run the load test.
---
# 31. CI
Create CI workflow.
Pipeline should include:
lint
|
type checking
|
unit tests
|
integration tests
|
migration test
|
container build
|
security/dependency checks
Use appropriate Python tooling.
Suggested tools may include:
- Ruff
- mypy or Pyright
- pytest
Codex may select equivalent mature tooling if justified.
---
# 32. Documentation
Create:
README.md
docs/architecture.md
docs/database.md
docs/api.md
docs/erd.md
README should explain:
- project purpose
- architecture
- prerequisites
- local setup
- environment configuration
- Docker startup
- migrations
- tests
- API documentation
- worker operation
FastAPI OpenAPI documentation should remain available in development.
Production exposure of API documentation should be configurable.
---
# 33. Core 0.1 Scope
DO build:
- database foundation
- Aurora IDs
- movies
- television
- seasons
- episodes
- people
- credits
- companies
- genres
- keywords
- collections
- artwork metadata
- videos
- external IDs
- provider abstraction
- ingestion pipeline
- normalization
- deduplication foundations
- search
- REST API
- caching
- workers
- production Docker architecture
- tests
- documentation
DO NOT build into Core 0.1:
- Plex integration
- Overseerr integration
- Radarr integration
- Sonarr integration
- video streaming
- transcoding
- CouchSurfer UI
- Aurora Media website UI
Those are separate applications/integrations.
---
# 34. Initial Milestone
The first meaningful milestone is:
A client can search Aurora Core for a movie, TV series, or person and retrieve a complete normalized Aurora-owned record.
Example:
GET /api/v1/search?q=the%20matrix
Aurora returns matching Aurora entities.
The client can then call:
GET /api/v1/movies/{aurora_id}
and receive:
- title
- overview
- release information
- runtime
- genres
- cast
- crew
- directors
- writers
- companies
- collection
- artwork metadata
- videos
- external IDs
Likewise:
GET /api/v1/people/{aurora_id}
returns:
- person information
- images
- external IDs
- known credits/filmography
---
# 35. Future Aurora Ecosystem
Aurora Core should eventually support:
Aurora Core
|
+--- Aurora Media
| IMDb-style web experience
|
+--- Aurora Curator
| Media identification/organization
|
+--- CouchSurfer
| Personal media front end
|
+--- Future Aurora streaming server
|
+--- Mobile applications
|
+--- Aurora AI/intelligence
Applications should consume Core APIs.
Do not tightly couple Core to any one application.
---
# 36. Important Engineering Principles
1. Production first.
2. Aurora owns its entity identities.
3. External metadata services are providers, not Aurora's database.
4. PostgreSQL is authoritative.
5. Redis is disposable acceleration/coordination.
6. Expensive work runs asynchronously.
7. APIs remain stateless and horizontally scalable.
8. Maintain clear module boundaries.
9. Avoid premature microservices.
10. Every schema change uses a migration.
11. Everything important is testable.
12. Provider provenance must be retained.
13. Do not duplicate entities because two providers describe them differently.
14. Do not silently merge uncertain matches.
15. Design interfaces so external providers can be replaced.
16. Core must function independently of Plex.
---
# 37. Codex Instructions
Begin implementing the repository.
Do not merely produce another architecture proposal.
Create the actual files and working code.
Implementation order:
1. Create repository structure.
2. Create Python project configuration.
3. Create application configuration.
4. Create database connection layer.
5. Create Redis layer.
6. Create Aurora ID system.
7. Implement SQLAlchemy models.
8. Create initial Alembic migration.
9. Create Pydantic schemas.
10. Create repositories.
11. Create service layer.
12. Create provider abstraction.
13. Implement initial metadata provider adapter.
14. Create ingestion/normalization pipeline.
15. Implement PostgreSQL search.
16. Implement REST API.
17. Implement workers.
18. Add caching.
19. Add logging/observability foundations.
20. Add Docker environment.
21. Add tests.
22. Add load tests.
23. Add CI.
24. Complete documentation.
After each major stage:
- run tests
- run linting
- run type checking
- fix failures before proceeding
Do not leave fake implementations silently returning successful responses.
If an external credential is required, create the integration interface and clearly document the required environment variable.
Use TODO comments only for genuinely deferred Core features.
---
# 38. Definition of Done — Aurora Core 0.1
Core 0.1 is complete when:
- Docker Compose can start the complete development environment.
- PostgreSQL migrations run successfully from an empty database.
- Aurora IDs are generated independently of providers.
- Movies can be ingested and normalized.
- TV entities can be ingested and normalized.
- People can be ingested and normalized.
- External IDs are linked without becoming primary IDs.
- Credits and relationships work.
- Artwork provenance is retained.
- Search returns movies, TV series, and people.
- Detail APIs return normalized records.
- Redis caching works.
- Background jobs work.
- Duplicate ingestion is protected.
- API instances can run horizontally.
- Automated tests pass.
- CI passes.
- Health/readiness checks work.
- Load-test scripts exist and are documented.
- The architecture and ERD are documented.
- No Plex dependency exists.
- No real secrets exist in Git.
- README provides enough information for another developer to launch the project.
At completion, output:
1. Complete repository tree.
2. Database/ERD summary.
3. API endpoint list.
4. Docker service list.
5. Test results.
6. Known limitations.
7. Commands required to launch Aurora Core.
8. Recommended next milestone.
Do not begin CouchSurfer or other Aurora applications until the Core 0.1 foundation is operational.
Reference in New Issue Copy Permalink