Initial Aurora Core 0.1 implementation
CI / lint (push) Canceled after 0s
CI / typecheck (push) Canceled after 0s
CI / unit-tests (push) Canceled after 0s
CI / integration-tests (push) Canceled after 0s
CI / container-build (push) Canceled after 0s
CI / dependency-audit (push) Canceled after 0s
CI / lint (push) Canceled after 0s
CI / typecheck (push) Canceled after 0s
CI / unit-tests (push) Canceled after 0s
CI / integration-tests (push) Canceled after 0s
CI / container-build (push) Canceled after 0s
CI / dependency-audit (push) Canceled after 0s
Production-grade media metadata backend per docs/aurora-core-spec.md: Aurora-owned UUIDv7 identity system, normalized movie/TV/people/credits schema, provider abstraction with a TMDb adapter, an idempotent async ingestion pipeline (normalize -> match -> upsert -> provenance), a Postgres FTS/trigram search layer, a stateless FastAPI v1 API with Redis caching/rate-limiting, arq workers, dev/prod Docker Compose topologies, unit + integration + load tests, CI, and architecture/ database/API/ERD documentation. Co-Authored-By: Claude Sonnet 5 <[email protected]>
This commit is contained in:
@@ -0,0 +1,22 @@
|
||||
"""arq task functions. Thin wrappers around the ingestion runner so retry/
|
||||
concurrency/observability behavior lives in arq's worker loop, not here.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from app.core.logging import get_logger
|
||||
from app.services.ingestion.runner import run_movie_ingestion, run_person_ingestion
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
async def ingest_movie_task(ctx: dict[str, Any], external_id: str) -> str:
|
||||
movie_id = await run_movie_ingestion(external_id)
|
||||
return str(movie_id)
|
||||
|
||||
|
||||
async def ingest_person_task(ctx: dict[str, Any], external_id: str) -> str:
|
||||
person_id = await run_person_ingestion(external_id)
|
||||
return str(person_id)
|
||||
@@ -0,0 +1,37 @@
|
||||
"""arq worker process entrypoint.
|
||||
|
||||
Run with: `arq app.workers.worker.WorkerSettings`
|
||||
|
||||
Workers scale independently of the API (section 1) — run as many worker
|
||||
processes/containers as needed; arq's Redis-backed queue coordinates them so
|
||||
each job is picked up by exactly one worker.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from arq.connections import RedisSettings
|
||||
|
||||
from app.core.config import get_settings
|
||||
from app.core.logging import configure_logging, get_logger
|
||||
from app.workers.tasks import ingest_movie_task, ingest_person_task
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
async def startup(ctx: dict) -> None:
|
||||
configure_logging()
|
||||
logger.info("worker_started")
|
||||
|
||||
|
||||
async def shutdown(ctx: dict) -> None:
|
||||
logger.info("worker_stopped")
|
||||
|
||||
|
||||
class WorkerSettings:
|
||||
functions = [ingest_movie_task, ingest_person_task]
|
||||
on_startup = startup
|
||||
on_shutdown = shutdown
|
||||
redis_settings = RedisSettings.from_dsn(get_settings().REDIS_URL)
|
||||
max_jobs = get_settings().WORKER_CONCURRENCY
|
||||
job_timeout = 300
|
||||
max_tries = 5
|
||||
Reference in New Issue
Block a user