Add series/season/episode ingestion pipeline
CI / lint (push) Canceled after 0s
CI / typecheck (push) Canceled after 0s
CI / unit-tests (push) Canceled after 0s
CI / integration-tests (push) Canceled after 0s
CI / container-build (push) Canceled after 0s
CI / dependency-audit (push) Canceled after 0s
CI / lint (push) Canceled after 0s
CI / typecheck (push) Canceled after 0s
CI / unit-tests (push) Canceled after 0s
CI / integration-tests (push) Canceled after 0s
CI / container-build (push) Canceled after 0s
CI / dependency-audit (push) Canceled after 0s
Movie and person ingestion existed, but the series/season/episode API
endpoints had no way to actually populate them from TMDb -- the
biggest remaining gap in Core 0.1's television scope.
- normalize_series/normalize_season in the metadata normalizer,
including two TMDb quirks that would otherwise silently produce
empty data: /tv nests keywords and alternative_titles under
"results" (not "keywords"/"titles" like /movie), and series-level
creators come from a separate created_by array, not the credits
cast/crew lists.
- ingest_series upserts the Series row plus season *stubs* (number,
title, air date, poster -- everything TMDb inlines on /tv/{id});
ingest_season fetches and upserts one season's full episode list,
and requires the series to already exist (TMDb has no single
endpoint for a whole series' episodes, so this has to be a separate
per-season call).
- Wired through the same job-lifecycle path as movie/person ingestion
(runner.py, arq tasks, admin endpoints), with the shared boilerplate
between them factored into runner._run_ingestion and
admin.py:_enqueue rather than copy-pasted a third and fourth time.
- Unit tests for both normalizers (runnable without a DB) and
integration tests for the full ingest_series -> ingest_season flow,
including the series-must-exist-first error and idempotency on
re-ingestion.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
669034896a
commit
3f6332559a
+18
-1
@@ -13,7 +13,12 @@ import uuid
|
||||
from typing import Any
|
||||
|
||||
from app.core.logging import get_logger
|
||||
from app.services.ingestion.runner import run_movie_ingestion, run_person_ingestion
|
||||
from app.services.ingestion.runner import (
|
||||
run_movie_ingestion,
|
||||
run_person_ingestion,
|
||||
run_season_ingestion,
|
||||
run_series_ingestion,
|
||||
)
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
@@ -26,3 +31,15 @@ async def ingest_movie_task(ctx: dict[str, Any], job_id: str, external_id: str)
|
||||
async def ingest_person_task(ctx: dict[str, Any], job_id: str, external_id: str) -> str:
|
||||
person_id = await run_person_ingestion(uuid.UUID(job_id), external_id)
|
||||
return str(person_id)
|
||||
|
||||
|
||||
async def ingest_series_task(ctx: dict[str, Any], job_id: str, external_id: str) -> str:
|
||||
series_id = await run_series_ingestion(uuid.UUID(job_id), external_id)
|
||||
return str(series_id)
|
||||
|
||||
|
||||
async def ingest_season_task(
|
||||
ctx: dict[str, Any], job_id: str, series_external_id: str, season_number: int
|
||||
) -> str:
|
||||
season_id = await run_season_ingestion(uuid.UUID(job_id), series_external_id, season_number)
|
||||
return str(season_id)
|
||||
|
||||
@@ -13,7 +13,12 @@ from arq.connections import RedisSettings
|
||||
|
||||
from app.core.config import get_settings
|
||||
from app.core.logging import configure_logging, get_logger
|
||||
from app.workers.tasks import ingest_movie_task, ingest_person_task
|
||||
from app.workers.tasks import (
|
||||
ingest_movie_task,
|
||||
ingest_person_task,
|
||||
ingest_season_task,
|
||||
ingest_series_task,
|
||||
)
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
@@ -28,7 +33,7 @@ async def shutdown(ctx: dict) -> None:
|
||||
|
||||
|
||||
class WorkerSettings:
|
||||
functions = [ingest_movie_task, ingest_person_task]
|
||||
functions = [ingest_movie_task, ingest_person_task, ingest_series_task, ingest_season_task]
|
||||
on_startup = startup
|
||||
on_shutdown = shutdown
|
||||
redis_settings = RedisSettings.from_dsn(get_settings().REDIS_URL)
|
||||
|
||||
Reference in New Issue
Block a user