NixReaperandClaude Sonnet 5
b4d1e8f755
CI / lint (push) Canceled after 0s
CI / typecheck (push) Canceled after 0s
CI / unit-tests (push) Canceled after 0s
CI / integration-tests (push) Canceled after 0s
CI / container-build (push) Canceled after 0s
CI / dependency-audit (push) Canceled after 0s
Ingesting a series only ever produced season stubs -- getting actual episodes meant calling POST /admin/ingest/season once per season by hand, which is exactly what manual end-to-end testing kept requiring for every multi-season show. POST /api/v1/admin/ingest/series now takes include_seasons (default false, since a show can have dozens of seasons and that's dozens of TMDb calls fired from one request). When true, ingest_series_task fans out one ingest_season job per season once the series succeeds, using arq's own ctx['redis'] connection for the job (the same pattern arq's docs recommend for job chaining) rather than a second unrelated connection. Each fanned-out season job still goes through get_or_create_ingestion_job, so re-running a cascaded series ingestion reuses the same season job rows instead of creating duplicates. Co-Authored-By: Claude Sonnet 5 <[email protected]>
68 lines
2.4 KiB
Python
68 lines
2.4 KiB
Python
"""arq task functions. Thin wrappers around the ingestion runner so retry/
|
|
concurrency/observability behavior lives in arq's worker loop, not here.
|
|
|
|
Each task receives the `IngestionJob` row's id (created synchronously by the
|
|
enqueuing endpoint via `get_or_create_ingestion_job`, *not* generated here)
|
|
so status polling has a real, immediately-valid id to look up from the
|
|
moment the HTTP response comes back.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import uuid
|
|
from typing import Any
|
|
|
|
from app.core.logging import get_logger
|
|
from app.services.ingestion.runner import (
|
|
get_or_create_ingestion_job,
|
|
get_series_season_numbers,
|
|
run_movie_ingestion,
|
|
run_person_ingestion,
|
|
run_season_ingestion,
|
|
run_series_ingestion,
|
|
)
|
|
|
|
logger = get_logger(__name__)
|
|
|
|
|
|
async def ingest_movie_task(ctx: dict[str, Any], job_id: str, external_id: str) -> str:
|
|
movie_id = await run_movie_ingestion(uuid.UUID(job_id), external_id)
|
|
return str(movie_id)
|
|
|
|
|
|
async def ingest_person_task(ctx: dict[str, Any], job_id: str, external_id: str) -> str:
|
|
person_id = await run_person_ingestion(uuid.UUID(job_id), external_id)
|
|
return str(person_id)
|
|
|
|
|
|
async def ingest_series_task(
|
|
ctx: dict[str, Any], job_id: str, external_id: str, include_seasons: bool = False
|
|
) -> str:
|
|
series_id = await run_series_ingestion(uuid.UUID(job_id), external_id)
|
|
|
|
if include_seasons:
|
|
# Fan out one ingest_season job per season this series has, using
|
|
# the same arq connection this task is running on (arq hands every
|
|
# job's ctx a "redis" entry precisely for chaining/fan-out like
|
|
# this) — not a second, unrelated connection.
|
|
redis = ctx["redis"]
|
|
for season_number in await get_series_season_numbers(series_id):
|
|
season_job_id = await get_or_create_ingestion_job(
|
|
"season_sync", "season", f"{external_id}:{season_number}"
|
|
)
|
|
await redis.enqueue_job(
|
|
"ingest_season_task", str(season_job_id), external_id, season_number
|
|
)
|
|
logger.info(
|
|
"series_seasons_cascaded", series_id=str(series_id), external_id=external_id
|
|
)
|
|
|
|
return str(series_id)
|
|
|
|
|
|
async def ingest_season_task(
|
|
ctx: dict[str, Any], job_id: str, series_external_id: str, season_number: int
|
|
) -> str:
|
|
season_id = await run_season_ingestion(uuid.UUID(job_id), series_external_id, season_number)
|
|
return str(season_id)
|