Fix job_id mismatch between ingestion enqueue and status endpoints
CI / lint (push) Canceled after 0s
CI / typecheck (push) Canceled after 0s
CI / unit-tests (push) Canceled after 0s
CI / integration-tests (push) Canceled after 0s
CI / container-build (push) Canceled after 0s
CI / dependency-audit (push) Canceled after 0s
CI / lint (push) Canceled after 0s
CI / typecheck (push) Canceled after 0s
CI / unit-tests (push) Canceled after 0s
CI / integration-tests (push) Canceled after 0s
CI / container-build (push) Canceled after 0s
CI / dependency-audit (push) Canceled after 0s
POST /api/v1/admin/ingest/movie returned arq's own internal queue
job id, but GET /api/v1/admin/ingest/jobs/{job_id} looked up our
IngestionJob DB row by its own UUID -- a completely different id that
didn't even exist yet at enqueue time (the row was only created once
the worker picked up the task). Every status poll 404'd.
Fix: create the IngestionJob row synchronously in the enqueue endpoint
(reusing the existing idempotency-key logic, so repeat enqueues for
the same tmdb_id reuse the same row rather than creating duplicates),
pass its real id into the arq task, and have the worker update that
row instead of creating a new one.
Found by actually running the milestone end-to-end: enqueue a real
TMDb ingestion, then poll its status.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
dd6ecce78d
commit
669034896a
+10
-4
@@ -1,9 +1,15 @@
|
||||
"""arq task functions. Thin wrappers around the ingestion runner so retry/
|
||||
concurrency/observability behavior lives in arq's worker loop, not here.
|
||||
|
||||
Each task receives the `IngestionJob` row's id (created synchronously by the
|
||||
enqueuing endpoint via `get_or_create_ingestion_job`, *not* generated here)
|
||||
so status polling has a real, immediately-valid id to look up from the
|
||||
moment the HTTP response comes back.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import uuid
|
||||
from typing import Any
|
||||
|
||||
from app.core.logging import get_logger
|
||||
@@ -12,11 +18,11 @@ from app.services.ingestion.runner import run_movie_ingestion, run_person_ingest
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
async def ingest_movie_task(ctx: dict[str, Any], external_id: str) -> str:
|
||||
movie_id = await run_movie_ingestion(external_id)
|
||||
async def ingest_movie_task(ctx: dict[str, Any], job_id: str, external_id: str) -> str:
|
||||
movie_id = await run_movie_ingestion(uuid.UUID(job_id), external_id)
|
||||
return str(movie_id)
|
||||
|
||||
|
||||
async def ingest_person_task(ctx: dict[str, Any], external_id: str) -> str:
|
||||
person_id = await run_person_ingestion(external_id)
|
||||
async def ingest_person_task(ctx: dict[str, Any], job_id: str, external_id: str) -> str:
|
||||
person_id = await run_person_ingestion(uuid.UUID(job_id), external_id)
|
||||
return str(person_id)
|
||||
|
||||
Reference in New Issue
Block a user