Make failed ingestion jobs actually retry, not just say "retrying"

arq only auto-retries a job on its own Retry exception (or
CancelledError) -- never on an arbitrary exception a task raises.
_run_ingestion caught failures, recorded them via mark_job_failed
(which sets ingestion_jobs.status to "retrying" when attempts hasn't
hit max_attempts), and then just re-raised the original exception --
which arq treats as a terminal failure. The DB said "retrying"
forever; nothing ever put the job back on the queue.

Confirmed live: cascading Better Call Saul's 6 seasons hit the
get_or_create race fixed in the previous commit on 4 of them, and the
worker's own shutdown summary showed "4 failed, 0 retries" -- every
one of those was permanently lost despite max_attempts=5 suggesting
otherwise.

_run_ingestion now raises arq.Retry (capped exponential backoff) when
mark_job_failed left the job in RETRYING state, so arq actually
requeues it; only once attempts are exhausted does the original
exception propagate as a genuine terminal failure.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

app/services/ingestion/runner.py

@@ -10,10 +10,12 @@ from __future__ import annotations
 import uuid
 from collections.abc import Awaitable, Callable
 
+from arq import Retry
 from sqlalchemy.ext.asyncio import AsyncSession
 
 from app.core.database import session_scope
 from app.core.logging import get_logger
+from app.models.enums import IngestionJobStatus
 from app.models.provider import IngestionJob
 from app.providers.base import MetadataProvider
 from app.providers.tmdb import TMDbProvider
@@ -94,11 +96,24 @@ async def _run_ingestion(
         async with session_scope() as session:
             job = await _require_job(session, job_id)
             await mark_job_failed(session, job, str(exc))
+            attempts = job.attempts
+            still_retryable = job.status == IngestionJobStatus.RETRYING
+        if still_retryable:
+            # arq only auto-retries on its own Retry (or CancelledError),
+            # never on an arbitrary exception — mark_job_failed setting the
+            # DB status to RETRYING was, on its own, purely cosmetic; this
+            # is what actually gets the job back on the queue instead of
+            # leaving it permanently "retrying" without ever being retried.
+            raise Retry(defer=_retry_backoff_seconds(attempts)) from exc
         raise
     finally:
         await provider.aclose()
 
 
+def _retry_backoff_seconds(attempt: int) -> float:
+    return min(60.0, 2.0**attempt)
+
+
 async def run_movie_ingestion(job_id: uuid.UUID, external_id: str) -> uuid.UUID:
     return await _run_ingestion(
         job_id,