Make failed ingestion jobs actually retry, not just say "retrying"

arq only auto-retries a job on its own Retry exception (or
CancelledError) -- never on an arbitrary exception a task raises.
_run_ingestion caught failures, recorded them via mark_job_failed
(which sets ingestion_jobs.status to "retrying" when attempts hasn't
hit max_attempts), and then just re-raised the original exception --
which arq treats as a terminal failure. The DB said "retrying"
forever; nothing ever put the job back on the queue.

Confirmed live: cascading Better Call Saul's 6 seasons hit the
get_or_create race fixed in the previous commit on 4 of them, and the
worker's own shutdown summary showed "4 failed, 0 retries" -- every
one of those was permanently lost despite max_attempts=5 suggesting
otherwise.

_run_ingestion now raises arq.Retry (capped exponential backoff) when
mark_job_failed left the job in RETRYING state, so arq actually
requeues it; only once attempts are exhausted does the original
exception propagate as a genuine terminal failure.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

docs/architecture.md

@@ -76,6 +76,20 @@ lifecycle in `ingestion_jobs` (idempotent via a unique `idempotency_key` —
 re-triggering ingestion for the same provider + external ID reuses the same
 job record rather than double-processing).
 
+Failures actually get retried, not just labeled retryable: arq only
+auto-retries a job on its own `arq.Retry` exception (or `CancelledError`) —
+never on an arbitrary exception a task raises — so `_run_ingestion` catches
+the failure, records it, and if `ingestion_jobs.attempts` hasn't hit
+`max_attempts` yet, raises `Retry(defer=...)` (capped exponential backoff)
+so arq actually puts the job back on the queue instead of the DB saying
+"retrying" while nothing retries it. Series ingestion cascading into
+concurrent season jobs (`include_seasons`) is exactly what surfaces
+transient failures worth retrying this way: two seasons sharing a guest
+star can race to create the same `Person` row (see
+`app/repositories/reference_repository.py:_create_or_recover`, which
+recovers from that specific race directly; the arq retry here is the
+backstop for whatever it doesn't).
+
 ## Search (section 16)
 
 `app/services/search/base.py:SearchProvider` is the abstraction; API