Fix get_or_create_provider race crashing concurrent bulk import workers
CI / lint (push) Canceled after 0s
CI / typecheck (push) Canceled after 0s
CI / unit-tests (push) Canceled after 0s
CI / integration-tests (push) Canceled after 0s
CI / container-build (push) Canceled after 0s
CI / dependency-audit (push) Canceled after 0s
CI / lint (push) Canceled after 0s
CI / typecheck (push) Canceled after 0s
CI / unit-tests (push) Canceled after 0s
CI / integration-tests (push) Canceled after 0s
CI / container-build (push) Canceled after 0s
CI / dependency-audit (push) Canceled after 0s
Found live: running bulk_import.py's fixed 50-worker pool against the real movie export, 49 of the first 50 concurrent workers crashed immediately with "duplicate key value violates unique constraint uq_providers_slug" -- all racing to create the one-and-only "tmdb" Provider row on first run. get_or_create_provider was a plain check-then-insert with no race recovery, unlike its sibling get_or_create_* helpers in reference_repository.py, which already handle exactly this class of race (arq's own concurrency and season- ingestion cascades already exercise it there). Extracted the SAVEPOINT-based create_or_recover helper out of reference_repository.py into app/repositories/concurrency.py so both modules share it, and applied it to get_or_create_provider and create_ingestion_job (the latter's idempotency-key race is less likely under bulk_import's own access pattern -- each of its workers processes a distinct external_id -- but a concurrent admin-triggered ingestion of the same id during the weeks this bulk import runs is a realistic way to hit it too). reference_repository.py re-exports the helper under its old private name so its 8 existing call sites and test_reference_repository.py need no changes. Co-Authored-By: Claude Sonnet 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
b05d34db5f
commit
71388e254e
@@ -0,0 +1,44 @@
|
||||
"""Integration test for provider_repository.get_or_create_provider's
|
||||
concurrency safety -- found broken live: 50 concurrent bulk-import workers
|
||||
(app/scripts/bulk_import.py) all racing to create the one-and-only "tmdb"
|
||||
Provider row on first run crashed 49 of them with a uq_providers_slug
|
||||
violation, since it was a plain check-then-insert with no race recovery
|
||||
(unlike its sibling get_or_create_* helpers in reference_repository.py,
|
||||
which already used create_or_recover for exactly this reason -- now moved
|
||||
to app/repositories/concurrency.py and shared by both).
|
||||
|
||||
Needs genuinely separate DB connections/transactions to reproduce -- the
|
||||
db_session fixture's single connection/transaction can't race against
|
||||
itself, so this uses session_scope() per worker (matching
|
||||
test_ingestion_retry.py), each a real, independent connection."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import uuid
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import delete
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.core.database import session_scope
|
||||
from app.models.provider import Provider
|
||||
from app.repositories.provider_repository import get_or_create_provider
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_or_create_provider_survives_concurrent_races(db_session: AsyncSession):
|
||||
# db_session is unused directly but keeps the Postgres-reachability
|
||||
# skip guard (see tests/conftest.py), matching test_ingestion_retry.py.
|
||||
slug = f"race-test-{uuid.uuid4().hex[:8]}"
|
||||
|
||||
async def _worker() -> Provider:
|
||||
async with session_scope() as session:
|
||||
return await get_or_create_provider(session, slug, "Race Test Provider")
|
||||
|
||||
try:
|
||||
results = await asyncio.gather(*(_worker() for _ in range(20)))
|
||||
assert len({provider.id for provider in results}) == 1
|
||||
finally:
|
||||
async with session_scope() as session:
|
||||
await session.execute(delete(Provider).where(Provider.slug == slug))
|
||||
Reference in New Issue
Block a user