NixReaperandClaude Sonnet 5
a52ac676ff
CI / lint (push) Canceled after 0s
CI / typecheck (push) Canceled after 0s
CI / unit-tests (push) Canceled after 0s
CI / integration-tests (push) Canceled after 0s
CI / container-build (push) Canceled after 0s
CI / dependency-audit (push) Canceled after 0s
app/services/bulk_import.py fetches and parses TMDb's daily ID export files (published once a day specifically for full-catalog sync, unlike /discover which caps at 10,000 results per query) -- today's file with a fallback to yesterday's, since TMDb publishes at a fixed time and "today's" may not exist yet. app/scripts/bulk_import.py is a standalone ops script (not an arq task, since enqueueing millions of jobs would blow past arq's job_timeout) that fans every id from the export into the existing per-entity ingestion pipeline -- get_or_create_ingestion_job + the same arq task each admin endpoint already uses, so a single id's behavior is identical whether it came from the admin API or this script. Bounded concurrency (50) on the enqueue phase; idempotency-keyed jobs make interrupting and re-running the script safe. Real TMDb catalog size (checked live): ~1.24M movies, 231K series, 4.9M people, 9.8K collections -- ~6.4M entities. At default worker concurrency this is realistically weeks of continuous processing, not hours, documented as such in README.md. Also added 429 (rate-limited) handling to TMDbProvider before running this at volume: previously a 429 fell into the generic >=400 branch and failed the request outright with zero retry, since only httpx.TransportError triggered the existing retry decorator. Now a 429 raises ProviderRateLimitedError carrying the response's own Retry-After hint, and tenacity's wait strategy honors that hint directly instead of guessing via exponential backoff. Co-Authored-By: Claude Sonnet 5 <[email protected]>
75 lines
2.8 KiB
Python
75 lines
2.8 KiB
Python
"""Provider abstraction (section 13).
|
|
|
|
A `MetadataProvider` fetches **raw**, unnormalized data from one external
|
|
source. It never writes to the database and never returns Aurora entities —
|
|
that's the normalizer/matcher/ingestion pipeline's job (see
|
|
`app/services/metadata/` and `app/services/ingestion/`). This keeps adapters
|
|
replaceable: swapping TMDb for another provider means writing a new class
|
|
here, not touching ingestion logic.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from abc import ABC, abstractmethod
|
|
from typing import Any
|
|
|
|
|
|
class ProviderError(Exception):
|
|
"""Raised for any provider-side failure (auth, not-found, rate limit,
|
|
network). Ingestion jobs catch this and record it, they don't crash."""
|
|
|
|
|
|
class ProviderNotConfiguredError(ProviderError):
|
|
"""Raised when a required credential (e.g. TMDB_API_KEY) is missing."""
|
|
|
|
|
|
class ProviderNotFoundError(ProviderError):
|
|
"""Raised when the provider has no record for the given external ID."""
|
|
|
|
|
|
class ProviderRateLimitedError(ProviderError):
|
|
"""Raised when the provider responds 429. Carries how long the provider
|
|
itself asked callers to wait (`Retry-After`, seconds) so a caller can
|
|
back off by the right amount rather than guessing — see
|
|
`app/providers/tmdb.py:TMDbProvider._get` for where this is retried."""
|
|
|
|
def __init__(self, message: str, *, retry_after: float) -> None:
|
|
super().__init__(message)
|
|
self.retry_after = retry_after
|
|
|
|
|
|
class MetadataProvider(ABC):
|
|
slug: str
|
|
name: str
|
|
|
|
@abstractmethod
|
|
async def get_movie(self, external_id: str) -> dict[str, Any]:
|
|
"""Fetch raw movie data, including credits/images/videos/keywords/
|
|
alternate titles/external IDs/release dates where the provider
|
|
supports fetching them in one call."""
|
|
|
|
@abstractmethod
|
|
async def get_series(self, external_id: str) -> dict[str, Any]:
|
|
"""Fetch raw series data (without season/episode bodies)."""
|
|
|
|
@abstractmethod
|
|
async def get_season(self, series_external_id: str, season_number: int) -> dict[str, Any]:
|
|
"""Fetch raw season data, including its episodes."""
|
|
|
|
@abstractmethod
|
|
async def get_person(self, external_id: str) -> dict[str, Any]:
|
|
"""Fetch raw person data, including external IDs and images."""
|
|
|
|
@abstractmethod
|
|
async def get_collection(self, external_id: str) -> dict[str, Any]:
|
|
"""Fetch raw collection/franchise data: name, overview, poster/
|
|
backdrop, and its member movies (TMDb's "parts")."""
|
|
|
|
@abstractmethod
|
|
async def search_movie(self, query: str) -> list[dict[str, Any]]:
|
|
"""Search this provider's catalog for movies matching `query`. Used
|
|
to seed ingestion, not exposed directly to API clients."""
|
|
|
|
async def aclose(self) -> None:
|
|
"""Release any held resources (HTTP clients, ...). Optional."""
|