2 Commits

Include renames

Author	SHA1	Message	Date
NixReaperandClaude Sonnet 5	b05d34db5f	Fix bulk_import.py stalling on real-scale exports (1M+ ids) ... asyncio.gather(*(_worker(id) for id in ids)) eagerly wraps every argument into a Task before awaiting any of them -- with a semaphore bounding concurrent *execution* but not concurrent Task *creation*, running this against the real movie export (1,240,787 ids) meant constructing 1.24 million live Task + coroutine objects up front, which looked indistinguishable from hanging for minutes before any work could start, confirmed live in production tonight. Replaced with a fixed pool of _CONCURRENCY (50) consumer tasks pulling from a plain asyncio.Queue pre-loaded with every id -- only 50 Task objects ever exist regardless of catalog size. Verified: 500K enqueues against no-op stubs now complete in 0.59s (previously: the setup loop alone dominated), and the real bottleneck is correctly back to the per-id DB round-trip + Redis push, gated by the pool of 50 -- not Python-level Task scheduling overhead. Co-Authored-By: Claude Sonnet 5 <[email protected]>	2026-09-05 23:08:37 -05:00
NixReaperandClaude Sonnet 5	a52ac676ff	Add full-catalog bulk import from TMDb's daily ID exports ... app/services/bulk_import.py fetches and parses TMDb's daily ID export files (published once a day specifically for full-catalog sync, unlike /discover which caps at 10,000 results per query) -- today's file with a fallback to yesterday's, since TMDb publishes at a fixed time and "today's" may not exist yet. app/scripts/bulk_import.py is a standalone ops script (not an arq task, since enqueueing millions of jobs would blow past arq's job_timeout) that fans every id from the export into the existing per-entity ingestion pipeline -- get_or_create_ingestion_job + the same arq task each admin endpoint already uses, so a single id's behavior is identical whether it came from the admin API or this script. Bounded concurrency (50) on the enqueue phase; idempotency-keyed jobs make interrupting and re-running the script safe. Real TMDb catalog size (checked live): ~1.24M movies, 231K series, 4.9M people, 9.8K collections -- ~6.4M entities. At default worker concurrency this is realistically weeks of continuous processing, not hours, documented as such in README.md. Also added 429 (rate-limited) handling to TMDbProvider before running this at volume: previously a 429 fell into the generic >=400 branch and failed the request outright with zero retry, since only httpx.TransportError triggered the existing retry decorator. Now a 429 raises ProviderRateLimitedError carrying the response's own Retry-After hint, and tenacity's wait strategy honors that hint directly instead of guessing via exponential backoff. Co-Authored-By: Claude Sonnet 5 <[email protected]>	2026-09-05 22:12:12 -05:00