Add artwork self-hosting: local-disk download worker + storage
CI / lint (push) Canceled after 0s
CI / typecheck (push) Canceled after 0s
CI / unit-tests (push) Canceled after 0s
CI / integration-tests (push) Canceled after 0s
CI / container-build (push) Canceled after 0s
CI / dependency-audit (push) Canceled after 0s
CI / lint (push) Canceled after 0s
CI / typecheck (push) Canceled after 0s
CI / unit-tests (push) Canceled after 0s
CI / integration-tests (push) Canceled after 0s
CI / container-build (push) Canceled after 0s
CI / dependency-audit (push) Canceled after 0s
Section 10: every image API responses reference has pointed straight at TMDb's CDN since day one -- Aurora never actually downloaded or served artwork itself, despite storing dimensions/checksum/local_path columns for exactly that. Closes the gap: - is_primary was declared on the Artwork model but never actually set anywhere -- every row defaulted to False. Now the normalizer marks the one image each provider designates as *the* image for an entity (TMDb's poster_path/backdrop_path/profile_path) as primary, distinct from the dozens of alternates also ingested, including correctly marking an existing row primary (not inserting a duplicate) when that image already appears in the alternates list. - app/services/artwork/: an ArtworkStorage interface with a LocalDiskStorage implementation (checksum-named files, single-host deployment via a Docker volume shared between api and worker -- see docs/architecture.md for the S3/R2 upgrade path), and a download pipeline with content-type validation, a size cap, and path-traversal protection. - After movie/series/person ingestion succeeds, tasks.py cascades one download_artwork_task per primary artwork id via arq's ctx['redis'] (same fan-out pattern as the season cascade) -- fire-and-forget, not tracked in ingestion_jobs, since a failed image cache is a known-safe fallback to the provider URL, not a user-facing failure. - app/main.py serves MEDIA_ROOT at MEDIA_URL_PREFIX; local_path is preferred over original_url once set (already-existing fallback logic in app/services/common.py:artwork_url, now actually exercised). Also fixes a real interaction bug this surfaced: every ingestion pipeline deleted and recreated all of an entity's Artwork rows on each re-sync, which would have silently wiped local_path/checksum on every re-ingestion, forcing a full re-download every time and defeating the point of caching locally at all. app/repositories/polymorphic.py's new sync_artwork upserts by provider_asset_id (TMDb's stable per-image path) instead, preserving downloaded copies across re-sync, and replaces three near-identical delete+recreate helpers that used to live separately in movie_ingestion.py, series_ingestion.py, and person_ingestion.py. Co-Authored-By: Claude Sonnet 5 <[email protected]>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
cd4dd98adb
commit
fef914db36
@@ -42,3 +42,10 @@ CACHE_DEFAULT_TTL=300
|
|||||||
# --- Rate limiting ---
|
# --- Rate limiting ---
|
||||||
RATE_LIMIT_ENABLED=true
|
RATE_LIMIT_ENABLED=true
|
||||||
RATE_LIMIT_PER_MINUTE=120
|
RATE_LIMIT_PER_MINUTE=120
|
||||||
|
|
||||||
|
# --- Artwork storage ---
|
||||||
|
# Local disk, shared between the api and worker containers via a volume
|
||||||
|
# (see docker-compose.yml) -- the worker downloads here, the api serves it.
|
||||||
|
MEDIA_ROOT=./media
|
||||||
|
MEDIA_URL_PREFIX=/media
|
||||||
|
ARTWORK_DOWNLOAD_MAX_BYTES=15728640
|
||||||
|
|||||||
@@ -103,6 +103,22 @@ curl http://localhost:8000/api/v1/admin/ingest/jobs/<job_id> \
|
|||||||
curl "http://localhost:8000/api/v1/search?q=the%20matrix"
|
curl "http://localhost:8000/api/v1/search?q=the%20matrix"
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Ingesting a series also queues its posters/backdrops/profile images for
|
||||||
|
local download automatically (only the provider's *primary* image per
|
||||||
|
entity, not every alternate) — see the next section.
|
||||||
|
|
||||||
|
## Artwork
|
||||||
|
|
||||||
|
Images are referenced by their provider CDN URL (TMDb) from the moment
|
||||||
|
they're ingested, and cached locally in the background afterward — API
|
||||||
|
responses point at the provider until a local copy exists, then switch over
|
||||||
|
automatically, with no client-visible change. Cached files live under
|
||||||
|
`MEDIA_ROOT` (a Docker volume shared between `api` and `worker`, see
|
||||||
|
`docker-compose.yml`) and are served at `MEDIA_URL_PREFIX` (default
|
||||||
|
`/media`). See [`docs/architecture.md`](docs/architecture.md#artwork-self-hosting-section-10)
|
||||||
|
and [`docs/database.md`](docs/database.md#artwork-storage-self-hosting) for
|
||||||
|
how it works and its current single-host-deployment assumption.
|
||||||
|
|
||||||
## Tests
|
## Tests
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@@ -170,14 +186,8 @@ See [`docs/database.md#known-limitations`](docs/database.md#known-limitations)
|
|||||||
and [`docs/api.md#known-limitations`](docs/api.md#known-limitations) for the
|
and [`docs/api.md#known-limitations`](docs/api.md#known-limitations) for the
|
||||||
current, tracked list (no curated language/country seed data, no fuzzy
|
current, tracked list (no curated language/country seed data, no fuzzy
|
||||||
entity matching, no field-level provenance, no pagination beyond search,
|
entity matching, no field-level provenance, no pagination beyond search,
|
||||||
person credits limited to movies). None of these block the Core 0.1
|
person credits' episode-level guest appearances rolled up to a per-series
|
||||||
milestone (section 34): search → movie/person detail works end-to-end.
|
count rather than broken out individually). None of these block the Core
|
||||||
|
0.1 milestone (section 34): search → movie/series/person detail works
|
||||||
This repository was built in an environment without Docker or a
|
end-to-end against real TMDb data, verified on a real PostgreSQL + Redis +
|
||||||
pre-existing PostgreSQL instance; the initial Alembic migration was
|
Docker deployment (not just this repo's own dev machine).
|
||||||
hand-written from the models (not autogenerated against a live database)
|
|
||||||
and validated with `alembic history`, but has not yet been executed against
|
|
||||||
a running PostgreSQL instance. **Run `docker compose up --build` and confirm
|
|
||||||
`alembic upgrade head` succeeds against a genuinely empty database before
|
|
||||||
treating the schema as final** — see
|
|
||||||
[`docs/database.md#migrations`](docs/database.md#migrations).
|
|
||||||
|
|||||||
@@ -65,6 +65,20 @@ class Settings(BaseSettings):
|
|||||||
RATE_LIMIT_ENABLED: bool = Field(default=True)
|
RATE_LIMIT_ENABLED: bool = Field(default=True)
|
||||||
RATE_LIMIT_PER_MINUTE: int = Field(default=120)
|
RATE_LIMIT_PER_MINUTE: int = Field(default=120)
|
||||||
|
|
||||||
|
# --- Artwork storage (section 10) ---
|
||||||
|
# Local-disk self-hosting, deliberately simple for 0.1: fine for a
|
||||||
|
# single-host deployment where the api and worker containers share one
|
||||||
|
# volume (see docker-compose.yml). A genuinely multi-host deployment
|
||||||
|
# would need real object storage (S3/R2/...) behind the same
|
||||||
|
# ArtworkStorage interface instead — see app/services/artwork/storage.py.
|
||||||
|
MEDIA_ROOT: str = Field(default="./media")
|
||||||
|
# Root-relative URL path artwork is served under (app/main.py mounts
|
||||||
|
# MEDIA_ROOT there). Kept relative rather than a full domain so it keeps
|
||||||
|
# working whether Aurora is reached via localhost, a Traefik host, or
|
||||||
|
# (once configured) amdbs.net.
|
||||||
|
MEDIA_URL_PREFIX: str = Field(default="/media")
|
||||||
|
ARTWORK_DOWNLOAD_MAX_BYTES: int = Field(default=15 * 1024 * 1024)
|
||||||
|
|
||||||
@field_validator("AURORA_ENV")
|
@field_validator("AURORA_ENV")
|
||||||
@classmethod
|
@classmethod
|
||||||
def _validate_env(cls, v: str) -> str:
|
def _validate_env(cls, v: str) -> str:
|
||||||
|
|||||||
+13
@@ -10,11 +10,13 @@ import time
|
|||||||
import uuid
|
import uuid
|
||||||
from collections.abc import Awaitable, Callable
|
from collections.abc import Awaitable, Callable
|
||||||
from contextlib import asynccontextmanager
|
from contextlib import asynccontextmanager
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
from fastapi import FastAPI, Request, Response
|
from fastapi import FastAPI, Request, Response
|
||||||
from fastapi.exceptions import RequestValidationError
|
from fastapi.exceptions import RequestValidationError
|
||||||
from fastapi.middleware.cors import CORSMiddleware
|
from fastapi.middleware.cors import CORSMiddleware
|
||||||
from fastapi.responses import JSONResponse
|
from fastapi.responses import JSONResponse
|
||||||
|
from fastapi.staticfiles import StaticFiles
|
||||||
from prometheus_client import CONTENT_TYPE_LATEST, Counter, Histogram, generate_latest
|
from prometheus_client import CONTENT_TYPE_LATEST, Counter, Histogram, generate_latest
|
||||||
from starlette.exceptions import HTTPException as StarletteHTTPException
|
from starlette.exceptions import HTTPException as StarletteHTTPException
|
||||||
|
|
||||||
@@ -78,6 +80,17 @@ app.add_middleware(
|
|||||||
allow_headers=["*"],
|
allow_headers=["*"],
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# Serves what app/services/artwork/ downloads (section 10: self-hosted
|
||||||
|
# artwork). The directory must exist before StaticFiles will mount — the
|
||||||
|
# worker creates subdirectories as it writes files, but the root itself
|
||||||
|
# needs to be there even before the first download.
|
||||||
|
Path(settings.MEDIA_ROOT).mkdir(parents=True, exist_ok=True)
|
||||||
|
app.mount(
|
||||||
|
settings.MEDIA_URL_PREFIX,
|
||||||
|
StaticFiles(directory=settings.MEDIA_ROOT),
|
||||||
|
name="media",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
@app.middleware("http")
|
@app.middleware("http")
|
||||||
async def request_context_middleware(
|
async def request_context_middleware(
|
||||||
|
|||||||
@@ -6,6 +6,8 @@ an (entity_type, entity_id) pair rather than a typed foreign key.
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import uuid
|
import uuid
|
||||||
|
from collections.abc import Sequence
|
||||||
|
from typing import Protocol
|
||||||
|
|
||||||
from sqlalchemy import select
|
from sqlalchemy import select
|
||||||
from sqlalchemy.ext.asyncio import AsyncSession
|
from sqlalchemy.ext.asyncio import AsyncSession
|
||||||
@@ -18,6 +20,24 @@ from app.models.external_id import ExternalID
|
|||||||
from app.models.provider import Provider
|
from app.models.provider import Provider
|
||||||
from app.models.video import Video
|
from app.models.video import Video
|
||||||
|
|
||||||
|
_VALID_ARTWORK_TYPES = {t.value for t in ArtworkType}
|
||||||
|
|
||||||
|
|
||||||
|
class _NormalizedArtworkLike(Protocol):
|
||||||
|
"""Structural type for app.services.metadata.normalizer.NormalizedArtwork
|
||||||
|
— a repository shouldn't import from the services layer, so this
|
||||||
|
describes the shape `sync_artwork` needs instead of the concrete class.
|
||||||
|
"""
|
||||||
|
|
||||||
|
artwork_type: str
|
||||||
|
url: str
|
||||||
|
provider_asset_id: str | None
|
||||||
|
width: int | None
|
||||||
|
height: int | None
|
||||||
|
aspect_ratio: float | None
|
||||||
|
vote_average: float | None
|
||||||
|
is_primary: bool
|
||||||
|
|
||||||
|
|
||||||
async def get_external_ids(
|
async def get_external_ids(
|
||||||
session: AsyncSession, entity_type: str, entity_id: uuid.UUID
|
session: AsyncSession, entity_type: str, entity_id: uuid.UUID
|
||||||
@@ -48,6 +68,104 @@ async def get_external_ids_bulk(
|
|||||||
return out
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
async def get_primary_artwork_ids(
|
||||||
|
session: AsyncSession, entity_type: str, entity_ids: list[uuid.UUID]
|
||||||
|
) -> list[uuid.UUID]:
|
||||||
|
"""IDs of artwork rows marked `is_primary` for any of `entity_ids` — the
|
||||||
|
small subset of a title's images (TMDb's own selected poster/backdrop/
|
||||||
|
profile) that get auto-downloaded for local hosting, out of the dozens
|
||||||
|
of alternates ingestion also stores. See app/services/artwork/."""
|
||||||
|
|
||||||
|
if not entity_ids:
|
||||||
|
return []
|
||||||
|
stmt = select(Artwork.id).where(
|
||||||
|
Artwork.entity_type == entity_type,
|
||||||
|
Artwork.entity_id.in_(entity_ids),
|
||||||
|
Artwork.is_primary.is_(True),
|
||||||
|
)
|
||||||
|
result = await session.execute(stmt)
|
||||||
|
return list(result.scalars().all())
|
||||||
|
|
||||||
|
|
||||||
|
async def sync_artwork(
|
||||||
|
session: AsyncSession,
|
||||||
|
*,
|
||||||
|
entity_type: str,
|
||||||
|
entity_id: uuid.UUID,
|
||||||
|
provider_id: uuid.UUID,
|
||||||
|
artwork_list: Sequence[_NormalizedArtworkLike],
|
||||||
|
) -> None:
|
||||||
|
"""Upsert an entity's artwork rows keyed by `provider_asset_id` (TMDb's
|
||||||
|
stable per-image file path), instead of deleting and recreating them on
|
||||||
|
every re-sync.
|
||||||
|
|
||||||
|
This matters specifically because of `app/services/artwork/` (section
|
||||||
|
10's self-hosting): a delete-then-recreate replace would generate a
|
||||||
|
fresh Artwork.id with `local_path=None` every time a title gets
|
||||||
|
re-ingested, silently throwing away every previously-downloaded local
|
||||||
|
copy and forcing a full re-download on each sync — defeating the point
|
||||||
|
of caching them locally at all. Rows for images no longer present in
|
||||||
|
`artwork_list` (dropped from the provider's alternates) are still
|
||||||
|
deleted, same as before.
|
||||||
|
"""
|
||||||
|
|
||||||
|
existing = (
|
||||||
|
(
|
||||||
|
await session.execute(
|
||||||
|
select(Artwork).where(
|
||||||
|
Artwork.entity_type == entity_type, Artwork.entity_id == entity_id
|
||||||
|
)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
.scalars()
|
||||||
|
.all()
|
||||||
|
)
|
||||||
|
existing_by_asset_id = {
|
||||||
|
row.provider_asset_id: row for row in existing if row.provider_asset_id is not None
|
||||||
|
}
|
||||||
|
|
||||||
|
seen_asset_ids: set[str] = set()
|
||||||
|
for art in artwork_list:
|
||||||
|
art_type = art.artwork_type if art.artwork_type in _VALID_ARTWORK_TYPES else "poster"
|
||||||
|
matched = (
|
||||||
|
existing_by_asset_id.get(art.provider_asset_id) if art.provider_asset_id else None
|
||||||
|
)
|
||||||
|
if matched is not None:
|
||||||
|
seen_asset_ids.add(art.provider_asset_id) # type: ignore[arg-type]
|
||||||
|
matched.artwork_type = ArtworkType(art_type)
|
||||||
|
matched.width = art.width
|
||||||
|
matched.height = art.height
|
||||||
|
matched.aspect_ratio = art.aspect_ratio
|
||||||
|
matched.vote_average = art.vote_average
|
||||||
|
matched.is_primary = art.is_primary
|
||||||
|
matched.original_url = art.url
|
||||||
|
# local_path/checksum deliberately untouched — see docstring.
|
||||||
|
else:
|
||||||
|
session.add(
|
||||||
|
Artwork(
|
||||||
|
entity_type=entity_type,
|
||||||
|
entity_id=entity_id,
|
||||||
|
artwork_type=ArtworkType(art_type),
|
||||||
|
width=art.width,
|
||||||
|
height=art.height,
|
||||||
|
aspect_ratio=art.aspect_ratio,
|
||||||
|
provider_id=provider_id,
|
||||||
|
provider_asset_id=art.provider_asset_id,
|
||||||
|
original_url=art.url,
|
||||||
|
vote_average=art.vote_average,
|
||||||
|
is_primary=art.is_primary,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if art.provider_asset_id:
|
||||||
|
seen_asset_ids.add(art.provider_asset_id)
|
||||||
|
|
||||||
|
for row in existing:
|
||||||
|
if row.provider_asset_id not in seen_asset_ids:
|
||||||
|
await session.delete(row)
|
||||||
|
|
||||||
|
await session.flush()
|
||||||
|
|
||||||
|
|
||||||
async def get_artwork(
|
async def get_artwork(
|
||||||
session: AsyncSession,
|
session: AsyncSession,
|
||||||
entity_type: str,
|
entity_type: str,
|
||||||
|
|||||||
@@ -0,0 +1,110 @@
|
|||||||
|
"""Downloads one Artwork row's original_url and stores it locally
|
||||||
|
(section 10; the "Artwork processing" background worker from section 1's
|
||||||
|
architecture diagram).
|
||||||
|
|
||||||
|
Deliberately narrow: only ever called for artwork already marked
|
||||||
|
`is_primary` (see `app/services/ingestion/runner.py:get_primary_artwork_ids`
|
||||||
|
and `app/workers/tasks.py`, which cascade a download job per primary image
|
||||||
|
after each ingestion succeeds) — TMDb returns dozens of backdrop/logo
|
||||||
|
alternates per title, and downloading every one of them automatically would
|
||||||
|
be a lot of unnecessary storage and bandwidth for images Aurora's own
|
||||||
|
clients are unlikely to ever show. Downloading on request for anything
|
||||||
|
beyond the primary image is a reasonable follow-up if that's ever needed.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import uuid
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
from sqlalchemy.ext.asyncio import AsyncSession
|
||||||
|
|
||||||
|
from app.core.config import get_settings
|
||||||
|
from app.core.logging import get_logger
|
||||||
|
from app.models.artwork import Artwork
|
||||||
|
from app.services.artwork.storage import ArtworkStorage
|
||||||
|
|
||||||
|
logger = get_logger(__name__)
|
||||||
|
|
||||||
|
_EXTENSION_BY_CONTENT_TYPE = {
|
||||||
|
"image/jpeg": "jpg",
|
||||||
|
"image/jpg": "jpg",
|
||||||
|
"image/png": "png",
|
||||||
|
"image/webp": "webp",
|
||||||
|
"image/gif": "gif",
|
||||||
|
"image/svg+xml": "svg",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class ArtworkDownloadError(Exception):
|
||||||
|
"""Raised for any download-side failure (bad content-type, too large,
|
||||||
|
network error, ...). Callers (the arq task) let this propagate so the
|
||||||
|
job gets retried like any other ingestion failure — see
|
||||||
|
app/services/ingestion/runner.py."""
|
||||||
|
|
||||||
|
|
||||||
|
async def download_artwork(
|
||||||
|
session: AsyncSession,
|
||||||
|
storage: ArtworkStorage,
|
||||||
|
artwork_id: uuid.UUID,
|
||||||
|
*,
|
||||||
|
client: httpx.AsyncClient | None = None,
|
||||||
|
force: bool = False,
|
||||||
|
) -> bool:
|
||||||
|
"""Fetch `artwork.original_url`, store it, and set `local_path` +
|
||||||
|
`checksum`. Returns False (no-op) if the row is gone or already has a
|
||||||
|
local copy (unless `force=True`); True if it downloaded something.
|
||||||
|
"""
|
||||||
|
|
||||||
|
artwork = await session.get(Artwork, artwork_id)
|
||||||
|
if artwork is None:
|
||||||
|
return False
|
||||||
|
if artwork.local_path and not force:
|
||||||
|
return False
|
||||||
|
|
||||||
|
settings = get_settings()
|
||||||
|
owns_client = client is None
|
||||||
|
http_client = client or httpx.AsyncClient(timeout=20.0, follow_redirects=True)
|
||||||
|
try:
|
||||||
|
content, content_type = await _fetch(http_client, artwork.original_url, settings.ARTWORK_DOWNLOAD_MAX_BYTES)
|
||||||
|
finally:
|
||||||
|
if owns_client:
|
||||||
|
await http_client.aclose()
|
||||||
|
|
||||||
|
checksum = hashlib.sha256(content).hexdigest()
|
||||||
|
extension = _EXTENSION_BY_CONTENT_TYPE.get(content_type, "jpg")
|
||||||
|
relative_path = f"{artwork.artwork_type.value}/{checksum[:2]}/{checksum}.{extension}"
|
||||||
|
|
||||||
|
local_url = await storage.save(relative_path, content)
|
||||||
|
|
||||||
|
artwork.local_path = local_url
|
||||||
|
artwork.checksum = checksum
|
||||||
|
await session.flush()
|
||||||
|
logger.info(
|
||||||
|
"artwork_downloaded",
|
||||||
|
artwork_id=str(artwork_id),
|
||||||
|
checksum=checksum,
|
||||||
|
bytes=len(content),
|
||||||
|
)
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
async def _fetch(client: httpx.AsyncClient, url: str, max_bytes: int) -> tuple[bytes, str]:
|
||||||
|
try:
|
||||||
|
async with client.stream("GET", url) as response:
|
||||||
|
response.raise_for_status()
|
||||||
|
content_type = response.headers.get("content-type", "").split(";")[0].strip().lower()
|
||||||
|
if not content_type.startswith("image/"):
|
||||||
|
raise ArtworkDownloadError(f"expected an image, got content-type={content_type!r}")
|
||||||
|
|
||||||
|
body = bytearray()
|
||||||
|
async for chunk in response.aiter_bytes():
|
||||||
|
body.extend(chunk)
|
||||||
|
if len(body) > max_bytes:
|
||||||
|
raise ArtworkDownloadError(
|
||||||
|
f"artwork at {url} exceeds ARTWORK_DOWNLOAD_MAX_BYTES ({max_bytes} bytes)"
|
||||||
|
)
|
||||||
|
return bytes(body), content_type
|
||||||
|
except httpx.HTTPError as exc:
|
||||||
|
raise ArtworkDownloadError(f"failed to download {url}: {exc}") from exc
|
||||||
@@ -0,0 +1,66 @@
|
|||||||
|
"""Artwork storage abstraction (section 10).
|
||||||
|
|
||||||
|
Downloaded artwork bytes need somewhere to live that both the worker
|
||||||
|
(writing) and the API (serving, via app/main.py's static mount) can reach.
|
||||||
|
`ArtworkStorage` is the seam: today there's only `LocalDiskStorage` — fine
|
||||||
|
for a single Docker host where the api and worker containers share one
|
||||||
|
volume (see docker-compose.yml), which is this project's actual deployment
|
||||||
|
shape (section 1: modular monolith, not a multi-host fleet). A deployment
|
||||||
|
that outgrows a single host would swap in an S3/R2-backed implementation
|
||||||
|
behind this same interface without touching the download pipeline that
|
||||||
|
calls it (app/services/artwork/download.py).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from abc import ABC, abstractmethod
|
||||||
|
from functools import lru_cache
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from app.core.config import get_settings
|
||||||
|
|
||||||
|
|
||||||
|
class ArtworkStorage(ABC):
|
||||||
|
@abstractmethod
|
||||||
|
async def save(self, relative_path: str, content: bytes) -> str:
|
||||||
|
"""Persist `content` at `relative_path` (already includes an
|
||||||
|
extension, e.g. "poster/ab/ab34...jpg"). Returns the root-relative
|
||||||
|
URL clients should use to fetch it."""
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
async def exists(self, relative_path: str) -> bool: ...
|
||||||
|
|
||||||
|
|
||||||
|
class LocalDiskStorage(ArtworkStorage):
|
||||||
|
def __init__(self, root: str, url_prefix: str) -> None:
|
||||||
|
self._root = Path(root)
|
||||||
|
self._url_prefix = url_prefix.rstrip("/")
|
||||||
|
|
||||||
|
def _full_path(self, relative_path: str) -> Path:
|
||||||
|
# relative_path is always built by us from a sha256 checksum, never
|
||||||
|
# from unsanitized external input — but guard anyway against ever
|
||||||
|
# writing outside MEDIA_ROOT.
|
||||||
|
root = self._root.resolve()
|
||||||
|
full = (self._root / relative_path).resolve()
|
||||||
|
if full != root and root not in full.parents:
|
||||||
|
raise ValueError(f"unsafe artwork path escapes MEDIA_ROOT: {relative_path!r}")
|
||||||
|
return full
|
||||||
|
|
||||||
|
async def save(self, relative_path: str, content: bytes) -> str:
|
||||||
|
full = self._full_path(relative_path)
|
||||||
|
full.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
# Blocking I/O, but artwork files are small (capped by
|
||||||
|
# ARTWORK_DOWNLOAD_MAX_BYTES) and this only ever runs inside the
|
||||||
|
# worker process, not a request handler — not worth wiring aiofiles
|
||||||
|
# for.
|
||||||
|
full.write_bytes(content)
|
||||||
|
return f"{self._url_prefix}/{relative_path}"
|
||||||
|
|
||||||
|
async def exists(self, relative_path: str) -> bool:
|
||||||
|
return self._full_path(relative_path).is_file()
|
||||||
|
|
||||||
|
|
||||||
|
@lru_cache
|
||||||
|
def get_artwork_storage() -> ArtworkStorage:
|
||||||
|
settings = get_settings()
|
||||||
|
return LocalDiskStorage(settings.MEDIA_ROOT, settings.MEDIA_URL_PREFIX)
|
||||||
@@ -20,15 +20,15 @@ from sqlalchemy import delete
|
|||||||
from sqlalchemy.ext.asyncio import AsyncSession
|
from sqlalchemy.ext.asyncio import AsyncSession
|
||||||
|
|
||||||
from app.core.logging import get_logger
|
from app.core.logging import get_logger
|
||||||
from app.models.artwork import Artwork
|
|
||||||
from app.models.company import MovieCompany
|
from app.models.company import MovieCompany
|
||||||
from app.models.credit import Credit
|
from app.models.credit import Credit
|
||||||
from app.models.enums import ArtworkType, MatchConfidence, MovieStatus, VideoType
|
from app.models.enums import MatchConfidence, MovieStatus, VideoType
|
||||||
from app.models.genre import MovieGenre
|
from app.models.genre import MovieGenre
|
||||||
from app.models.keyword import MovieKeyword
|
from app.models.keyword import MovieKeyword
|
||||||
from app.models.movie import Movie, MovieRecommendation
|
from app.models.movie import Movie, MovieRecommendation
|
||||||
from app.models.video import Video
|
from app.models.video import Video
|
||||||
from app.providers.base import MetadataProvider
|
from app.providers.base import MetadataProvider
|
||||||
|
from app.repositories import polymorphic
|
||||||
from app.repositories import reference_repository as refs
|
from app.repositories import reference_repository as refs
|
||||||
from app.repositories.provider_repository import get_or_create_provider, upsert_provider_record
|
from app.repositories.provider_repository import get_or_create_provider, upsert_provider_record
|
||||||
from app.services.metadata.matcher import match_entity
|
from app.services.metadata.matcher import match_entity
|
||||||
@@ -115,36 +115,9 @@ async def _replace_movie_credits(
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
_ARTWORK_TYPES = {t.value for t in ArtworkType}
|
|
||||||
_VIDEO_TYPES = {t.value for t in VideoType}
|
_VIDEO_TYPES = {t.value for t in VideoType}
|
||||||
|
|
||||||
|
|
||||||
async def _replace_movie_artwork(
|
|
||||||
session: AsyncSession, movie_id: uuid.UUID, provider_id: uuid.UUID, artwork_list
|
|
||||||
) -> None:
|
|
||||||
await session.execute(
|
|
||||||
delete(Artwork).where(Artwork.entity_type == "movie", Artwork.entity_id == movie_id)
|
|
||||||
)
|
|
||||||
for art in artwork_list:
|
|
||||||
art_type = art.artwork_type if art.artwork_type in _ARTWORK_TYPES else "poster"
|
|
||||||
session.add(
|
|
||||||
Artwork(
|
|
||||||
entity_type="movie",
|
|
||||||
entity_id=movie_id,
|
|
||||||
artwork_type=ArtworkType(art_type),
|
|
||||||
language_code=None,
|
|
||||||
width=art.width,
|
|
||||||
height=art.height,
|
|
||||||
aspect_ratio=art.aspect_ratio,
|
|
||||||
provider_id=provider_id,
|
|
||||||
provider_asset_id=art.provider_asset_id,
|
|
||||||
original_url=art.url,
|
|
||||||
vote_average=art.vote_average,
|
|
||||||
is_primary=False,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
async def _replace_movie_videos(
|
async def _replace_movie_videos(
|
||||||
session: AsyncSession, movie_id: uuid.UUID, provider_id: uuid.UUID, videos
|
session: AsyncSession, movie_id: uuid.UUID, provider_id: uuid.UUID, videos
|
||||||
) -> None:
|
) -> None:
|
||||||
@@ -259,7 +232,13 @@ async def ingest_movie(
|
|||||||
await _replace_movie_keywords(session, movie.id, normalized.keywords)
|
await _replace_movie_keywords(session, movie.id, normalized.keywords)
|
||||||
await _replace_movie_companies(session, movie.id, tmdb_provider.id, normalized.companies)
|
await _replace_movie_companies(session, movie.id, tmdb_provider.id, normalized.companies)
|
||||||
await _replace_movie_credits(session, movie.id, tmdb_provider.id, normalized.cast, normalized.crew)
|
await _replace_movie_credits(session, movie.id, tmdb_provider.id, normalized.cast, normalized.crew)
|
||||||
await _replace_movie_artwork(session, movie.id, tmdb_provider.id, normalized.artwork)
|
await polymorphic.sync_artwork(
|
||||||
|
session,
|
||||||
|
entity_type="movie",
|
||||||
|
entity_id=movie.id,
|
||||||
|
provider_id=tmdb_provider.id,
|
||||||
|
artwork_list=normalized.artwork,
|
||||||
|
)
|
||||||
await _replace_movie_videos(session, movie.id, tmdb_provider.id, normalized.videos)
|
await _replace_movie_videos(session, movie.id, tmdb_provider.id, normalized.videos)
|
||||||
await _link_external_ids(session, movie.id, tmdb_provider.id, normalized)
|
await _link_external_ids(session, movie.id, tmdb_provider.id, normalized)
|
||||||
|
|
||||||
|
|||||||
@@ -10,14 +10,13 @@ from __future__ import annotations
|
|||||||
|
|
||||||
import uuid
|
import uuid
|
||||||
|
|
||||||
from sqlalchemy import delete
|
|
||||||
from sqlalchemy.ext.asyncio import AsyncSession
|
from sqlalchemy.ext.asyncio import AsyncSession
|
||||||
|
|
||||||
from app.core.logging import get_logger
|
from app.core.logging import get_logger
|
||||||
from app.models.artwork import Artwork
|
from app.models.enums import MatchConfidence
|
||||||
from app.models.enums import ArtworkType, MatchConfidence
|
|
||||||
from app.models.person import Person
|
from app.models.person import Person
|
||||||
from app.providers.base import MetadataProvider
|
from app.providers.base import MetadataProvider
|
||||||
|
from app.repositories import polymorphic
|
||||||
from app.repositories import reference_repository as refs
|
from app.repositories import reference_repository as refs
|
||||||
from app.repositories.provider_repository import get_or_create_provider, upsert_provider_record
|
from app.repositories.provider_repository import get_or_create_provider, upsert_provider_record
|
||||||
from app.services.metadata.matcher import match_entity
|
from app.services.metadata.matcher import match_entity
|
||||||
@@ -64,24 +63,13 @@ async def ingest_person(
|
|||||||
|
|
||||||
await session.flush()
|
await session.flush()
|
||||||
|
|
||||||
await session.execute(
|
await polymorphic.sync_artwork(
|
||||||
delete(Artwork).where(Artwork.entity_type == "person", Artwork.entity_id == person.id)
|
session,
|
||||||
|
entity_type="person",
|
||||||
|
entity_id=person.id,
|
||||||
|
provider_id=tmdb_provider.id,
|
||||||
|
artwork_list=normalized.artwork,
|
||||||
)
|
)
|
||||||
for art in normalized.artwork:
|
|
||||||
session.add(
|
|
||||||
Artwork(
|
|
||||||
entity_type="person",
|
|
||||||
entity_id=person.id,
|
|
||||||
artwork_type=ArtworkType.PROFILE,
|
|
||||||
width=art.width,
|
|
||||||
height=art.height,
|
|
||||||
aspect_ratio=art.aspect_ratio,
|
|
||||||
provider_id=tmdb_provider.id,
|
|
||||||
provider_asset_id=art.provider_asset_id,
|
|
||||||
original_url=art.url,
|
|
||||||
vote_average=art.vote_average,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
await refs.link_external_id(
|
await refs.link_external_id(
|
||||||
session,
|
session,
|
||||||
|
|||||||
@@ -19,6 +19,7 @@ from app.models.enums import IngestionJobStatus
|
|||||||
from app.models.provider import IngestionJob
|
from app.models.provider import IngestionJob
|
||||||
from app.providers.base import MetadataProvider
|
from app.providers.base import MetadataProvider
|
||||||
from app.providers.tmdb import TMDbProvider
|
from app.providers.tmdb import TMDbProvider
|
||||||
|
from app.repositories.polymorphic import get_primary_artwork_ids
|
||||||
from app.repositories.provider_repository import (
|
from app.repositories.provider_repository import (
|
||||||
create_ingestion_job,
|
create_ingestion_job,
|
||||||
get_ingestion_job,
|
get_ingestion_job,
|
||||||
@@ -165,3 +166,22 @@ async def get_series_season_numbers(series_id: uuid.UUID) -> list[int]:
|
|||||||
async with session_scope() as session:
|
async with session_scope() as session:
|
||||||
seasons = await get_seasons_for_series(session, series_id)
|
seasons = await get_seasons_for_series(session, series_id)
|
||||||
return [season.season_number for season in seasons]
|
return [season.season_number for season in seasons]
|
||||||
|
|
||||||
|
|
||||||
|
async def get_series_season_ids(series_id: uuid.UUID) -> list[uuid.UUID]:
|
||||||
|
async with session_scope() as session:
|
||||||
|
seasons = await get_seasons_for_series(session, series_id)
|
||||||
|
return [season.id for season in seasons]
|
||||||
|
|
||||||
|
|
||||||
|
async def get_primary_artwork_for_download(
|
||||||
|
entity_type: str, entity_ids: list[uuid.UUID]
|
||||||
|
) -> list[uuid.UUID]:
|
||||||
|
"""Artwork ids worth auto-downloading for local hosting after an
|
||||||
|
ingestion succeeds — see app/services/artwork/download.py and
|
||||||
|
app/workers/tasks.py, which enqueue one download_artwork_task per id
|
||||||
|
this returns. A short-lived session of its own, same reasoning as
|
||||||
|
get_series_season_numbers above."""
|
||||||
|
|
||||||
|
async with session_scope() as session:
|
||||||
|
return await get_primary_artwork_ids(session, entity_type, entity_ids)
|
||||||
|
|||||||
@@ -20,17 +20,16 @@ from sqlalchemy import delete
|
|||||||
from sqlalchemy.ext.asyncio import AsyncSession
|
from sqlalchemy.ext.asyncio import AsyncSession
|
||||||
|
|
||||||
from app.core.logging import get_logger
|
from app.core.logging import get_logger
|
||||||
from app.models.artwork import Artwork
|
|
||||||
from app.models.company import SeriesCompany
|
from app.models.company import SeriesCompany
|
||||||
from app.models.credit import Credit
|
from app.models.credit import Credit
|
||||||
from app.models.enums import ArtworkType, CompanyType, MatchConfidence, SeriesStatus, VideoType
|
from app.models.enums import CompanyType, MatchConfidence, SeriesStatus, VideoType
|
||||||
from app.models.genre import SeriesGenre
|
from app.models.genre import SeriesGenre
|
||||||
from app.models.keyword import SeriesKeyword
|
from app.models.keyword import SeriesKeyword
|
||||||
from app.models.tv import Series
|
from app.models.tv import Series
|
||||||
from app.models.video import Video
|
from app.models.video import Video
|
||||||
from app.providers.base import MetadataProvider, ProviderNotFoundError
|
from app.providers.base import MetadataProvider, ProviderNotFoundError
|
||||||
|
from app.repositories import polymorphic, tv_repository
|
||||||
from app.repositories import reference_repository as refs
|
from app.repositories import reference_repository as refs
|
||||||
from app.repositories import tv_repository
|
|
||||||
from app.repositories.provider_repository import get_or_create_provider, upsert_provider_record
|
from app.repositories.provider_repository import get_or_create_provider, upsert_provider_record
|
||||||
from app.services.metadata.matcher import match_entity
|
from app.services.metadata.matcher import match_entity
|
||||||
from app.services.metadata.normalizer import (
|
from app.services.metadata.normalizer import (
|
||||||
@@ -41,7 +40,6 @@ from app.services.metadata.normalizer import (
|
|||||||
|
|
||||||
logger = get_logger(__name__)
|
logger = get_logger(__name__)
|
||||||
|
|
||||||
_ARTWORK_TYPES = {t.value for t in ArtworkType}
|
|
||||||
_VIDEO_TYPES = {t.value for t in VideoType}
|
_VIDEO_TYPES = {t.value for t in VideoType}
|
||||||
|
|
||||||
_LEADING_ARTICLES = ("the ", "a ", "an ")
|
_LEADING_ARTICLES = ("the ", "a ", "an ")
|
||||||
@@ -140,30 +138,6 @@ async def _replace_series_credits(
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
async def _replace_entity_artwork(
|
|
||||||
session: AsyncSession, entity_type: str, entity_id: uuid.UUID, provider_id: uuid.UUID, artwork_list
|
|
||||||
) -> None:
|
|
||||||
await session.execute(
|
|
||||||
delete(Artwork).where(Artwork.entity_type == entity_type, Artwork.entity_id == entity_id)
|
|
||||||
)
|
|
||||||
for art in artwork_list:
|
|
||||||
art_type = art.artwork_type if art.artwork_type in _ARTWORK_TYPES else "poster"
|
|
||||||
session.add(
|
|
||||||
Artwork(
|
|
||||||
entity_type=entity_type,
|
|
||||||
entity_id=entity_id,
|
|
||||||
artwork_type=ArtworkType(art_type),
|
|
||||||
width=art.width,
|
|
||||||
height=art.height,
|
|
||||||
aspect_ratio=art.aspect_ratio,
|
|
||||||
provider_id=provider_id,
|
|
||||||
provider_asset_id=art.provider_asset_id,
|
|
||||||
original_url=art.url,
|
|
||||||
vote_average=art.vote_average,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
async def _replace_entity_videos(
|
async def _replace_entity_videos(
|
||||||
session: AsyncSession, entity_type: str, entity_id: uuid.UUID, provider_id: uuid.UUID, videos
|
session: AsyncSession, entity_type: str, entity_id: uuid.UUID, provider_id: uuid.UUID, videos
|
||||||
) -> None:
|
) -> None:
|
||||||
@@ -252,7 +226,13 @@ async def ingest_series(
|
|||||||
await _replace_series_credits(
|
await _replace_series_credits(
|
||||||
session, series.id, tmdb_provider.id, normalized.cast, normalized.crew, normalized.creators
|
session, series.id, tmdb_provider.id, normalized.cast, normalized.crew, normalized.creators
|
||||||
)
|
)
|
||||||
await _replace_entity_artwork(session, "series", series.id, tmdb_provider.id, normalized.artwork)
|
await polymorphic.sync_artwork(
|
||||||
|
session,
|
||||||
|
entity_type="series",
|
||||||
|
entity_id=series.id,
|
||||||
|
provider_id=tmdb_provider.id,
|
||||||
|
artwork_list=normalized.artwork,
|
||||||
|
)
|
||||||
await _replace_entity_videos(session, "series", series.id, tmdb_provider.id, normalized.videos)
|
await _replace_entity_videos(session, "series", series.id, tmdb_provider.id, normalized.videos)
|
||||||
|
|
||||||
for stub in normalized.seasons:
|
for stub in normalized.seasons:
|
||||||
@@ -262,12 +242,12 @@ async def ingest_series(
|
|||||||
season.air_date = stub.air_date
|
season.air_date = stub.air_date
|
||||||
season.episode_count = stub.episode_count
|
season.episode_count = stub.episode_count
|
||||||
await session.flush()
|
await session.flush()
|
||||||
await _replace_entity_artwork(
|
await polymorphic.sync_artwork(
|
||||||
session,
|
session,
|
||||||
"season",
|
entity_type="season",
|
||||||
season.id,
|
entity_id=season.id,
|
||||||
tmdb_provider.id,
|
provider_id=tmdb_provider.id,
|
||||||
[stub.poster] if stub.poster else [],
|
artwork_list=[stub.poster] if stub.poster else [],
|
||||||
)
|
)
|
||||||
|
|
||||||
await refs.link_external_id(
|
await refs.link_external_id(
|
||||||
@@ -373,7 +353,13 @@ async def ingest_season(
|
|||||||
season.episode_count = normalized.episode_count
|
season.episode_count = normalized.episode_count
|
||||||
await session.flush()
|
await session.flush()
|
||||||
|
|
||||||
await _replace_entity_artwork(session, "season", season.id, tmdb_provider.id, normalized.artwork)
|
await polymorphic.sync_artwork(
|
||||||
|
session,
|
||||||
|
entity_type="season",
|
||||||
|
entity_id=season.id,
|
||||||
|
provider_id=tmdb_provider.id,
|
||||||
|
artwork_list=normalized.artwork,
|
||||||
|
)
|
||||||
|
|
||||||
for ep in normalized.episodes:
|
for ep in normalized.episodes:
|
||||||
episode = await tv_repository.get_or_create_episode(
|
episode = await tv_repository.get_or_create_episode(
|
||||||
@@ -390,7 +376,13 @@ async def ingest_season(
|
|||||||
await session.flush()
|
await session.flush()
|
||||||
|
|
||||||
await _replace_episode_credits(session, episode.id, tmdb_provider.id, ep.guest_stars, ep.crew)
|
await _replace_episode_credits(session, episode.id, tmdb_provider.id, ep.guest_stars, ep.crew)
|
||||||
await _replace_entity_artwork(session, "episode", episode.id, tmdb_provider.id, ep.stills)
|
await polymorphic.sync_artwork(
|
||||||
|
session,
|
||||||
|
entity_type="episode",
|
||||||
|
entity_id=episode.id,
|
||||||
|
provider_id=tmdb_provider.id,
|
||||||
|
artwork_list=ep.stills,
|
||||||
|
)
|
||||||
await refs.link_external_id(
|
await refs.link_external_id(
|
||||||
session,
|
session,
|
||||||
entity_type="episode",
|
entity_type="episode",
|
||||||
|
|||||||
@@ -66,6 +66,12 @@ class NormalizedArtwork:
|
|||||||
aspect_ratio: float | None
|
aspect_ratio: float | None
|
||||||
language: str | None
|
language: str | None
|
||||||
vote_average: float | None = None
|
vote_average: float | None = None
|
||||||
|
# True for the single image the provider designates as *the* image for
|
||||||
|
# this entity (TMDb's top-level poster_path/backdrop_path/profile_path),
|
||||||
|
# as opposed to the full alternates list — this is what gets
|
||||||
|
# auto-downloaded for local hosting (app/services/artwork/), so it's
|
||||||
|
# deliberately narrow rather than "the first one in the list".
|
||||||
|
is_primary: bool = False
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
@@ -182,6 +188,7 @@ def normalize_movie(raw: dict) -> NormalizedMovie:
|
|||||||
None,
|
None,
|
||||||
None,
|
None,
|
||||||
None,
|
None,
|
||||||
|
is_primary=True,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
if raw.get("backdrop_path"):
|
if raw.get("backdrop_path"):
|
||||||
@@ -194,6 +201,7 @@ def normalize_movie(raw: dict) -> NormalizedMovie:
|
|||||||
None,
|
None,
|
||||||
None,
|
None,
|
||||||
None,
|
None,
|
||||||
|
is_primary=True,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
for kind, tmdb_key in (("poster", "posters"), ("backdrop", "backdrops"), ("logo", "logos")):
|
for kind, tmdb_key in (("poster", "posters"), ("backdrop", "backdrops"), ("logo", "logos")):
|
||||||
@@ -309,19 +317,26 @@ def normalize_person(raw: dict) -> NormalizedPerson:
|
|||||||
)
|
)
|
||||||
for img in images
|
for img in images
|
||||||
]
|
]
|
||||||
if raw.get("profile_path") and not any(a.provider_asset_id == raw["profile_path"] for a in artwork):
|
if raw.get("profile_path"):
|
||||||
artwork.insert(
|
existing = next(
|
||||||
0,
|
(a for a in artwork if a.provider_asset_id == raw["profile_path"]), None
|
||||||
NormalizedArtwork(
|
|
||||||
"profile",
|
|
||||||
_image_url_required(raw["profile_path"]),
|
|
||||||
raw["profile_path"],
|
|
||||||
None,
|
|
||||||
None,
|
|
||||||
None,
|
|
||||||
None,
|
|
||||||
),
|
|
||||||
)
|
)
|
||||||
|
if existing is not None:
|
||||||
|
existing.is_primary = True
|
||||||
|
else:
|
||||||
|
artwork.insert(
|
||||||
|
0,
|
||||||
|
NormalizedArtwork(
|
||||||
|
"profile",
|
||||||
|
_image_url_required(raw["profile_path"]),
|
||||||
|
raw["profile_path"],
|
||||||
|
None,
|
||||||
|
None,
|
||||||
|
None,
|
||||||
|
None,
|
||||||
|
is_primary=True,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
also_known_as = raw.get("also_known_as") or []
|
also_known_as = raw.get("also_known_as") or []
|
||||||
|
|
||||||
@@ -473,6 +488,7 @@ def normalize_series(raw: dict) -> NormalizedSeries:
|
|||||||
None,
|
None,
|
||||||
None,
|
None,
|
||||||
None,
|
None,
|
||||||
|
is_primary=True,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
if raw.get("backdrop_path"):
|
if raw.get("backdrop_path"):
|
||||||
@@ -485,6 +501,7 @@ def normalize_series(raw: dict) -> NormalizedSeries:
|
|||||||
None,
|
None,
|
||||||
None,
|
None,
|
||||||
None,
|
None,
|
||||||
|
is_primary=True,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
for kind, tmdb_key in (("poster", "posters"), ("backdrop", "backdrops"), ("logo", "logos")):
|
for kind, tmdb_key in (("poster", "posters"), ("backdrop", "backdrops"), ("logo", "logos")):
|
||||||
@@ -526,7 +543,7 @@ def normalize_series(raw: dict) -> NormalizedSeries:
|
|||||||
poster=(
|
poster=(
|
||||||
NormalizedArtwork(
|
NormalizedArtwork(
|
||||||
"poster", _image_url_required(s["poster_path"]), s["poster_path"],
|
"poster", _image_url_required(s["poster_path"]), s["poster_path"],
|
||||||
None, None, None, None,
|
None, None, None, None, is_primary=True,
|
||||||
)
|
)
|
||||||
if s.get("poster_path")
|
if s.get("poster_path")
|
||||||
else None
|
else None
|
||||||
@@ -621,7 +638,7 @@ def normalize_season(raw: dict) -> NormalizedSeason:
|
|||||||
artwork.append(
|
artwork.append(
|
||||||
NormalizedArtwork(
|
NormalizedArtwork(
|
||||||
"poster", _image_url_required(raw["poster_path"]), raw["poster_path"],
|
"poster", _image_url_required(raw["poster_path"]), raw["poster_path"],
|
||||||
None, None, None, None,
|
None, None, None, None, is_primary=True,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
for img in images.get("posters", []):
|
for img in images.get("posters", []):
|
||||||
@@ -656,7 +673,7 @@ def normalize_season(raw: dict) -> NormalizedSeason:
|
|||||||
stills.append(
|
stills.append(
|
||||||
NormalizedArtwork(
|
NormalizedArtwork(
|
||||||
"still", _image_url_required(ep["still_path"]), ep["still_path"],
|
"still", _image_url_required(ep["still_path"]), ep["still_path"],
|
||||||
None, None, None, None,
|
None, None, None, None, is_primary=True,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
episodes.append(
|
episodes.append(
|
||||||
|
|||||||
+40
-4
@@ -1,10 +1,15 @@
|
|||||||
"""arq task functions. Thin wrappers around the ingestion runner so retry/
|
"""arq task functions. Thin wrappers around the ingestion runner so retry/
|
||||||
concurrency/observability behavior lives in arq's worker loop, not here.
|
concurrency/observability behavior lives in arq's worker loop, not here.
|
||||||
|
|
||||||
Each task receives the `IngestionJob` row's id (created synchronously by the
|
Each ingest_* task receives the `IngestionJob` row's id (created
|
||||||
enqueuing endpoint via `get_or_create_ingestion_job`, *not* generated here)
|
synchronously by the enqueuing endpoint via `get_or_create_ingestion_job`,
|
||||||
so status polling has a real, immediately-valid id to look up from the
|
*not* generated here) so status polling has a real, immediately-valid id to
|
||||||
moment the HTTP response comes back.
|
look up from the moment the HTTP response comes back. `download_artwork_task`
|
||||||
|
is the exception — it's an internal side effect of ingestion (see
|
||||||
|
`_cascade_artwork_downloads`), not something a client triggers or polls, so
|
||||||
|
it isn't tracked in `ingestion_jobs` at all; arq's own per-function retries
|
||||||
|
are enough for it (section 10: artwork self-hosting is best-effort caching,
|
||||||
|
not a user-facing operation with a status to report).
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
@@ -12,9 +17,14 @@ from __future__ import annotations
|
|||||||
import uuid
|
import uuid
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
|
from app.core.database import session_scope
|
||||||
from app.core.logging import get_logger
|
from app.core.logging import get_logger
|
||||||
|
from app.services.artwork.download import download_artwork
|
||||||
|
from app.services.artwork.storage import get_artwork_storage
|
||||||
from app.services.ingestion.runner import (
|
from app.services.ingestion.runner import (
|
||||||
get_or_create_ingestion_job,
|
get_or_create_ingestion_job,
|
||||||
|
get_primary_artwork_for_download,
|
||||||
|
get_series_season_ids,
|
||||||
get_series_season_numbers,
|
get_series_season_numbers,
|
||||||
run_movie_ingestion,
|
run_movie_ingestion,
|
||||||
run_person_ingestion,
|
run_person_ingestion,
|
||||||
@@ -25,13 +35,29 @@ from app.services.ingestion.runner import (
|
|||||||
logger = get_logger(__name__)
|
logger = get_logger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
async def _cascade_artwork_downloads(
|
||||||
|
ctx: dict[str, Any], entity_type: str, entity_ids: list[uuid.UUID]
|
||||||
|
) -> None:
|
||||||
|
artwork_ids = await get_primary_artwork_for_download(entity_type, entity_ids)
|
||||||
|
if not artwork_ids:
|
||||||
|
return
|
||||||
|
redis = ctx["redis"]
|
||||||
|
for artwork_id in artwork_ids:
|
||||||
|
await redis.enqueue_job("download_artwork_task", str(artwork_id))
|
||||||
|
logger.info(
|
||||||
|
"artwork_downloads_cascaded", entity_type=entity_type, count=len(artwork_ids)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
async def ingest_movie_task(ctx: dict[str, Any], job_id: str, external_id: str) -> str:
|
async def ingest_movie_task(ctx: dict[str, Any], job_id: str, external_id: str) -> str:
|
||||||
movie_id = await run_movie_ingestion(uuid.UUID(job_id), external_id)
|
movie_id = await run_movie_ingestion(uuid.UUID(job_id), external_id)
|
||||||
|
await _cascade_artwork_downloads(ctx, "movie", [movie_id])
|
||||||
return str(movie_id)
|
return str(movie_id)
|
||||||
|
|
||||||
|
|
||||||
async def ingest_person_task(ctx: dict[str, Any], job_id: str, external_id: str) -> str:
|
async def ingest_person_task(ctx: dict[str, Any], job_id: str, external_id: str) -> str:
|
||||||
person_id = await run_person_ingestion(uuid.UUID(job_id), external_id)
|
person_id = await run_person_ingestion(uuid.UUID(job_id), external_id)
|
||||||
|
await _cascade_artwork_downloads(ctx, "person", [person_id])
|
||||||
return str(person_id)
|
return str(person_id)
|
||||||
|
|
||||||
|
|
||||||
@@ -39,6 +65,11 @@ async def ingest_series_task(
|
|||||||
ctx: dict[str, Any], job_id: str, external_id: str, include_seasons: bool = False
|
ctx: dict[str, Any], job_id: str, external_id: str, include_seasons: bool = False
|
||||||
) -> str:
|
) -> str:
|
||||||
series_id = await run_series_ingestion(uuid.UUID(job_id), external_id)
|
series_id = await run_series_ingestion(uuid.UUID(job_id), external_id)
|
||||||
|
await _cascade_artwork_downloads(ctx, "series", [series_id])
|
||||||
|
# Season posters come from the season *stubs* ingest_series already
|
||||||
|
# creates, independent of include_seasons (which only controls whether
|
||||||
|
# full per-season episode data gets fetched) -- so this always runs.
|
||||||
|
await _cascade_artwork_downloads(ctx, "season", await get_series_season_ids(series_id))
|
||||||
|
|
||||||
if include_seasons:
|
if include_seasons:
|
||||||
# Fan out one ingest_season job per season this series has, using
|
# Fan out one ingest_season job per season this series has, using
|
||||||
@@ -65,3 +96,8 @@ async def ingest_season_task(
|
|||||||
) -> str:
|
) -> str:
|
||||||
season_id = await run_season_ingestion(uuid.UUID(job_id), series_external_id, season_number)
|
season_id = await run_season_ingestion(uuid.UUID(job_id), series_external_id, season_number)
|
||||||
return str(season_id)
|
return str(season_id)
|
||||||
|
|
||||||
|
|
||||||
|
async def download_artwork_task(ctx: dict[str, Any], artwork_id: str) -> bool:
|
||||||
|
async with session_scope() as session:
|
||||||
|
return await download_artwork(session, get_artwork_storage(), uuid.UUID(artwork_id))
|
||||||
|
|||||||
@@ -14,6 +14,7 @@ from arq.connections import RedisSettings
|
|||||||
from app.core.config import get_settings
|
from app.core.config import get_settings
|
||||||
from app.core.logging import configure_logging, get_logger
|
from app.core.logging import configure_logging, get_logger
|
||||||
from app.workers.tasks import (
|
from app.workers.tasks import (
|
||||||
|
download_artwork_task,
|
||||||
ingest_movie_task,
|
ingest_movie_task,
|
||||||
ingest_person_task,
|
ingest_person_task,
|
||||||
ingest_season_task,
|
ingest_season_task,
|
||||||
@@ -33,7 +34,13 @@ async def shutdown(ctx: dict) -> None:
|
|||||||
|
|
||||||
|
|
||||||
class WorkerSettings:
|
class WorkerSettings:
|
||||||
functions = [ingest_movie_task, ingest_person_task, ingest_series_task, ingest_season_task]
|
functions = [
|
||||||
|
ingest_movie_task,
|
||||||
|
ingest_person_task,
|
||||||
|
ingest_series_task,
|
||||||
|
ingest_season_task,
|
||||||
|
download_artwork_task,
|
||||||
|
]
|
||||||
on_startup = startup
|
on_startup = startup
|
||||||
on_shutdown = shutdown
|
on_shutdown = shutdown
|
||||||
redis_settings = RedisSettings.from_dsn(get_settings().REDIS_URL)
|
redis_settings = RedisSettings.from_dsn(get_settings().REDIS_URL)
|
||||||
|
|||||||
@@ -87,6 +87,9 @@ services:
|
|||||||
DATABASE_BEHIND_TRANSACTION_POOLER: "true"
|
DATABASE_BEHIND_TRANSACTION_POOLER: "true"
|
||||||
DATABASE_URL: postgresql+asyncpg://${POSTGRES_USER:-aurora}:${POSTGRES_PASSWORD}@pgbouncer:5432/${POSTGRES_DB:-aurora_core}
|
DATABASE_URL: postgresql+asyncpg://${POSTGRES_USER:-aurora}:${POSTGRES_PASSWORD}@pgbouncer:5432/${POSTGRES_DB:-aurora_core}
|
||||||
REDIS_URL: redis://redis:6379/0
|
REDIS_URL: redis://redis:6379/0
|
||||||
|
MEDIA_ROOT: /app/media
|
||||||
|
volumes:
|
||||||
|
- aurora_media_data:/app/media
|
||||||
depends_on:
|
depends_on:
|
||||||
pgbouncer:
|
pgbouncer:
|
||||||
condition: service_healthy
|
condition: service_healthy
|
||||||
@@ -119,6 +122,9 @@ services:
|
|||||||
DATABASE_BEHIND_TRANSACTION_POOLER: "true"
|
DATABASE_BEHIND_TRANSACTION_POOLER: "true"
|
||||||
DATABASE_URL: postgresql+asyncpg://${POSTGRES_USER:-aurora}:${POSTGRES_PASSWORD}@pgbouncer:5432/${POSTGRES_DB:-aurora_core}
|
DATABASE_URL: postgresql+asyncpg://${POSTGRES_USER:-aurora}:${POSTGRES_PASSWORD}@pgbouncer:5432/${POSTGRES_DB:-aurora_core}
|
||||||
REDIS_URL: redis://redis:6379/0
|
REDIS_URL: redis://redis:6379/0
|
||||||
|
MEDIA_ROOT: /app/media
|
||||||
|
volumes:
|
||||||
|
- aurora_media_data:/app/media
|
||||||
depends_on:
|
depends_on:
|
||||||
pgbouncer:
|
pgbouncer:
|
||||||
condition: service_healthy
|
condition: service_healthy
|
||||||
@@ -135,3 +141,4 @@ services:
|
|||||||
volumes:
|
volumes:
|
||||||
aurora_postgres_data:
|
aurora_postgres_data:
|
||||||
aurora_redis_data:
|
aurora_redis_data:
|
||||||
|
aurora_media_data:
|
||||||
|
|||||||
@@ -54,6 +54,7 @@ services:
|
|||||||
DATABASE_URL: postgresql+asyncpg://aurora:aurora@postgres:5432/aurora_core
|
DATABASE_URL: postgresql+asyncpg://aurora:aurora@postgres:5432/aurora_core
|
||||||
DATABASE_URL_SYNC: postgresql+psycopg2://aurora:aurora@postgres:5432/aurora_core
|
DATABASE_URL_SYNC: postgresql+psycopg2://aurora:aurora@postgres:5432/aurora_core
|
||||||
REDIS_URL: redis://redis:6379/0
|
REDIS_URL: redis://redis:6379/0
|
||||||
|
MEDIA_ROOT: /app/media
|
||||||
ports:
|
ports:
|
||||||
- "8000:8000"
|
- "8000:8000"
|
||||||
depends_on:
|
depends_on:
|
||||||
@@ -67,6 +68,7 @@ services:
|
|||||||
["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000", "--reload"]
|
["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000", "--reload"]
|
||||||
volumes:
|
volumes:
|
||||||
- ./app:/app/app
|
- ./app:/app/app
|
||||||
|
- aurora_media_data:/app/media
|
||||||
|
|
||||||
worker:
|
worker:
|
||||||
build:
|
build:
|
||||||
@@ -78,6 +80,7 @@ services:
|
|||||||
DATABASE_URL: postgresql+asyncpg://aurora:aurora@postgres:5432/aurora_core
|
DATABASE_URL: postgresql+asyncpg://aurora:aurora@postgres:5432/aurora_core
|
||||||
DATABASE_URL_SYNC: postgresql+psycopg2://aurora:aurora@postgres:5432/aurora_core
|
DATABASE_URL_SYNC: postgresql+psycopg2://aurora:aurora@postgres:5432/aurora_core
|
||||||
REDIS_URL: redis://redis:6379/0
|
REDIS_URL: redis://redis:6379/0
|
||||||
|
MEDIA_ROOT: /app/media
|
||||||
depends_on:
|
depends_on:
|
||||||
postgres:
|
postgres:
|
||||||
condition: service_healthy
|
condition: service_healthy
|
||||||
@@ -87,6 +90,8 @@ services:
|
|||||||
condition: service_completed_successfully
|
condition: service_completed_successfully
|
||||||
volumes:
|
volumes:
|
||||||
- ./app:/app/app
|
- ./app:/app/app
|
||||||
|
- aurora_media_data:/app/media
|
||||||
|
|
||||||
volumes:
|
volumes:
|
||||||
aurora_postgres_data:
|
aurora_postgres_data:
|
||||||
|
aurora_media_data:
|
||||||
|
|||||||
@@ -90,6 +90,40 @@ star can race to create the same `Person` row (see
|
|||||||
recovers from that specific race directly; the arq retry here is the
|
recovers from that specific race directly; the arq retry here is the
|
||||||
backstop for whatever it doesn't).
|
backstop for whatever it doesn't).
|
||||||
|
|
||||||
|
## Artwork self-hosting (section 10)
|
||||||
|
|
||||||
|
```
|
||||||
|
ingest_movie/ingest_series/ingest_person succeeds
|
||||||
|
-> tasks.py: get_primary_artwork_for_download(entity_type, [entity_id, ...])
|
||||||
|
-> one download_artwork_task enqueued per primary artwork id
|
||||||
|
-> download_artwork(): fetch original_url, verify it's an image, cap
|
||||||
|
size, sha256 it, storage.save(), set Artwork.local_path + checksum
|
||||||
|
```
|
||||||
|
|
||||||
|
Every image API responses reference is stored with its provider CDN URL
|
||||||
|
(`original_url`) from the moment it's ingested; `local_path` starts empty
|
||||||
|
and is filled in by this pipeline running asynchronously afterward, so
|
||||||
|
responses transparently point at TMDb until a local copy exists, then
|
||||||
|
switch over. Only artwork marked `is_primary` (the one image TMDb itself
|
||||||
|
designates for an entity, not the dozens of alternates also ingested) gets
|
||||||
|
downloaded automatically — see `docs/database.md#artwork-storage-self-hosting`.
|
||||||
|
|
||||||
|
Storage is behind an `ArtworkStorage` interface
|
||||||
|
(`app/services/artwork/storage.py`) with one implementation,
|
||||||
|
`LocalDiskStorage` — a shared Docker volume between the `api` and `worker`
|
||||||
|
containers (`docker-compose.yml`'s `aurora_media_data`), which is enough
|
||||||
|
for this project's actual deployment shape (one Docker host, principle 9:
|
||||||
|
avoid premature complexity) but assumes exactly that. A deployment that
|
||||||
|
outgrew a single host would implement the same interface against S3/R2
|
||||||
|
instead, without the download pipeline or callers changing.
|
||||||
|
|
||||||
|
This is also why `app/repositories/polymorphic.py:sync_artwork` upserts
|
||||||
|
artwork by `provider_asset_id` instead of the delete-then-recreate pattern
|
||||||
|
every other satellite table uses (movie/series/person credits, videos,
|
||||||
|
genre/keyword/company links): a delete-then-recreate would generate a fresh
|
||||||
|
`Artwork.id` with `local_path=None` on every re-ingestion, silently
|
||||||
|
discarding every downloaded copy and forcing a full re-download each sync.
|
||||||
|
|
||||||
## Search (section 16)
|
## Search (section 16)
|
||||||
|
|
||||||
`app/services/search/base.py:SearchProvider` is the abstraction; API
|
`app/services/search/base.py:SearchProvider` is the abstraction; API
|
||||||
|
|||||||
+43
-12
@@ -71,6 +71,31 @@ externally-sourced images can be freely redistributed. Aurora never
|
|||||||
downloads video media; `videos.url` is a reference to the provider's
|
downloads video media; `videos.url` is a reference to the provider's
|
||||||
hosted copy (YouTube, Vimeo, ...).
|
hosted copy (YouTube, Vimeo, ...).
|
||||||
|
|
||||||
|
### Artwork storage (self-hosting)
|
||||||
|
|
||||||
|
`artwork.original_url` (the provider's CDN link) is always populated by
|
||||||
|
ingestion; `artwork.local_path` starts `NULL` and is filled in
|
||||||
|
asynchronously by `app/services/artwork/`. `app/services/common.py:artwork_url()`
|
||||||
|
prefers `local_path` and falls back to `original_url`, so API responses
|
||||||
|
point at TMDb's CDN until a copy has been downloaded, then switch over
|
||||||
|
automatically once it has.
|
||||||
|
|
||||||
|
Only artwork marked `is_primary` gets downloaded automatically — the single
|
||||||
|
image the provider designates as *the* image for an entity (TMDb's
|
||||||
|
`poster_path`/`backdrop_path`/`profile_path`), not the dozens of alternates
|
||||||
|
also stored. `app/services/ingestion/runner.py:get_primary_artwork_for_download`
|
||||||
|
finds them and `app/workers/tasks.py` enqueues a `download_artwork_task` per
|
||||||
|
id once movie/series/person/season ingestion succeeds — see
|
||||||
|
`docs/architecture.md` for the full flow.
|
||||||
|
|
||||||
|
Storage is local disk (`app/services/artwork/storage.py:LocalDiskStorage`)
|
||||||
|
behind an `ArtworkStorage` interface, deliberately simple for a
|
||||||
|
single-Docker-host deployment where the `api` and `worker` containers share
|
||||||
|
one volume (`docker-compose.yml`'s `aurora_media_data`). A deployment that
|
||||||
|
outgrows one host would need an S3/R2-backed implementation of the same
|
||||||
|
interface instead. Files are named by content (`<type>/<sha256[:2]>/<sha256>.<ext>`),
|
||||||
|
so the same image referenced by two entities is only ever stored once.
|
||||||
|
|
||||||
## Polymorphic associations
|
## Polymorphic associations
|
||||||
|
|
||||||
`credits`, `artwork`, `videos`, `external_ids`, and `release_dates` attach to
|
`credits`, `artwork`, `videos`, `external_ids`, and `release_dates` attach to
|
||||||
@@ -120,14 +145,15 @@ same lowercase strings.
|
|||||||
|
|
||||||
Alembic (section 27); the application **never** creates tables at startup —
|
Alembic (section 27); the application **never** creates tables at startup —
|
||||||
see `app/core/database.py`. The initial migration
|
see `app/core/database.py`. The initial migration
|
||||||
(`migrations/versions/0001_initial_schema.py`) is hand-written rather than
|
(`migrations/versions/0001_initial_schema.py`) was hand-written rather than
|
||||||
autogenerated (no throwaway Postgres was available in the environment this
|
autogenerated (no throwaway Postgres was available in the environment it
|
||||||
was built in — see the root README's Known Limitations), but was written
|
was originally written in), then verified by actually running
|
||||||
directly from the same models this document describes and validated with
|
`alembic upgrade head` against a genuinely empty PostgreSQL on a real
|
||||||
`alembic history`; treat a first `alembic upgrade head` against a scratch
|
devbox — that run caught a real bug (pre-creating enum types and then also
|
||||||
database as the real acceptance test, and autogenerate future migrations
|
letting `CREATE TABLE` auto-create them raised `DuplicateObject`; fixed by
|
||||||
against it (`alembic revision --autogenerate`) rather than the empty
|
`create_type=False` on each enum), now fixed and verified clean. Autogenerate
|
||||||
database this one was written from.
|
future migrations against a database that already has this one applied
|
||||||
|
(`alembic revision --autogenerate`), not from scratch.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
alembic upgrade head # apply all migrations
|
alembic upgrade head # apply all migrations
|
||||||
@@ -159,10 +185,15 @@ indexes on the alternate-title tables. See
|
|||||||
- Field-level provenance (which provider supplied *this* value) is not
|
- Field-level provenance (which provider supplied *this* value) is not
|
||||||
tracked, only entity-level (`source_provider_id`) and per-provider raw
|
tracked, only entity-level (`source_provider_id`) and per-provider raw
|
||||||
payloads (`provider_records.raw_data`).
|
payloads (`provider_records.raw_data`).
|
||||||
- Satellite associations (credits, artwork, videos, genre/keyword/company
|
- Satellite associations (credits, videos, genre/keyword/company links) are
|
||||||
links) are replaced wholesale on each re-sync rather than diffed, so their
|
replaced wholesale on each re-sync rather than diffed, so their Aurora IDs
|
||||||
Aurora IDs are not stable across re-ingestion — see
|
are not stable across re-ingestion — see
|
||||||
`app/services/ingestion/movie_ingestion.py`.
|
`app/services/ingestion/movie_ingestion.py`. **Artwork is the exception:**
|
||||||
|
`app/repositories/polymorphic.py:sync_artwork` upserts by
|
||||||
|
`provider_asset_id` instead, specifically so a previously-downloaded
|
||||||
|
`local_path`/`checksum` (see "Artwork storage" below) survives
|
||||||
|
re-ingestion instead of being silently reset to `None` and re-downloaded
|
||||||
|
every sync.
|
||||||
- Seasons don't get their own `external_ids` row (identified instead by the
|
- Seasons don't get their own `external_ids` row (identified instead by the
|
||||||
natural `(series_id, season_number)` key) — episodes do, since TMDb gives
|
natural `(series_id, season_number)` key) — episodes do, since TMDb gives
|
||||||
each episode a distinct global id. Revisit if a provider ever needs a
|
each episode a distinct global id. Revisit if a provider ever needs a
|
||||||
|
|||||||
@@ -0,0 +1,116 @@
|
|||||||
|
"""Integration tests for the artwork download pipeline
|
||||||
|
(app/services/artwork/download.py). Uses httpx.MockTransport to simulate
|
||||||
|
TMDb's CDN so these run without network access, while still exercising a
|
||||||
|
real Artwork row against PostgreSQL."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
import pytest
|
||||||
|
from sqlalchemy.ext.asyncio import AsyncSession
|
||||||
|
|
||||||
|
from app.models.artwork import Artwork
|
||||||
|
from app.models.enums import ArtworkType
|
||||||
|
from app.services.artwork.download import ArtworkDownloadError, download_artwork
|
||||||
|
from app.services.artwork.storage import LocalDiskStorage
|
||||||
|
from tests.integration.factories import make_provider
|
||||||
|
|
||||||
|
FAKE_JPEG_BYTES = b"\xff\xd8\xff\xe0" + b"0" * 100 # not a *real* JPEG, just image-shaped bytes
|
||||||
|
|
||||||
|
|
||||||
|
def _client_for(handler) -> httpx.AsyncClient:
|
||||||
|
return httpx.AsyncClient(transport=httpx.MockTransport(handler))
|
||||||
|
|
||||||
|
|
||||||
|
async def _make_artwork(session: AsyncSession, *, local_path: str | None = None) -> Artwork:
|
||||||
|
provider = await make_provider(session)
|
||||||
|
artwork = Artwork(
|
||||||
|
entity_type="movie",
|
||||||
|
entity_id=provider.id, # any UUID; polymorphic tables have no real FK
|
||||||
|
artwork_type=ArtworkType.POSTER,
|
||||||
|
provider_id=provider.id,
|
||||||
|
original_url="https://image.tmdb.org/t/p/original/poster.jpg",
|
||||||
|
local_path=local_path,
|
||||||
|
)
|
||||||
|
session.add(artwork)
|
||||||
|
await session.flush()
|
||||||
|
return artwork
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_download_artwork_saves_file_and_sets_local_path_and_checksum(
|
||||||
|
db_session: AsyncSession, tmp_path
|
||||||
|
):
|
||||||
|
artwork = await _make_artwork(db_session)
|
||||||
|
storage = LocalDiskStorage(str(tmp_path), "/media")
|
||||||
|
|
||||||
|
def handler(request: httpx.Request) -> httpx.Response:
|
||||||
|
return httpx.Response(200, headers={"content-type": "image/jpeg"}, content=FAKE_JPEG_BYTES)
|
||||||
|
|
||||||
|
async with _client_for(handler) as client:
|
||||||
|
result = await download_artwork(db_session, storage, artwork.id, client=client)
|
||||||
|
|
||||||
|
assert result is True
|
||||||
|
await db_session.refresh(artwork)
|
||||||
|
assert artwork.local_path is not None
|
||||||
|
assert artwork.local_path.startswith("/media/poster/")
|
||||||
|
assert artwork.checksum is not None
|
||||||
|
assert await storage.exists(artwork.local_path.removeprefix("/media/"))
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_download_artwork_skips_when_already_downloaded(db_session: AsyncSession, tmp_path):
|
||||||
|
artwork = await _make_artwork(db_session, local_path="/media/poster/ab/existing.jpg")
|
||||||
|
storage = LocalDiskStorage(str(tmp_path), "/media")
|
||||||
|
|
||||||
|
def handler(request: httpx.Request) -> httpx.Response:
|
||||||
|
raise AssertionError("should not make an HTTP request when already downloaded")
|
||||||
|
|
||||||
|
async with _client_for(handler) as client:
|
||||||
|
result = await download_artwork(db_session, storage, artwork.id, client=client)
|
||||||
|
|
||||||
|
assert result is False
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_download_artwork_force_redownloads(db_session: AsyncSession, tmp_path):
|
||||||
|
artwork = await _make_artwork(db_session, local_path="/media/poster/ab/existing.jpg")
|
||||||
|
storage = LocalDiskStorage(str(tmp_path), "/media")
|
||||||
|
|
||||||
|
def handler(request: httpx.Request) -> httpx.Response:
|
||||||
|
return httpx.Response(200, headers={"content-type": "image/png"}, content=FAKE_JPEG_BYTES)
|
||||||
|
|
||||||
|
async with _client_for(handler) as client:
|
||||||
|
result = await download_artwork(db_session, storage, artwork.id, client=client, force=True)
|
||||||
|
|
||||||
|
assert result is True
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_download_artwork_rejects_non_image_content_type(
|
||||||
|
db_session: AsyncSession, tmp_path
|
||||||
|
):
|
||||||
|
artwork = await _make_artwork(db_session)
|
||||||
|
storage = LocalDiskStorage(str(tmp_path), "/media")
|
||||||
|
|
||||||
|
def handler(request: httpx.Request) -> httpx.Response:
|
||||||
|
return httpx.Response(200, headers={"content-type": "text/html"}, content=b"<html></html>")
|
||||||
|
|
||||||
|
async with _client_for(handler) as client:
|
||||||
|
with pytest.raises(ArtworkDownloadError, match="expected an image"):
|
||||||
|
await download_artwork(db_session, storage, artwork.id, client=client)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_download_artwork_returns_false_for_missing_row(db_session: AsyncSession, tmp_path):
|
||||||
|
import uuid
|
||||||
|
|
||||||
|
storage = LocalDiskStorage(str(tmp_path), "/media")
|
||||||
|
|
||||||
|
def handler(request: httpx.Request) -> httpx.Response:
|
||||||
|
raise AssertionError("should not make an HTTP request for a nonexistent artwork row")
|
||||||
|
|
||||||
|
async with _client_for(handler) as client:
|
||||||
|
result = await download_artwork(db_session, storage, uuid.uuid4(), client=client)
|
||||||
|
|
||||||
|
assert result is False
|
||||||
@@ -0,0 +1,162 @@
|
|||||||
|
"""Integration tests for app/repositories/polymorphic.py:sync_artwork.
|
||||||
|
|
||||||
|
The one guarantee this module exists for: re-ingesting a title must not
|
||||||
|
wipe out artwork that app/services/artwork/ already downloaded locally.
|
||||||
|
Before sync_artwork, every ingestion pipeline deleted and recreated all of
|
||||||
|
an entity's Artwork rows on each sync, which would silently reset
|
||||||
|
local_path/checksum to None every time -- defeating the entire point of
|
||||||
|
caching artwork locally.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import uuid
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from sqlalchemy import select
|
||||||
|
from sqlalchemy.ext.asyncio import AsyncSession
|
||||||
|
|
||||||
|
from app.models.artwork import Artwork
|
||||||
|
from app.repositories.polymorphic import sync_artwork
|
||||||
|
from app.services.metadata.normalizer import NormalizedArtwork
|
||||||
|
from tests.integration.factories import make_provider
|
||||||
|
|
||||||
|
|
||||||
|
def _artwork(asset_id: str, *, vote_average: float = 5.0, is_primary: bool = False) -> NormalizedArtwork:
|
||||||
|
return NormalizedArtwork(
|
||||||
|
artwork_type="poster",
|
||||||
|
url=f"https://image.tmdb.org/t/p/original/{asset_id}",
|
||||||
|
provider_asset_id=asset_id,
|
||||||
|
width=1000,
|
||||||
|
height=1500,
|
||||||
|
aspect_ratio=0.667,
|
||||||
|
language=None,
|
||||||
|
vote_average=vote_average,
|
||||||
|
is_primary=is_primary,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_sync_artwork_creates_rows_on_first_sync(db_session: AsyncSession):
|
||||||
|
provider = await make_provider(db_session)
|
||||||
|
entity_id = uuid.uuid4()
|
||||||
|
|
||||||
|
await sync_artwork(
|
||||||
|
db_session,
|
||||||
|
entity_type="movie",
|
||||||
|
entity_id=entity_id,
|
||||||
|
provider_id=provider.id,
|
||||||
|
artwork_list=[_artwork("a.jpg", is_primary=True), _artwork("b.jpg")],
|
||||||
|
)
|
||||||
|
await db_session.commit()
|
||||||
|
|
||||||
|
rows = (
|
||||||
|
(await db_session.execute(select(Artwork).where(Artwork.entity_id == entity_id)))
|
||||||
|
.scalars()
|
||||||
|
.all()
|
||||||
|
)
|
||||||
|
assert {r.provider_asset_id for r in rows} == {"a.jpg", "b.jpg"}
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_sync_artwork_preserves_local_path_across_resync(db_session: AsyncSession):
|
||||||
|
provider = await make_provider(db_session)
|
||||||
|
entity_id = uuid.uuid4()
|
||||||
|
|
||||||
|
await sync_artwork(
|
||||||
|
db_session,
|
||||||
|
entity_type="movie",
|
||||||
|
entity_id=entity_id,
|
||||||
|
provider_id=provider.id,
|
||||||
|
artwork_list=[_artwork("a.jpg", vote_average=5.0)],
|
||||||
|
)
|
||||||
|
await db_session.flush()
|
||||||
|
|
||||||
|
row = (
|
||||||
|
await db_session.execute(select(Artwork).where(Artwork.entity_id == entity_id))
|
||||||
|
).scalar_one()
|
||||||
|
row.local_path = "/media/poster/ab/deadbeef.jpg"
|
||||||
|
row.checksum = "deadbeef"
|
||||||
|
await db_session.flush()
|
||||||
|
|
||||||
|
# Re-sync as if the movie were re-ingested: same image, but TMDb now
|
||||||
|
# reports a different vote_average.
|
||||||
|
await sync_artwork(
|
||||||
|
db_session,
|
||||||
|
entity_type="movie",
|
||||||
|
entity_id=entity_id,
|
||||||
|
provider_id=provider.id,
|
||||||
|
artwork_list=[_artwork("a.jpg", vote_average=9.0)],
|
||||||
|
)
|
||||||
|
await db_session.commit()
|
||||||
|
|
||||||
|
refreshed = (
|
||||||
|
await db_session.execute(select(Artwork).where(Artwork.entity_id == entity_id))
|
||||||
|
).scalar_one()
|
||||||
|
assert refreshed.id == row.id # same row, not delete+recreate
|
||||||
|
assert refreshed.local_path == "/media/poster/ab/deadbeef.jpg"
|
||||||
|
assert refreshed.checksum == "deadbeef"
|
||||||
|
assert refreshed.vote_average == 9.0 # non-download fields still update
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_sync_artwork_removes_rows_no_longer_present(db_session: AsyncSession):
|
||||||
|
provider = await make_provider(db_session)
|
||||||
|
entity_id = uuid.uuid4()
|
||||||
|
|
||||||
|
await sync_artwork(
|
||||||
|
db_session,
|
||||||
|
entity_type="movie",
|
||||||
|
entity_id=entity_id,
|
||||||
|
provider_id=provider.id,
|
||||||
|
artwork_list=[_artwork("a.jpg"), _artwork("b.jpg")],
|
||||||
|
)
|
||||||
|
await db_session.flush()
|
||||||
|
|
||||||
|
# "b.jpg" dropped from the provider's alternates on re-sync.
|
||||||
|
await sync_artwork(
|
||||||
|
db_session,
|
||||||
|
entity_type="movie",
|
||||||
|
entity_id=entity_id,
|
||||||
|
provider_id=provider.id,
|
||||||
|
artwork_list=[_artwork("a.jpg")],
|
||||||
|
)
|
||||||
|
await db_session.commit()
|
||||||
|
|
||||||
|
rows = (
|
||||||
|
(await db_session.execute(select(Artwork).where(Artwork.entity_id == entity_id)))
|
||||||
|
.scalars()
|
||||||
|
.all()
|
||||||
|
)
|
||||||
|
assert {r.provider_asset_id for r in rows} == {"a.jpg"}
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_sync_artwork_adds_new_rows_on_resync(db_session: AsyncSession):
|
||||||
|
provider = await make_provider(db_session)
|
||||||
|
entity_id = uuid.uuid4()
|
||||||
|
|
||||||
|
await sync_artwork(
|
||||||
|
db_session,
|
||||||
|
entity_type="movie",
|
||||||
|
entity_id=entity_id,
|
||||||
|
provider_id=provider.id,
|
||||||
|
artwork_list=[_artwork("a.jpg")],
|
||||||
|
)
|
||||||
|
await db_session.flush()
|
||||||
|
|
||||||
|
await sync_artwork(
|
||||||
|
db_session,
|
||||||
|
entity_type="movie",
|
||||||
|
entity_id=entity_id,
|
||||||
|
provider_id=provider.id,
|
||||||
|
artwork_list=[_artwork("a.jpg"), _artwork("c.jpg")],
|
||||||
|
)
|
||||||
|
await db_session.commit()
|
||||||
|
|
||||||
|
rows = (
|
||||||
|
(await db_session.execute(select(Artwork).where(Artwork.entity_id == entity_id)))
|
||||||
|
.scalars()
|
||||||
|
.all()
|
||||||
|
)
|
||||||
|
assert {r.provider_asset_id for r in rows} == {"a.jpg", "c.jpg"}
|
||||||
@@ -0,0 +1,58 @@
|
|||||||
|
"""Unit tests for the low-level HTTP fetch used by artwork downloads
|
||||||
|
(app/services/artwork/download.py:_fetch). No database needed — this is
|
||||||
|
pure request/response handling, tested against a mocked transport."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from app.services.artwork.download import ArtworkDownloadError, _fetch
|
||||||
|
|
||||||
|
FAKE_JPEG_BYTES = b"\xff\xd8\xff\xe0" + b"0" * 100
|
||||||
|
|
||||||
|
|
||||||
|
def _client_for(handler) -> httpx.AsyncClient:
|
||||||
|
return httpx.AsyncClient(transport=httpx.MockTransport(handler))
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_fetch_returns_content_and_content_type():
|
||||||
|
def handler(request: httpx.Request) -> httpx.Response:
|
||||||
|
return httpx.Response(200, headers={"content-type": "image/jpeg"}, content=FAKE_JPEG_BYTES)
|
||||||
|
|
||||||
|
async with _client_for(handler) as client:
|
||||||
|
content, content_type = await _fetch(client, "https://example.test/poster.jpg", 1_000_000)
|
||||||
|
|
||||||
|
assert content == FAKE_JPEG_BYTES
|
||||||
|
assert content_type == "image/jpeg"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_fetch_rejects_non_image_content_type():
|
||||||
|
def handler(request: httpx.Request) -> httpx.Response:
|
||||||
|
return httpx.Response(200, headers={"content-type": "text/html"}, content=b"<html></html>")
|
||||||
|
|
||||||
|
async with _client_for(handler) as client:
|
||||||
|
with pytest.raises(ArtworkDownloadError, match="expected an image"):
|
||||||
|
await _fetch(client, "https://example.test/oops.html", 1_000_000)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_fetch_rejects_response_over_max_bytes():
|
||||||
|
def handler(request: httpx.Request) -> httpx.Response:
|
||||||
|
return httpx.Response(200, headers={"content-type": "image/jpeg"}, content=FAKE_JPEG_BYTES)
|
||||||
|
|
||||||
|
async with _client_for(handler) as client:
|
||||||
|
with pytest.raises(ArtworkDownloadError, match="exceeds ARTWORK_DOWNLOAD_MAX_BYTES"):
|
||||||
|
await _fetch(client, "https://example.test/huge.jpg", max_bytes=10)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_fetch_wraps_http_errors():
|
||||||
|
def handler(request: httpx.Request) -> httpx.Response:
|
||||||
|
return httpx.Response(404)
|
||||||
|
|
||||||
|
async with _client_for(handler) as client:
|
||||||
|
with pytest.raises(ArtworkDownloadError, match="failed to download"):
|
||||||
|
await _fetch(client, "https://example.test/missing.jpg", 1_000_000)
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
"""Unit tests for LocalDiskStorage (app/services/artwork/storage.py)."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from app.services.artwork.storage import LocalDiskStorage
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def storage(tmp_path) -> LocalDiskStorage:
|
||||||
|
return LocalDiskStorage(str(tmp_path), "/media")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_save_writes_file_and_returns_public_url(storage: LocalDiskStorage):
|
||||||
|
url = await storage.save("poster/ab/abcdef.jpg", b"hello world")
|
||||||
|
assert url == "/media/poster/ab/abcdef.jpg"
|
||||||
|
assert await storage.exists("poster/ab/abcdef.jpg")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_save_creates_intermediate_directories(storage: LocalDiskStorage):
|
||||||
|
await storage.save("still/de/deadbeef.png", b"data")
|
||||||
|
assert await storage.exists("still/de/deadbeef.png")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_exists_is_false_for_missing_file(storage: LocalDiskStorage):
|
||||||
|
assert not await storage.exists("poster/00/never-written.jpg")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_save_rejects_path_traversal(storage: LocalDiskStorage):
|
||||||
|
with pytest.raises(ValueError, match="escapes MEDIA_ROOT"):
|
||||||
|
await storage.save("../../etc/passwd", b"malicious")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_url_prefix_trailing_slash_is_normalized(tmp_path):
|
||||||
|
storage = LocalDiskStorage(str(tmp_path), "/media/")
|
||||||
|
url = await storage.save("poster/ab/x.jpg", b"data")
|
||||||
|
assert url == "/media/poster/ab/x.jpg"
|
||||||
@@ -145,6 +145,21 @@ def test_normalize_movie_maps_artwork_and_videos():
|
|||||||
assert movie.videos[0].is_official is True
|
assert movie.videos[0].is_official is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_normalize_movie_marks_exactly_one_primary_poster_and_backdrop():
|
||||||
|
# is_primary picks out TMDb's top-level poster_path/backdrop_path (the
|
||||||
|
# image auto-downloaded for local hosting) from the full alternates
|
||||||
|
# list -- not just "the first one".
|
||||||
|
movie = normalize_movie(FAKE_MOVIE)
|
||||||
|
primary_posters = [a for a in movie.artwork if a.artwork_type == "poster" and a.is_primary]
|
||||||
|
primary_backdrops = [
|
||||||
|
a for a in movie.artwork if a.artwork_type == "backdrop" and a.is_primary
|
||||||
|
]
|
||||||
|
assert len(primary_posters) == 1
|
||||||
|
assert primary_posters[0].provider_asset_id == "/poster.jpg"
|
||||||
|
assert len(primary_backdrops) == 1
|
||||||
|
assert primary_backdrops[0].provider_asset_id == "/backdrop.jpg"
|
||||||
|
|
||||||
|
|
||||||
def test_normalize_movie_external_ids_never_include_tmdb_itself():
|
def test_normalize_movie_external_ids_never_include_tmdb_itself():
|
||||||
# TMDb's own id is handled separately by the ingestion pipeline via
|
# TMDb's own id is handled separately by the ingestion pipeline via
|
||||||
# `normalized.external_id`, not folded into `external_ids` (section 3:
|
# `normalized.external_id`, not folded into `external_ids` (section 3:
|
||||||
@@ -187,6 +202,9 @@ def test_normalize_person_profile_artwork_deduplicated():
|
|||||||
# not be inserted twice.
|
# not be inserted twice.
|
||||||
paths = [a.provider_asset_id for a in person.artwork]
|
paths = [a.provider_asset_id for a in person.artwork]
|
||||||
assert paths.count("/keanu.jpg") == 1
|
assert paths.count("/keanu.jpg") == 1
|
||||||
|
# ...and since it wasn't inserted as a second row, the existing one must
|
||||||
|
# be the one marked primary instead -- not silently left unmarked.
|
||||||
|
assert next(a for a in person.artwork if a.provider_asset_id == "/keanu.jpg").is_primary
|
||||||
|
|
||||||
|
|
||||||
FAKE_SERIES = {
|
FAKE_SERIES = {
|
||||||
|
|||||||
Reference in New Issue
Block a user