106 lines
4.5 KiB
Markdown
106 lines
4.5 KiB
Markdown
|
# Real generation backend (DiffRhythm) — GPU machine setup
|
|||
|
|
|
|||
|
|
Aurora's worker (`apps/worker`) talks to generation backends through one
|
|||
|
|
interface (`packages/inference/base.py`). Locally you run it with
|
|||
|
|
`GENERATION_BACKEND=mock`, which synthesizes a placeholder clip instantly so
|
|||
|
|
you can build/test the product without a GPU. On your machine with the
|
|||
|
|
NVIDIA card, switch to `GENERATION_BACKEND=diffrhythm`, which shells out to
|
|||
|
|
[DiffRhythm](https://github.com/ASLP-lab/DiffRhythm) (Apache-2.0) — an
|
|||
|
|
open-source model that generates a full song (vocals + instrumentation) from
|
|||
|
|
a text style prompt and lyrics in one pass.
|
|||
|
|
|
|||
|
|
Why DiffRhythm over the other realistic open-source option (YuE): YuE needs
|
|||
|
|
~20GB+ VRAM and multiple minutes per song; DiffRhythm runs on as little as
|
|||
|
|
6–8GB VRAM and generates a full song in roughly the time it takes to read
|
|||
|
|
this sentence. With 128GB system RAM and an NVIDIA card, DiffRhythm will run
|
|||
|
|
comfortably; if your card has 24GB+ VRAM you can also try YuE for
|
|||
|
|
potentially higher-fidelity vocals — the same `MusicGenerationProvider`
|
|||
|
|
interface can host either.
|
|||
|
|
|
|||
|
|
## 1. Run Aurora's own services
|
|||
|
|
|
|||
|
|
Follow the root [README](../../README.md) to get Postgres, Redis, the API,
|
|||
|
|
and the worker running on this machine (same steps as the dev machine).
|
|||
|
|
|
|||
|
|
## 2. Install DiffRhythm
|
|||
|
|
|
|||
|
|
DiffRhythm is a script repo with its own heavy CUDA dependency set (torch,
|
|||
|
|
MuQ, etc.) — keep it in its own Python environment, separate from
|
|||
|
|
`apps/worker/.venv`, so version pins never collide.
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
git clone https://github.com/ASLP-lab/DiffRhythm.git
|
|||
|
|
cd DiffRhythm
|
|||
|
|
|
|||
|
|
# espeak-ng is a hard dependency of the phonemizer used for lyrics
|
|||
|
|
sudo apt-get install -y espeak-ng # or your distro's equivalent
|
|||
|
|
|
|||
|
|
python3.10 -m venv .venv
|
|||
|
|
source .venv/bin/activate
|
|||
|
|
pip install -r requirements.txt
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Confirm your CUDA build of torch actually sees the GPU before going further:
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
The first real inference run downloads model checkpoints from Hugging Face
|
|||
|
|
(`ASLP-lab/DiffRhythm-1_2-full` for the 4m45s-capable model, or the smaller
|
|||
|
|
`-base` checkpoint capped at 95s) — expect a multi-GB download the first
|
|||
|
|
time. Check `infer/infer.py` in the checked-out repo for how checkpoint
|
|||
|
|
selection works in the version you land on; this has moved around across
|
|||
|
|
DiffRhythm releases.
|
|||
|
|
|
|||
|
|
Sanity-check the install with the repo's own example before wiring it into
|
|||
|
|
Aurora:
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
bash scripts/infer_prompt_ref.sh
|
|||
|
|
# should produce a .wav under infer/example/output
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## 3. Point Aurora's worker at it
|
|||
|
|
|
|||
|
|
In `apps/worker/.env` on this machine:
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
GENERATION_BACKEND=diffrhythm
|
|||
|
|
DIFFRHYTHM_REPO_DIR=/absolute/path/to/DiffRhythm
|
|||
|
|
DIFFRHYTHM_PYTHON=/absolute/path/to/DiffRhythm/.venv/bin/python
|
|||
|
|
DIFFRHYTHM_CHUNKED=1 # keeps VRAM usage low; drop if you have headroom
|
|||
|
|
DIFFRHYTHM_MAX_DURATION=285 # 95 for the -base checkpoint, 285 for -full
|
|||
|
|
DIFFRHYTHM_DEFAULT_STYLE_PROMPT=melodic, well-produced, modern
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
`DIFFRHYTHM_PYTHON` must point at the interpreter *inside DiffRhythm's own
|
|||
|
|
venv*, not Aurora's worker venv — the provider (`packages/inference/diffrhythm_provider.py`)
|
|||
|
|
invokes it as a subprocess, so the two dependency sets never need to
|
|||
|
|
reconcile.
|
|||
|
|
|
|||
|
|
Bump `apps/worker/.env`'s `DEFAULT_SONG_DURATION_SECONDS` to whatever you
|
|||
|
|
want the default generated length to be (capped by `DIFFRHYTHM_MAX_DURATION`
|
|||
|
|
above).
|
|||
|
|
|
|||
|
|
Restart the worker; it'll route generation jobs through DiffRhythm instead
|
|||
|
|
of the mock provider. Generation will now take real wall-clock time (seconds
|
|||
|
|
to low minutes depending on length/GPU), so the frontend's library polling
|
|||
|
|
(already built to handle `pending`/`processing` states) is what makes this
|
|||
|
|
transition invisible to the product.
|
|||
|
|
|
|||
|
|
## Known rough edges to expect
|
|||
|
|
|
|||
|
|
- **Instrumental-only mode**: the provider writes an empty `.lrc` file when
|
|||
|
|
no lyrics are given. Verify this actually produces an instrumental for
|
|||
|
|
whatever DiffRhythm version you land on — vocal-less generation has been
|
|||
|
|
handled differently across releases; check `infer/infer.py` if results look
|
|||
|
|
off.
|
|||
|
|
- **Vocal quality vs. Suno**: DiffRhythm is a strong open-source result, not
|
|||
|
|
a drop-in Suno replacement. Expect rougher vocal fidelity, occasional
|
|||
|
|
lyric/timing drift, and more sensitivity to how the style prompt is
|
|||
|
|
worded. Iterate on `DIFFRHYTHM_DEFAULT_STYLE_PROMPT` and per-song prompts.
|
|||
|
|
- **Checkpoint choice**: `-full` unlocks up to 4m45s songs; `-base` is
|
|||
|
|
faster/lighter but caps at 95s. Match `DIFFRHYTHM_MAX_DURATION` to whichever
|
|||
|
|
you configure DiffRhythm to load.
|