Skip to content
Research Agent Skillsby Kalaris Labs
Install
Multimodal and Emergingv1.0.0AdaptedMIT

audiocraft-audio-generation

PyTorch library for audio generation including text-to-music (MusicGen) and text-to-sound (AudioGen). Use when you need to generate music from text descriptions, create sound effects, or perform melody-conditioned music generation.

Install this skill

Works with Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, OpenCode, and 70+ other harnesses.

skills.sh CLI (Recommended)
npx skills add KalarisLabs/research-agent-skills --skill audiocraft-audio-generation
uvx (Python tool runner)
uvx research-agent-skills install audiocraft-audio-generation

Documentation & Instructions

AudioCraft: Audio Generation

Comprehensive guide to using Meta's AudioCraft for text-to-music and text-to-audio generation with MusicGen, AudioGen, and EnCodec.

When to use AudioCraft

Use AudioCraft when:

  • Need to generate music from text descriptions
  • Creating sound effects and environmental audio
  • Building music generation applications
  • Need melody-conditioned music generation
  • Want stereo audio output
  • Require controllable music generation with style transfer

Key features:

  • MusicGen: Text-to-music generation with melody conditioning
  • AudioGen: Text-to-sound effects generation
  • EnCodec: High-fidelity neural audio codec
  • Multiple model sizes: Small (300M) to Large (3.3B)
  • Stereo support: Full stereo audio generation
  • Style conditioning: MusicGen-Style for reference-based generation

Use alternatives instead:

  • Stable Audio: For longer commercial music generation
  • Bark: For text-to-speech with music/sound effects
  • Riffusion: For spectogram-based music generation
  • OpenAI Jukebox: For raw audio generation with lyrics

Quick start

Installation

# From PyPI
pip install audiocraft

# From GitHub (latest)
pip install git+https://github.com/facebookresearch/audiocraft.git

# Or use HuggingFace Transformers
pip install transformers torch torchaudio

Basic text-to-music (AudioCraft)

import torchaudio
from audiocraft.models import MusicGen

# Load model
model = MusicGen.get_pretrained('facebook/musicgen-small')

# Set generation parameters
model.set_generation_params(
    duration=8,  # seconds
    top_k=250,
    temperature=1.0
)

# Generate from text
descriptions = ["happy upbeat electronic dance music with synths"]
wav = model.generate(descriptions)

# Save audio
torchaudio.save("output.wav", wav[0].cpu(), sample_rate=32000)

Using HuggingFace Transformers

from transformers import AutoProcessor, MusicgenForConditionalGeneration
import scipy

# Load model and processor
processor = AutoProcessor.from_pretrained("facebook/musicgen-small")
model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small")
model.to("cuda")

# Generate music
inputs = processor(
    text=["80s pop track with bassy drums and synth"],
    padding=True,
    return_tensors="pt"
).to("cuda")

audio_values = model.generate(
    **inputs,
    do_sample=True,
    guidance_scale=3,
    max_new_tokens=256
)

# Save
sampling_rate = model.config.audio_encoder.sampling_rate
scipy.io.wavfile.write("output.wav", rate=sampling_rate, data=audio_values[0, 0].cpu().numpy())

Text-to-sound with AudioGen

from audiocraft.models import AudioGen

# Load AudioGen
model = AudioGen.get_pretrained('facebook/audiogen-medium')

model.set_generation_params(duration=5)

# Generate sound effects
descriptions = ["dog barking in a park with birds chirping"]
wav = model.generate(descriptions)

torchaudio.save("sound.wav", wav[0].cpu(), sample_rate=16000)

Core concepts

Architecture overview

AudioCraft Architecture:
┌──────────────────────────────────────────────────────────────┐
│                    Text Encoder (T5)                          │
│                         │                                     │
│                    Text Embeddings                            │
└────────────────────────┬─────────────────────────────────────┘
                         │
┌────────────────────────▼─────────────────────────────────────┐
│              Transformer Decoder (LM)                         │
│     Auto-regressively generates audio tokens                  │
│     Using efficient token interleaving patterns               │
└────────────────────────┬─────────────────────────────────────┘
                         │
┌────────────────────────▼─────────────────────────────────────┐
│                EnCodec Audio Decoder                          │
│        Converts tokens back to audio waveform                 │
└──────────────────────────────────────────────────────────────┘

Model variants

Model Size Description Use Case
musicgen-small 300M Text-to-music Quick generation
musicgen-medium 1.5B Text-to-music Balanced
musicgen-large 3.3B Text-to-music Best quality
musicgen-melody 1.5B Text + melody Melody conditioning
musicgen-melody-large 3.3B Text + melody Best melody
musicgen-stereo-* Varies Stereo output Stereo generation
musicgen-style 1.5B Style transfer Reference-based
audiogen-medium 1.5B Text-to-sound Sound effects

Generation parameters

Parameter Default Description
duration 8.0 Length in seconds (1-120)
top_k 250 Top-k sampling
top_p 0.0 Nucleus sampling (0 = disabled)
temperature 1.0 Sampling temperature
cfg_coef 3.0 Classifier-free guidance

MusicGen usage

Details, code examples and parameter tables: references/musicgen-usage.md. Read it when this step applies.

MusicGen-Style usage

Details, code examples and parameter tables: references/musicgen-style-usage.md. Read it when this step applies.

AudioGen usage

Sound effect generation

from audiocraft.models import AudioGen
import torchaudio

model = AudioGen.get_pretrained('facebook/audiogen-medium')
model.set_generation_params(duration=10)

# Generate various sounds
descriptions = [
    "thunderstorm with heavy rain and lightning",
    "busy city traffic with car horns",
    "ocean waves crashing on rocks",
    "crackling campfire in forest"
]

wav = model.generate(descriptions)

for i, audio in enumerate(wav):
    torchaudio.save(f"sound_{i}.wav", audio.cpu(), sample_rate=16000)

EnCodec usage

Audio compression

from audiocraft.models import CompressionModel
import torch
import torchaudio

# Load EnCodec
model = CompressionModel.get_pretrained('facebook/encodec_32khz')

# Load audio
wav, sr = torchaudio.load("audio.wav")

# Ensure correct sample rate
if sr != 32000:
    resampler = torchaudio.transforms.Resample(sr, 32000)
    wav = resampler(wav)

# Encode to tokens
with torch.no_grad():
    encoded = model.encode(wav.unsqueeze(0))
    codes = encoded[0]  # Audio codes

# Decode back to audio
with torch.no_grad():
    decoded = model.decode(codes)

torchaudio.save("reconstructed.wav", decoded[0].cpu(), sample_rate=32000)

Common workflows

Workflow 1: Music generation pipeline

import torch
import torchaudio
from audiocraft.models import MusicGen

class MusicGenerator:
    def __init__(self, model_name="facebook/musicgen-medium"):
        self.model = MusicGen.get_pretrained(model_name)
        self.sample_rate = 32000

    def generate(self, prompt, duration=30, temperature=1.0, cfg=3.0):
        self.model.set_generation_params(
            duration=duration,
            top_k=250,
            temperature=temperature,
            cfg_coef=cfg
        )

        with torch.no_grad():
            wav = self.model.generate([prompt])

        return wav[0].cpu()

    def generate_batch(self, prompts, duration=30):
        self.model.set_generation_params(duration=duration)

        with torch.no_grad():
            wav = self.model.generate(prompts)

        return wav.cpu()

    def save(self, audio, path):
        torchaudio.save(path, audio, sample_rate=self.sample_rate)

# Usage
generator = MusicGenerator()
audio = generator.generate(
    "epic cinematic orchestral music",
    duration=30,
    temperature=1.0
)
generator.save(audio, "epic_music.wav")

Workflow 2: Sound design batch processing

import json
from pathlib import Path
from audiocraft.models import AudioGen
import torchaudio

def batch_generate_sounds(sound_specs, output_dir):
    """
    Generate multiple sounds from specifications.

    Args:
        sound_specs: list of {"name": str, "description": str, "duration": float}
        output_dir: output directory path
    """
    model = AudioGen.get_pretrained('facebook/audiogen-medium')
    output_dir = Path(output_dir)
    output_dir.mkdir(exist_ok=True)

    results = []

    for spec in sound_specs:
        model.set_generation_params(duration=spec.get("duration", 5))

        wav = model.generate([spec["description"]])

        output_path = output_dir / f"{spec['name']}.wav"
        torchaudio.save(str(output_path), wav[0].cpu(), sample_rate=16000)

        results.append({
            "name": spec["name"],
            "path": str(output_path),
            "description": spec["description"]
        })

    return results

# Usage
sounds = [
    {"name": "explosion", "description": "massive explosion with debris", "duration": 3},
    {"name": "footsteps", "description": "footsteps on wooden floor", "duration": 5},
    {"name": "door", "description": "wooden door creaking and closing", "duration": 2}
]

results = batch_generate_sounds(sounds, "sound_effects/")

Workflow 3: Gradio demo

import gradio as gr
import torch
import torchaudio
from audiocraft.models import MusicGen

model = MusicGen.get_pretrained('facebook/musicgen-small')

def generate_music(prompt, duration, temperature, cfg_coef):
    model.set_generation_params(
        duration=duration,
        temperature=temperature,
        cfg_coef=cfg_coef
    )

    with torch.no_grad():
        wav = model.generate([prompt])

    # Save to temp file
    path = "temp_output.wav"
    torchaudio.save(path, wav[0].cpu(), sample_rate=32000)
    return path

demo = gr.Interface(
    fn=generate_music,
    inputs=[
        gr.Textbox(label="Music Description", placeholder="upbeat electronic dance music"),
        gr.Slider(1, 30, value=8, label="Duration (seconds)"),
        gr.Slider(0.5, 2.0, value=1.0, label="Temperature"),
        gr.Slider(1.0, 10.0, value=3.0, label="CFG Coefficient")
    ],
    outputs=gr.Audio(label="Generated Music"),
    title="MusicGen Demo"
)

demo.launch()

Performance optimization

Memory optimization

# Use smaller model
model = MusicGen.get_pretrained('facebook/musicgen-small')

# Clear cache between generations
torch.cuda.empty_cache()

# Generate shorter durations
model.set_generation_params(duration=10)  # Instead of 30

# Use half precision
model = model.half()

Batch processing efficiency

# Process multiple prompts at once (more efficient)
descriptions = ["prompt1", "prompt2", "prompt3", "prompt4"]
wav = model.generate(descriptions)  # Single batch

# Instead of
for desc in descriptions:
    wav = model.generate([desc])  # Multiple batches (slower)

GPU memory requirements

Model FP32 VRAM FP16 VRAM
musicgen-small ~4GB ~2GB
musicgen-medium ~8GB ~4GB
musicgen-large ~16GB ~8GB

Common issues

Issue Solution
CUDA OOM Use smaller model, reduce duration
Poor quality Increase cfg_coef, better prompts
Generation too short Check max duration setting
Audio artifacts Try different temperature
Stereo not working Use stereo model variant

References

Resources

Agent operating procedure

  1. Check the environment. Confirm GPU memory, model checkpoint and licence, framework versions and input formats.
  2. Pin down the inputs. Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.
  3. Run a small version first. Run inference on a single sample at low resolution or short length first.
  4. Execute the full task using the instructions and references above.
  5. Validate the result. Inspect outputs qualitatively and with task metrics; check licence restrictions for generated content.
  6. Report. State what was run (versions, commands, parameters), what was checked, and what is still uncertain.
If this happens Do this
Out-of-memory on large inputs Reduce resolution or sequence length, use half precision, or tile the input.
A function, flag or endpoint in these instructions is missing in the installed version Check the installed version's own documentation (help(), --help, official docs), adapt, and tell the user. Never invent an API.
A required input, identifier or parameter is ambiguous Ask the user, or state the assumption explicitly before running.

Integrity rules

  • Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.
  • Check model and dataset licences before commercial or published use.
  • Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.
  • Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.

Related skills

  • transformers: Hugging Face Transformers for loading Hub models, running pipeline inference, text generation, and Trainer fine-tuning on NLP, vision, audi…

Trigger Prompts & Evaluation Corpus

These evaluation prompts test that the agent correctly selects audiocraft-audio-generation for relevant tasks and leaves it out for near misses.

Should trigger (3)

  • should trigger

    “I need a 15 second lo-fi hip hop clip with mellow piano and vinyl crackle. What's the quickest way to get that going locally?”

    expects Uses MusicGen (audiocraft or the HuggingFace Transformers route) with a small model, sets duration to about 15 seconds, and writes a text prompt. Checks GPU memory and installation first, and runs a short test generation before anything larger.

  • should trigger

    “I'm getting CUDA out of memory when I run MusicGen.get_pretrained('facebook/musicgen-large') with duration=30 on my 8GB RTX 3070 in generate_music.py. How do I fix this? I still want batches of different prompts for a user study stimulus set.”

    expects Explains that large needs about 16GB in FP32 and about 8GB in FP16, so it is too tight on 8GB. Recommends musicgen-small or medium, half precision, shorter duration, or smaller batches. Notes the licence check for the generated stimuli and gives a batching approach that fits the memory.

  • should trigger

    “For a perception experiment I need a set of short ambient clips, such as dog barking, rain on a window and a door slamming, all generated from written descriptions. I also have a reference 10 s humming recording that I want one of the music clips to follow. How should I set this up?”

    expects Uses AudioGen (audiogen-medium) for the sound effects from text. Uses a melody-conditioned MusicGen model (musicgen-melody) with the humming file as the melody input for the music clip. Mentions generation parameters such as duration, cfg_coef and temperature, and states the model licence and that the output should be checked.

Near misses (2)

  • near miss

    “Transcribe this 20 minute interview recording (interview.wav) to text with timestamps using Whisper from Hugging Face Transformers.”

    expects Recognises this as speech recognition, not audio generation. Uses the transformers ASR pipeline with Whisper, with chunking for long audio and timestamps, and does not use AudioCraft.

  • near miss

    “Write a Python function that computes the mean MFCC features from a folder of WAV files with librosa and saves them to a CSV.”

    expects Writes librosa feature-extraction code (load, MFCC, mean, CSV output). Does not use any generative audio model.