axolotl
Provides guidance for fine-tuning large language models with Axolotl, covering YAML training configs, LoRA and QLoRA, preference training with DPO, KTO, ORPO and GRPO, multimodal models, FSDP and multi-GPU setups, sequence (context) parallelism, NCCL bandwidth tests, dataset formats, compressed model saving for vLLM and llmcompressor, and custom integrations. Use when writing or debugging an Axolotl YAML config, choosing a dataset format for a fine-tuning run, setting up FSDP or context_parallel_size across GPUs, or running DPO, KTO, ORPO or GRPO training. Use when saving a compressed model for vLLM inference, or when writing a custom Axolotl plugin or integration. Not for inference serving or general Hugging Face Trainer scripts outside Axolotl.
Install this skill
Works with Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, OpenCode, and 70+ other harnesses.
npx skills add KalarisLabs/research-agent-skills --skill axolotluvx research-agent-skills install axolotlDocumentation & Instructions
Axolotl Skill
Comprehensive assistance with axolotl development, generated from official documentation.
When to Use This Skill
This skill should be triggered when:
- Working with axolotl
- Asking about axolotl features or APIs
- Implementing axolotl solutions
- Debugging axolotl code
- Learning axolotl best practices
Quick Reference
Common Patterns
Pattern 1: To validate that acceptable data transfer speeds exist for your training job, running NCCL Tests can help pinpoint bottlenecks, for example:
./build/all_reduce_perf -b 8 -e 128M -f 2 -g 3
Pattern 2: Configure your model to use FSDP in the Axolotl yaml. For example:
fsdp_version: 2
fsdp_config:
offload_params: true
state_dict_type: FULL_STATE_DICT
auto_wrap_policy: TRANSFORMER_BASED_WRAP
transformer_layer_cls_to_wrap: LlamaDecoderLayer
reshard_after_forward: true
Pattern 3: The context_parallel_size should be a divisor of the total number of GPUs. For example:
context_parallel_size
Pattern 4: For example: - With 8 GPUs and no sequence parallelism: 8 different batches processed per step - With 8 GPUs and context_parallel_size=4: Only 2 different batches processed per step (each split across 4 GPUs) - If your per-GPU micro_batch_size is 2, the global batch size decreases from 16 to 4
context_parallel_size=4
Pattern 5: Setting save_compressed: true in your configuration enables saving models in a compressed format, which: - Reduces disk space usage by approximately 40% - Maintains compatibility with vLLM for accelerated inference - Maintains compatibility with llmcompressor for further optimization (example: quantization)
save_compressed: true
Pattern 6: Note It is not necessary to place your integration in the integrations folder. It can be in any location, so long as it’s installed in a package in your python env. See this repo for an example: https://github.com/axolotl-ai-cloud/diff-transformer
integrations
Pattern 7: Handle both single-example and batched data. - single example: sample[‘input_ids’] is a list[int] - batched data: sample[‘input_ids’] is a list[list[int]]
utils.trainer.drop_long_seq(sample, sequence_len=2048, min_sequence_len=2)
Example Code Patterns
Example 1 (python):
cli.cloud.modal_.ModalCloud(config, app=None)
Example 2 (python):
cli.cloud.modal_.run_cmd(cmd, run_folder, volumes=None)
Example 3 (python):
core.trainers.base.AxolotlTrainer(
*_args,
bench_data_collator=None,
eval_data_collator=None,
dataset_tags=None,
**kwargs,
)
Example 4 (python):
core.trainers.base.AxolotlTrainer.log(logs, start_time=None)
Example 5 (python):
prompt_strategies.input_output.RawInputOutputPrompter()
Reference Files
This skill includes comprehensive documentation in references/:
- api.md - Api documentation
- dataset-formats.md - Dataset-Formats documentation
- other.md - Other documentation
Use view to read specific reference files when detailed information is needed.
Working with This Skill
For Beginners
Start with the getting_started or tutorials reference files for foundational concepts.
For Specific Features
Use the appropriate category reference file (api, guides, etc.) for detailed information.
For Code Examples
The quick reference section above contains common patterns extracted from the official docs.
Resources
references/
Organized documentation extracted from official sources. These files contain:
- Detailed explanations
- Code examples with language annotations
- Links to original documentation
- Table of contents for quick navigation
scripts/
Add helper scripts here for common automation tasks.
assets/
Add templates, boilerplate, or example projects here.
Notes
- This skill was automatically generated from official documentation
- Reference files preserve the structure and examples from source docs
- Code examples include language detection for better syntax highlighting
- Quick reference patterns are extracted from common usage examples in the docs
Updating
To refresh this skill with updated documentation:
- Re-run the scraper with the same configuration
- The skill will be rebuilt with the latest information
Agent operating procedure
- Check the environment. Check GPU type, memory and driver/CUDA versions (
nvidia-smi), framework versions, and dataset location and size. - Pin down the inputs. Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.
- Run a small version first. Do a smoke run: tiny model or subset, few steps, and confirm loss decreases and checkpoints save.
- Execute the full task using the instructions and references above.
- Validate the result. Track metrics on held-out data, compare against a baseline, and record seeds, configs and hardware.
- Report. State what was run (versions, commands, parameters), what was checked, and what is still uncertain.
| If this happens | Do this |
|---|---|
| CUDA out-of-memory | Reduce batch size, enable gradient accumulation/checkpointing or mixed precision, or shard the model. |
| Loss is NaN or diverges | Lower the learning rate, check data for invalid values, and enable gradient clipping. |
| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (help(), --help, official docs), adapt, and tell the user. Never invent an API. |
| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |
Integrity rules
- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.
- Never claim training results without logs; estimate compute cost before launching large jobs.
- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.
- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.
Related skills
llama-factory: Guides fine-tuning of large language models with LLaMA-Factory, covering the WebUI no-code interface, training across 100+ supported models…unsloth: Provides guidance on fine-tuning large language models with Unsloth, a library for faster, lower-memory training using LoRA and QLoRA, base…peft-fine-tuning: Fine-tunes LLMs with Hugging Face PEFT, using LoRA, QLoRA, IA3, AdaLoRA, prefix tuning, and prompt tuning so that under 1% of parameters ar…
Trigger Prompts & Evaluation Corpus
These evaluation prompts test that the agent correctly selects axolotl for relevant tasks and leaves it out for near misses.
Should trigger (3)
- should trigger
“my axolotl run dies with OOM on 2x A100 40GB when I turn on 8k context. how do I split sequences across the two GPUs?”
expects Explains sequence/context parallelism via context_parallel_size in the Axolotl YAML, which must divide the GPU count (here 2). Notes that the effective global batch size shrinks because each batch is split across GPUs, and suggests checking the installed version's docs and doing a small smoke run first.
- should trigger
“I'm fine-tuning Llama-3.1-8B with QLoRA in Axolotl on a JSONL file of {"prompt", "chosen", "rejected"} rows at data/prefs.jsonl. Can you write me a config for DPO and tell me which dataset type to set? I get 'KeyError: messages' right now.”
expects Provides an Axolotl YAML with rl: dpo, adapter: qlora, 4-bit loading and a datasets entry whose type matches the chosen/rejected format. Explains the KeyError as a mismatch between the data columns and the dataset type or format, points to the dataset-formats reference, and recommends a few-step smoke run first.
- should trigger
“I trained a model and want to ship a smaller checkpoint to serve with vLLM, and maybe quantize it further later with llmcompressor. How can I save it so the files take less disk space while still loading in those tools?”
expects Recommends setting save_compressed: true in the Axolotl config, which cuts disk use by roughly 40% and stays compatible with vLLM and llmcompressor. Cautions that the exact figure and option behaviour should be verified against the installed version.
Near misses (2)
- near miss
“I want to do GRPO on Qwen2.5-1.5B with my own correctness and format reward functions using TRL's GRPOTrainer and Unsloth. How should I structure the rewards?”
expects Recognises this as TRL GRPOTrainer with Unsloth, not Axolotl. Should route to grpo-rl-training for reward composition and chat-format datasets.
- near miss
“Which LoRA rank, alpha and target_modules should I use if I'm loading and merging adapters with Hugging Face PEFT directly in my own training script?”
expects Recognises this as plain PEFT usage outside Axolotl and routes to peft-fine-tuning for rank, alpha, target module selection and adapter merging.