tensorrt-llm
Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi-GPU scaling.
Install this skill
Works with Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, OpenCode, and 70+ other harnesses.
npx skills add KalarisLabs/research-agent-skills --skill tensorrt-llmuvx research-agent-skills install tensorrt-llmDocumentation & Instructions
TensorRT-LLM
NVIDIA's open-source library for optimizing LLM inference with state-of-the-art performance on NVIDIA GPUs.
When to use TensorRT-LLM
Use TensorRT-LLM when:
- Deploying on NVIDIA GPUs (A100, H100, GB200)
- Need maximum throughput (24,000+ tokens/sec on Llama 3)
- Require low latency for real-time applications
- Working with quantized models (FP8, INT4, FP4)
- Scaling across multiple GPUs or nodes
Use vLLM instead when:
- Need simpler setup and Python-first API
- Want PagedAttention without TensorRT compilation
- Working with AMD GPUs or non-NVIDIA hardware
Use llama.cpp instead when:
- Deploying on CPU or Apple Silicon
- Need edge deployment without NVIDIA GPUs
- Want simpler GGUF quantization format
Quick start
Installation
# Docker (recommended)
docker pull nvidia/tensorrt_llm:latest
# pip install
pip install tensorrt_llm==1.2.0rc3
# Requires CUDA 13.0.0, TensorRT 10.13.2, Python 3.10-3.12
Basic inference
from tensorrt_llm import LLM, SamplingParams
# Initialize model
llm = LLM(model="meta-llama/Meta-Llama-3-8B")
# Configure sampling
sampling_params = SamplingParams(
max_tokens=100,
temperature=0.7,
top_p=0.9
)
# Generate
prompts = ["Explain quantum computing"]
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
print(output.text)
Serving with trtllm-serve
# Start server (automatic model download and compilation)
trtllm-serve meta-llama/Meta-Llama-3-8B \
--tp_size 4 \ # Tensor parallelism (4 GPUs)
--max_batch_size 256 \
--max_num_tokens 4096
# Client request
curl -X POST http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Meta-Llama-3-8B",
"messages": [{"role": "user", "content": "Hello!"}],
"temperature": 0.7,
"max_tokens": 100
}'
Key features
Performance optimizations
- In-flight batching: Dynamic batching during generation
- Paged KV cache: Efficient memory management
- Flash Attention: Optimized attention kernels
- Quantization: FP8, INT4, FP4 for 2-4× faster inference
- CUDA graphs: Reduced kernel launch overhead
Parallelism
- Tensor parallelism (TP): Split model across GPUs
- Pipeline parallelism (PP): Layer-wise distribution
- Expert parallelism: For Mixture-of-Experts models
- Multi-node: Scale beyond single machine
Advanced features
- Speculative decoding: Faster generation with draft models
- LoRA serving: Efficient multi-adapter deployment
- Disaggregated serving: Separate prefill and generation
Common patterns
Quantized model (FP8)
from tensorrt_llm import LLM
# Load FP8 quantized model (2× faster, 50% memory)
llm = LLM(
model="meta-llama/Meta-Llama-3-70B",
dtype="fp8",
max_num_tokens=8192
)
# Inference same as before
outputs = llm.generate(["Summarize this article..."])
Multi-GPU deployment
# Tensor parallelism across 8 GPUs
llm = LLM(
model="meta-llama/Meta-Llama-3-405B",
tensor_parallel_size=8,
dtype="fp8"
)
Batch inference
# Process 100 prompts efficiently
prompts = [f"Question {i}: ..." for i in range(100)]
outputs = llm.generate(
prompts,
sampling_params=SamplingParams(max_tokens=200)
)
# Automatic in-flight batching for maximum throughput
Performance benchmarks
Meta Llama 3-8B (H100 GPU):
- Throughput: 24,000 tokens/sec
- Latency: ~10ms per token
- vs PyTorch: 100× faster
Llama 3-70B (8× A100 80GB):
- FP8 quantization: 2× faster than FP16
- Memory: 50% reduction with FP8
Supported models
- LLaMA family: Llama 2, Llama 3, CodeLlama
- GPT family: GPT-2, GPT-J, GPT-NeoX
- Qwen: Qwen, Qwen2, QwQ
- DeepSeek: DeepSeek-V2, DeepSeek-V3
- Mixtral: Mixtral-8x7B, Mixtral-8x22B
- Vision: LLaVA, Phi-3-vision
- 100+ models on HuggingFace
References
- Optimization Guide - Quantization, batching, KV cache tuning
- Multi-GPU Setup - Tensor/pipeline parallelism, multi-node
- Serving Guide - Production deployment, monitoring, autoscaling
Resources
- Docs: https://nvidia.github.io/TensorRT-LLM/
- GitHub: https://github.com/NVIDIA/TensorRT-LLM
- Models: https://huggingface.co/models?library=tensorrt_llm
Agent operating procedure
- Check the environment. Confirm hardware, framework and server versions, model format, and expected load.
- Pin down the inputs. Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.
- Run a small version first. Serve or log a single request or run end to end before scaling.
- Execute the full task using the instructions and references above.
- Validate the result. Measure latency, throughput and output correctness against a reference; check resource usage and costs.
- Report. State what was run (versions, commands, parameters), what was checked, and what is still uncertain.
| If this happens | Do this |
|---|---|
| The server fails to start or OOMs | Check model size versus memory, quantization and parallelism settings. |
| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (help(), --help, official docs), adapt, and tell the user. Never invent an API. |
| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |
Integrity rules
- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.
- Do not expose services or credentials publicly; confirm cloud costs before provisioning.
- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.
- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.
Related skills
serving-llms-vllm: Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching.miles-rl-training: Provides guidance for enterprise-grade RL training using miles, a production-ready fork of slime.llama-cpp: Runs LLM inference on CPU, Apple Silicon, and consumer GPUs without NVIDIA hardware.
Trigger Prompts & Evaluation Corpus
These evaluation prompts test that the agent correctly selects tensorrt-llm for relevant tasks and leaves it out for near misses.
Should trigger (3)
- should trigger
“How do I serve Llama 3 70B with TensorRT-LLM on 8 H100s?”
expects Gives a TensorRT-LLM workflow (trtllm-serve or the LLM API) with tensor_parallel_size=8, optionally FP8 quantization. It first checks the installed version, GPU memory and model format, tests with a small request before scaling, and says that flags may differ by version.
- should trigger
“I'm trying to deploy meta-llama/Meta-Llama-3-8B-Instruct on a single H100 for our lab's chat tool, and p95 latency is too high with plain HF transformers generate(). I built an engine with trtllm-build but the server throws a CUDA out of memory error when I send 64 concurrent requests. Can you help me fix it and get the quantization settings right?”
expects Diagnoses the OOM by checking model size against GPU memory, the KV cache fraction, max batch size and max sequence length. It suggests FP8 quantization, in-flight batching and paged KV cache tuning, and confirms the TensorRT-LLM version and exact engine build parameters. It asks for or states assumptions about the load, then validates latency, throughput and output correctness against a baseline without inventing benchmark numbers.
- should trigger
“We have a cluster of A100 80GB nodes and need to serve a 70B model to our research group with the lowest possible time per token and the highest tokens per second, using NVIDIA's compiled-engine approach with FP8 weights. What should the setup look like, and how do we check that the quantized model is still accurate?”
expects Recommends compiling TensorRT engines with FP8 quantization, noting that A100 lacks native FP8 support (H100 or newer is needed), so it proposes INT8/INT4 alternatives or confirms the hardware. It covers tensor/pipeline parallelism, in-flight batching, benchmarking latency and throughput, and comparing outputs or perplexity with an FP16 reference. It asks about expected load and reports uncertainty.
Near misses (2)
- near miss
“I want to run a quantized Mistral 7B GGUF on my MacBook M2 for offline use. How do I set that up?”
expects Recommends llama.cpp (GGUF, Apple Silicon Metal) and not TensorRT-LLM, because TensorRT-LLM requires NVIDIA GPUs. It gives steps to pick a quantization level and run the model.
- near miss
“I need an OpenAI-compatible endpoint for Qwen2-7B with AWQ quantization, and I'd like the simplest Python-first setup without compiling engines. What should I use?”
expects Recommends vLLM, which has a simple Python-first setup, an OpenAI-compatible server and AWQ support, with no TensorRT compilation step. It gives a short serve command and checks the vLLM version and GPU memory.