TensorRT-LLM
NVIDIA optimized LLM inference — maximum speed on NVIDIA GPUs
📋 Overview
TensorRT-LLM is NVIDIA optimized inference engine for LLMs — providing maximum speed on NVIDIA GPUs through kernel fusion, quantization, and memory optimization. Unlike vLLM (which is GPU-agnostic), TensorRT-LLM is NVIDIA-specific — delivering the best possible performance on A100, H100, and RTX GPUs.
✨ Key Features
- •10K+ GitHub stars — NVIDIA optimized inference
- •Maximum speed on NVIDIA GPUs
- •Kernel fusion and quantization
- •Tensor parallelism
- •In-flight batching
- •Apache 2.0 licensed
🎯 The Problem It Solves
LLM inference is slow and expensive. TensorRT-LLM provides maximum speed on NVIDIA GPUs through deep optimization.
🔧 How It Works
TensorRT-LLM compiles LLM models into optimized TensorRT engines — fusing kernels, quantizing weights, and optimizing memory. It supports tensor parallelism, in-flight batching, and multiple quantization schemes. The result is the fastest inference possible on NVIDIA hardware.
🚀 Installation & Quick Start
Installation
See websiteQuick Start
- See documentation
✅ Pros
- •Fastest NVIDIA inference
- •Deep optimization
- •Apache 2.0 licensed
- •Active community
- •Comprehensive documentation
- •Production-ready
❌ Cons
- •NVIDIA only
- •Complex setup
- •Compilation takes time
💬 Practitioner Verdict
“TensorRT-LLM is the fastest LLM inference engine on NVIDIA GPUs — the optimization is genuinely differentiated. The trade-off: NVIDIA-only, complex setup, and compilation takes time. For teams that need maximum inference speed on NVIDIA hardware, TensorRT-LLM is the default.”
Self-Hosted (Free)
Open source, MIT/Apache licensed. Run it yourself.
⭐ Star & Clone on GitHubFree forever. Your infrastructure, your data.
📊 Specifications
- Language
- Python/C++
- License
- Apache-2.0
- Platform
- Linux, macOS, Windows
- Supported Models
- REST API, CLI
💰 Pricing Reality
100% free Apache 2.0. No paid tier.