TensorRT-LLM is NVIDIA's open-source library for fast LLM and visual generation inference on NVIDIA GPUs. It offers a Python LLM API for single-GPU, multi-GPU and multi-node setups.
It includes custom kernels for attention, GEMMs and MoE operations, and runtime optimizations such as prefill-decode disaggregation, wide expert parallelism and speculative decoding. Pre-defined models can be edited in native PyTorch, and it integrates with NVIDIA Dynamo and Triton Inference Server.
See the Installation and Quick Start guides at nvidia.github.io/TensorRT-LLM. It is Apache 2.0 licensed and collects anonymous usage data by default; opt out with TRTLLM_NO_USAGE_STATS=1.