Text Generation Inference (TGI) is Hugging Face's server for running large language models, written in Rust, Python and gRPC. It provides tensor parallelism for multi-GPU inference, continuous batching, token streaming over Server-Sent Events and an OpenAI Chat Completion compatible Messages API, and launches popular models such as Llama, Falcon and StarCoder.
Run it with Docker (--gpus all, port 8080, --model-id <model>) or install from source with make install and start text-generation-launcher. It is Apache-2.0 licensed.
The repository is now archived and in maintenance mode, accepting only minor bug fixes and documentation updates, so for new projects consider vLLM, SGLang or TensorRT-LLM.