AI for Developers
Text Generation Inference logo

Text Generation Inference

Visit Website

Hugging Face's Rust, Python and gRPC server for serving LLMs with continuous batching and streaming. The repo is in maintenance mode, so consider vLLM or SGLang for new projects.

Text Generation Inference (TGI) is Hugging Face's server for running large language models, written in Rust, Python and gRPC. It provides tensor parallelism for multi-GPU inference, continuous batching, token streaming over Server-Sent Events and an OpenAI Chat Completion compatible Messages API, and launches popular models such as Llama, Falcon and StarCoder.

Run it with Docker (--gpus all, port 8080, --model-id <model>) or install from source with make install and start text-generation-launcher. It is Apache-2.0 licensed.

The repository is now archived and in maintenance mode, accepting only minor bug fixes and documentation updates, so for new projects consider vLLM, SGLang or TensorRT-LLM.

Text Generation Inference reviews

  • No reviews yet.
See something outdated? Suggest an update