vLLM

GitHub - vllm-project/vllm: A high-throughput and memory-efficient inference and serving engine for LLMs

Welcome to vLLM! — vLLM

vLLM is a fast and easy-to-use library for LLM inference and serving.

vLLM is fast with:

• State-of-the-art serving throughput

• Efficient management of attention key and value memory with PagedAttention

• Continuous batching of incoming requests

• Fast model execution with CUDA/HIP graph

• Quantizations: GPTQ, AWQ, INT4, INT8, and FP8.

• Optimized CUDA kernels, including integration with FlashAttention and FlashInfer.

• Speculative decoding

• Chunked prefill

• More on Overview

  Performance benchmark: We include a performance benchmark at the end of our blog post. It compares the performance of vLLM against other LLM serving engines (TensorRT-LLM, SGLang and LMDeploy). The implementation is under nightly-benchmarks folder and you can reproduce this benchmark using our one-click runnable script.

  vLLM is flexible and easy to use with:

  • Seamless integration with popular Hugging Face models
  • High-throughput serving with various decoding algorithms, including parallel sampling, beam search, and more
  • Tensor parallelism and pipeline parallelism support for distributed inference
  • Streaming outputs
  • OpenAI-compatible API server
  • Support NVIDIA GPUs, AMD CPUs and GPUs, Intel CPUs and GPUs, PowerPC CPUs, TPU, and AWS Neuron.
  • Prefix caching support
  • Multi-lora support

  vLLM seamlessly supports most popular open-source models on HuggingFace, including:

  • Transformer-like LLMs (e.g., Llama)
  • Mixture-of-Expert LLMs (e.g., Mixtral)
  • Embedding Models (e.g. E5-Mistral)
  • Multi-modal LLMs (e.g., LLaVA)

  Find the full list of supported models here.

---

Assessment: Trial

I’ve heard about vLLM several months ago, and I’ve listed it here because it deserves a proper trial and an exhaustive one at that since the engine is shaping up to be the de-facto standard for hosting an inference engine in the enterprise.

It’s not as if the other big projects are unable to, but it is the one that supports the most hardware and has the most potential for being configured as a backend service that scales.