vllm-project

vllm-project/llm-compressor

State-of-the-art LLM compression, built for production inference with vLLM

View on GitHub Website / Demo Back to listing

3.9kstars 727forks 100+contributors Python

README.md

[![docs](https://img.shields.io/badge/docs-LLM--Compressor-blue)](https://docs.vllm.ai/projects/llm-compressor/en/latest/) [![PyPI](https://img.shields.io/pypi/v/llmcompressor.svg)](https://pypi.org/project/llmcompressor/) [![slack](https://img.shields.io/badge/slack-llm--compressor-green)](https://inviter.co/vllm-slack) [![huggingface](https://img.shields.io/badge/huggingface-RedHatAI-red)](https://huggingface.co/RedHatAI)

</div>

<p align="center">
  <b>English</b> · <a href="README.zh.md">简体中文</a>
</p>

`llmcompressor` is the fast, efficient, and easy-to-use library for optimizing models for deployment with vLLM, including:

* Comprehensive set of quantization algorithms and transforms for weight, activation, KV cache, and attention quantization
* Seamless integration with Hugging Face models and repositories
* Models saved in the `compressed-tensors` format, compatible with vLLM
* DDP and disk offloading support for compressing very large models with hardware efficiency

**✨ Read the announcement blog [here](https://neuralmagic.com/blog/llm-compressor-is-here-faster-inference-with-vllm/)! ✨**

---

💬 Join us on the [vLLM Community Slack](https://inviter.co/vllm-slack) and share your questions, thoughts, or ideas in:

- `#sig-quantization`
- `#llm-compressor`

---
## 🚀 What's New!

Big updates have landed in LLM Compressor! To get a more in-depth look, check out the [LLM Compressor overview](https://docs.google.com/presentation/d/1WNkYBKv_CsrYs69lb7bJKjh2dWt8U1HXUw7Gr4Wn3gE/edit?usp=sharing).

Since the v0.14.0 release, a number of meaningful improvements have landed:

* **HIGGS mixed-precision quantization**: [HIGGS](https://arxiv.org/pdf/2411.17525) collects layer-wise MSE sensitivity data and formulates an integer linear program to select an optimal mixed-precision configuration for a target average bitwidth. The resulting config can be passed to `oneshot` or `model_free_ptq`.
* **Block-wise QAD**: A new `QADModifier` runs quantization-aware distillation after a weight-quantization modifier in the same recipe. It caches calibration inputs and the unquantized subgraph output as the teacher, then trains the quantized weights through compressed-tensors fake quantization, without loading a separate teacher model.
* **Layerwise compression and sequential offloading**: The sequential pipeline can now decompress and compress one layer at a time, avoiding upfront whole-model conversion for large checkpoints. Onload/offload is managed manually with optional CPU staging and pinned memory.
* **REAP reports and non-uniform pruning**: REAP now writes per-expert saliency reports, supports calibration-only report generation with `sparsity=0.0`, and accepts per-layer non-uniform expert sparsity. `n_routed_experts` is now recorded in config names, and GPT-OSS is skipped.
* **MoNe expert pruning**: MoNe collapses low-saliency experts into constant "novice" vectors instead of removing them, keeping the router at full width while reducing expert compute.
* **Faster observers for E8M0 formats**: The MSE and iMatrix observers gained exponential grid search and magnitude normalization for E8M0 scales, and the iMatrix observer now shares the MSE Triton kernel, with roughly 1.2-3.4x end-to-end speedups on tested models.
* **MFPTQ SpinQuant rotations**: `SpinQuantConverter` applies offline R1/R2 Hadamard rotations directly to safetensors before model-free quantization, and `model_free_ptq` now accepts converter chains.
* **MXFP6 and MXFP4xFP8 support**: Added MXFP6A8/MXFP6A16 schemes and examples, plus an MXFP4xFP8 example and refreshed quantization documentation.
* **Broader hardware and model support**: The fused GPTQ Triton kernel now runs on Intel XPU, MTP layers load through Transformers `MtpModel`, and new checkpoints and examples include Nemotron 3.5 Super VL FP8, Zyphra Zaya1, Muse Spark, and Qwen3.6 MXFP4xFP8.
* **Docs and community**: Added the Open Research Hub for community experiment results and a full Simplified Chinese README translation.

### Model highlights

The Red Hat AI team has been using LLM Compressor to produce a fresh batch of production-ready quantized checkpoints:

* **GLM-5.3 MXFP4xFP8**: A mixed-precision quantized checkpoint for [GLM-5.3](https://huggingface.co/zai-org/GLM-5.3) that assigns MXFP4 or FP8 precision per layer, averaging ~4.5 bits per parameter. Routed experts and dense MLPs use MXFP4 weights with per-group FP8 dynamic activations, while attention, shared experts, and the multi-token prediction layer remain FP8 block-wise. This reduces the checkpoint from ~1507 GB to ~415 GB (~72% smaller), and the checkpoint is optimized for Hopper GPUs and supported in vLLM v0.32+.
  - [RedHatAI/GLM-5.3-MXFP4xFP8](https://huggingface.co/RedHatAI/GLM-5.3-MXFP4xFP8)
* **Nemotron 3.5 Super VL FP8 Checkpoint**: An FP8 quantized checkpoint for [Nemotron 3.5 Super VL](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16). Weights, activations and MTP layers are all quantized to FP8. See the model card for full agentic metrics and performance benchmarks:
  - [RedHatAI/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-FP8](https://huggingface.co/RedHatAI/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-FP8)
* **GLM-5.3 FP4 Checkpoints**: NVFP4 and MXFP4 quantized checkpoints for [GLM-5.3](https://huggingface.co/zai-org/GLM-5.3). The linear operators within the transformer blocks are quantized to FP4, while the MoE router, embeddings, DSA indexer, and output head are kept in their original precision to maintain accuracy recovery.
  - [RedHatAI/GLM-5.3-MXFP4](https://huggingface.co/RedHatAI/GLM-5.3-MXFP4)
  - [RedHatAI/GLM-5.3-NVFP4](https://huggingface.co/RedHatAI/GLM-5.3-NVFP4)
  - [GLM-5.3 MXFP4 Example](examples/model_free_ptq/glm_5_3_mxfp4.py)
* **GLM-5.3-Flash NVFP4**: An NVFP4 quantized checkpoint for [GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash). The expert layers are quantized to NVFP4, while the MTP (multi-token prediction) layers are quantized to per-block FP8.
  - [RedHatAI/GLM-5.3-Flash-NVFP4](https://huggingface.co/RedHatAI/GLM-5.3-Flash-NVFP4)
  - [GLM-5.3-Flash NVFP4 Example](examples/quantizing_moe/glm53_flash_example.py)
* **Qwen3.8-Flash-Next NVFP4**: An NVFP4 quantized checkpoint for [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next).
  - [RedHatAI/Qwen3.8-Flash-Next-NVFP4](https://huggingface.co/RedHatAI/Qwen3.8-Flash-Next-NVFP4)
* **Qwen3.8-27B INT4, NVFP4, and MXFP4**: 4-bit quantized checkpoints for [Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) across three formats — INT4, NVFP4, and MXFP4 — covering a range of hardware and accuracy trade-offs.
  - [RedHatAI/Qwen3.8-27B-INT4](https://huggingface.co/RedHatAI/Qwen3.8-27B-INT4)
  - [RedHatAI/Qwen3.8-27B-NVFP4](https://huggingface.co/RedHatAI/Qwen3.8-27B-NVFP4)
  - [RedHatAI/Qwen3.8-27B-MXFP4](https://huggingface.co/RedHatAI/Qwen3.8-27B-MXFP4)
  - [Qwen3.8-27B INT4 Example](examples/quantization_w4a16/qwen3_8_gptq_awq_example.py)
  - [Qwen3.8-27B NVFP4 Example](examples/quantization_w4a4_fp4/qwen3_8_gptq_awq_example.py)
  - [Qwen3.8-27B MXFP4 Example](examples/quantization_w4a4_mxfp4/qwen3_8_gptq_awq_example.py)
* **Nemotron 3.5 Lightning FP8**: An FP8 quantized checkpoint for [Nemotron 3.5 Lightning](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16), created using GPTQ-based FP8 quantization.
  - [RedHatAI/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-FP8](https://huggingface.co/RedHatAI/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-FP8)
  - [Nemotron 3.5 Lightning FP8 Example](examples/quantization_w8a8_fp8/nemotron_3_5_lightning_example.py)
* **Qwen3.8-2.4T-A95B NVFP4, NVFP4+FP8, and REAP+NVFP4**: NVFP4 and NVFP4+FP8 quantized checkpoints for [Qwen3.8-2.4T-A95B](https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B), along with `REAP-25` and `REAP-50` variants that combine [REAP](https://arxiv.org/pdf/2510.13999) expert pruning (25% and 50% of the least-salient experts pruned prior to quantization) with NVFP4, further reducing VRAM requirements while maintaining accuracy recovery.
  - [RedHatAI/Qwen3.8-2.4T-A95B-NVFP4](https://huggingface.co/RedHatAI/Qwen3.8-2.4T-A95B-NVFP4)
  - [RedHatAI/Qwen3.8-2.4T-A95B-NVFP4-FP8](https://huggingface.co/RedHatAI/Qwen3.8-2.4T-A95B-NVFP4-FP8)
  - [RedHatAI/Qwen3.8-2.4T-A95B-NVFP4-REAP-25](https://huggingface.co/RedHatAI/Qwen3.8-2.4T-A95B-NVFP4-REAP-25)
  - [RedHatAI/Qwen3.8-2.4T-A95B-NVFP4-REAP-50](https://huggingface.co/RedHatAI/Qwen3.8-2.4T-A95B-NVFP4-REAP-50)
  - [Qwen3.8-2.4T-A95B NVFP4+FP8 Example](examples/quantizing_moe/qwen_3_8_example.py)
  - [Qwen3.8 REAP + NVFP4 Example](examples/reap_expert_pruning/qwen38_example.py)
* **Muse-Glimmer-30B FP8, NVFP4, and INT4**: FP8, NVFP4, and INT4 checkpoints for [Muse-Glimmer-30B](https://huggingface.co/meta-models/Muse-Glimmer-30B), enabling single-GPU deployment of this multimodal model.
  - [RedHatAI/Muse-Glimmer-30B-FP8-block](https://huggingface.co/RedHatAI/Muse-Glimmer-30B-FP8-block)
  - [RedHatAI/Muse-Glimmer-30B-NVFP4](https://huggingface.co/RedHatAI/Muse-Glimmer-30B-NVFP4)
  - [RedHatAI/Muse-Glimmer-30B-INT4](https://huggingface.co/RedHatAI/Muse-Glimmer-30B-INT4)
  - [Muse-Glimmer FP8_Block Example](examples/model_free_ptq/muse_glimmer_fp8_block.py)
* **Kimi-K3 NVFP4 and FP8**: NVFP4 and per-block FP8 quantized checkpoints for [Kimi-K3](https://huggingface.co/moonshotai/Kimi-K3). A native `KimiK3ForConditionalGeneration` model definition is now shipped in the library to support quantizing this architecture.
  - [RedHatAI/Kimi-K3-NVFP4](https://huggingface.co/RedHatAI/Kimi-K3-NVFP4)
  - [RedHatAI/Kimi-K3-FP8-BLOCK](https://huggingface.co/RedHatAI/Kimi-K3-FP8-BLOCK)
  - [Kimi-K3 Quantization Example](examples/quantizing_moe/kimi_k3_example.py)

### Supported Precisions and Types
* Activation Quantization: W8A8 (int8 and fp8), W4AFP8, Microscale (NVFP4, MXFP4, MXFP6A8, MXFP8)
* Mixed Precision: W4A16, W8A16, MXFP8A16, MXFP6A16, MXFP4A16, NVFP4A16
* Attention and KV Cache Quantization: FP8, NVFP4
* Low/Arbitrary-bit Quantization: WNA4, WNA8, WNA16

### Supported Algorithms
* Simple PTQ
* GPTQ
* AWQ
* SmoothQuant
* AutoRound
* Rotation-based (SpinQuant, QuIP)
* REAP expert pruning

### Quantizing your model, step-by-step

Please refer to our [step-by-step compression guide](https://docs.vllm.ai/projects/llm-compressor/en/latest/steps/choosing-model/) for detailed information about selecting quantization schemes, algorithms, and their use cases.

Additional information about LLM Compressor functionality is also available in our [User Guides](https://docs.vllm.ai/projects/llm-compressor/en/latest/guides/entrypoints/) and [FAQ](https://docs.vllm.ai/projects/llm-compressor/en/latest/faq/faq/).

## Installation

```bash
pip install llmcompressor
```

## Get Started

### End-to-End Examples

Applying quantization with `llmcompressor`:

### Weight and Activation Quantization
* [Activation quantization to `int8`](examples/quantization_w8a8_int8/README.md)
* [Activation quantization to `fp8`](examples/quantization_w8a8_fp8/README.md)
* [Activation quantization to MXFP8](examples/quantization_w8a8_mxfp8)
* [MXFP6 weights with MXFP8 activations](examples/quantization_w6a8_mxfp6)
* [Activation quantization to `fp4` (NVFP4)](examples/quantization_w4a4_fp4)
* [Activation quantization to `fp4` (MXFP4)](examples/quantization_w4a4_mxfp4)
* [Activation quantization to `fp4` using AutoRound](examples/autoround/quantization_w4a4_fp4/README.md)
* [Activation quantization to `fp8` and weight quantization to `int4`](examples/quantization_w4a8_fp8)

### Weight Only Quantization
* [Weight only quantization to `fp4` (NVFP4 format)](examples/quantization_w4a16_fp4/nvfp4)
* [Weight only quantization to `fp4` (MXFP4 format)](examples/quantization_w4a16_fp4/mxfp4)
* [Weight only quantization to `int4` using GPTQ](examples/quantization_w4a16/README.md)
* [Weight only quantization to `int4` using AWQ](examples/awq/README.md)
* [Weight only quantization with AutoRound (`wNa16`)](examples/autoround/quantization_wNa16/README.md)

### Attention and KV Cache Quantization
* [KV Cache quantization to `fp8`](examples/quantization_kv_cache/README.md)
* [KV Cache quantization to `fp8` using per-head](examples/quantization_kv_cache/llama3_fp8_head_kv_example.py)
* [Attention quantization to `fp8`](examples/quantization_attention/README.md)
* [Attention quantization to `NVFP4` with SpinQuant (experimental)](experimental/attention/README.md)

### Architecture-Specific Quantization
* [Quantizing MoE LLMs](examples/quantizing_moe/README.md)
* [Quantizing Vision-Language Models](examples/multimodal_vision/README.md)
* [Quantizing Audio-Language Models](examples/multimodal_audio/README.md)

### Non-Uniform Quantization
* [Quantizing Models Non-uniformly](examples/quantization_non_uniform/README.md)

### Big Model Quantization Support
* [Quantizing large models with sequential onloading](examples/big_models_with_sequential_onloading/README.md)
* [Quantizing large models with disk offloading](examples/disk_offloading/README.md)

### Model-Free Definition Quantization
* [Quantizing models without a Hugging Face model definition](examples/model_free_ptq/README.md)

### DDP Quantization
* [Distributed data parallel quantization with GPTQ](examples/quantization_w4a16/llama3_ddp_example.py)

## Quick Tour
Let's quantize `Qwen3-30B-A3B` with FP8 weights and activations using the `Round-to-Nearest` algorithm.

Note that the model can be swapped for a local or remote HF-compatible checkpoint and the `recipe` may be changed to target different quantization algorithms or formats.

### Apply Quantization
Quantization is applied by selecting an algorithm and calling the `oneshot` API.

```python
from compressed_tensors.offload import dispatch_model
from transformers import AutoModelForCausalLM, AutoTokenizer

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

MODEL_ID = "Qwen/Qwen3-30B-A3B"

# Load model.
model = AutoModelForCausalLM.from_pretrained(MODEL_ID)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

# Configure the quantization algorithm and scheme.
# In this case, we:
#   * quantize the weights to FP8 using RTN with block_size 128
#   * quantize the activations dynamically to FP8 during inference
recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_BLOCK",
    ignore=["lm_head", "re:.*mlp.gate$"],
)

# Apply quantization.
oneshot(model=model, recipe=recipe)

# Confirm generations of the quantized model look sane.
print("========== SAMPLE GENERATION ==============")
dispatch_model(model)
input_ids = tokenizer("Hello my name is", return_tensors="pt").input_ids.to(
    model.device
)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))
print("==========================================")

# Save to disk in compressed-tensors format.
SAVE_DIR = MODEL_ID.split("/")[1] + "-FP8-BLOCK"
model.save_pretrained(SAVE_DIR)
tokenizer.save_pretrained(SAVE_DIR)
```

### Inference with vLLM

The checkpoints created by `llmcompressor` can be loaded and run in `vllm`:

Install:

```bash
pip install vllm
```

Run:

```python
from vllm import LLM
model = LLM("Qwen/Qwen3-30B-A3B-FP8-BLOCK")
output = model.generate("My name is")
```

## Questions / Contribution

- If you have any questions or requests open an [issue](https://github.com/vllm-project/llm-compressor/issues) and we will add an example or documentation.
- We appreciate contributions to the code, examples, integrations, and documentation as well as bug reports and feature requests! [Learn how here](CONTRIBUTING.md).
- Browse community experiments and share your compression results in the [Open Research Hub](https://docs.vllm.ai/projects/llm-compressor/en/latest/research/). All results are welcome.

## Citation

If you find LLM Compressor useful in your research or projects, please consider citing it:

```bibtex
@software{llmcompressor2024,
    title={{LLM Compressor}},
    author={Red Hat AI and vLLM Project},
    year={2024},
    month={8},
    url={https://github.com/vllm-project/llm-compressor},
}
```

Repository info

Default branch: main
Latest version: 0.15.0
Created: 21/06/2024
Last activity: 04/11/2026
License: Apache-2.0
Open issues: 136
Watchers: 29

Topics

compression quantization

Contributors

kylesayrs

401 contributions

dsikka

302 contributions

markurtz

297 contributions

bfineran

293 contributions

rahul-tuli

209 contributions