ValueError: Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the quantized model. If you want to dispatch the model on the CPU or the disk while keeping these modules in 32-bit, you need to set llm_int8_enable_fp32_cpu_offload=True and pass a custom device_map to from_pretrained. Check https://huggingface.co/docs/transformers/main/en/main_classes/quantization#offload-between-cpu-and-gpu for more details.

 

hat error usually pops up when you try to load a quantized model (like 4-bit QLoRA) with Hugging Face’s transformers and bitsandbytes, but your GPU doesn’t have enough VRAM to hold all the model layers. By default, Hugging Face tries to put everything on the GPU, and when it can’t, it errors out.

---

🔹 Why the Error Happens

• The model is too large for your GPU (e.g., a 13B model on 8GB VRAM).

• Some layers got “dispatched” (offloaded) onto CPU or disk.

• You didn’t explicitly allow CPU offloading, so transformers raised a ValueError.

---

🔹 Solutions

✅ Option 1: Enable CPU Offload (Recommended if low VRAM)

You can allow the model to split between GPU + CPU:

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_id = "mistralai/Mistral-7B-Instruct-v0.2"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype="bfloat16"
)

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto",  # automatically spread across GPU + CPU
    llm_int8_enable_fp32_cpu_offload=True  # ✅ allows CPU fallback in FP32
)

---

✅ Option 2: Force Specific Device Map

If you want more control, specify which parts go where:

device_map = {
    "transformer.wte": "cuda:0",   # embeddings on GPU
    "transformer.h": "cpu",        # some layers on CPU
    "lm_head": "cuda:0"
}

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map=device_map,
    llm_int8_enable_fp32_cpu_offload=True
)

---

✅ Option 3: Use a Smaller Model

If you only have ~8–12 GB VRAM, try:

• Mistral-7B-Instruct-v0.1 instead of v0.2

• Or even smaller models (LLaMA-2-7B, Gemma-2B, etc.).

---

✅ Option 4: Use accelerate for Better Device Placement

pip install accelerate

Then run:

from accelerate import init_empty_weights, load_checkpoint_and_dispatch

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto"
)

This lets accelerate decide where to put layers across GPU/CPU/Disk.

---

🔹 Key Takeaway

• If GPU VRAM < model size → must offload to CPU/disk.

• Add llm_int8_enable_fp32_cpu_offload=True and/or device_map="auto".

• Or use a smaller model to fit fully in GPU.