Complete Guide to the GGUF Format for AI Models

If you've delved into the world of local artificial intelligence, you've probably come across files that end in .ggufThis format allows you to distribute and run language models on home computers using tools such as llama.cpp, Ollama, LM Studio, or Jan.

Its popularity stems from the fact that it consolidates the model's weights, architecture, and various metadata necessary for loading it into one or more files. Furthermore, it supports quantized models that They take up less space and require less memory than their original versions in FP16 or FP32.

Thanks to this, a model that would originally require a professional GPU can run on a laptop or a standard desktop computer. This doesn't mean that any model is compatible with any device: you have to choose the right size, quantization, and context length.

The speed of the drive where it's stored also plays a role. Although GGUF is designed to promote efficient loading through techniques like memory mapping, a slow disk can still affect startup. If it takes too long, it's worth checking if The SSD is acting as a bottleneck when loading AI models.

What is the GGUF format and what is it used for?

GGUF is a binary format developed to store models for inference with GGML and engines based on this project. Its name comes from GGML Universal File and was created as a successor to earlier formats such as GGML, GGMF, and GGJT.

Its main objective is to provide a format extendable, easy to distribute and quick to loadA GGUF file can contain the model tensors along with information about its architecture, vocabulary, tokenizer, context, and other parameters necessary to interpret it correctly.

Its main features include:

• Storage of model weights and metadata.
• Compatibility with different levels and methods of quantization.
• Efficient loading through memory mapping or mmap.
• Possibility of expanding metadata without redesigning the entire format.
• Simple distribution of models prepared for inference.
• Compatibility with numerous architectures supported by llama.cpp.

GGUF is not a program that directly runs artificial intelligence. It is the container where the model is storedTo use it, you need a compatible engine, such as llama.cpp, or an application built on this ecosystem.

The evolution from GGML to GGUF

To understand the emergence of GGUF, one must go back to GGML, the library created by Georgi Gerganov to perform machine learning calculations efficiently, especially on CPUs and on devices with limited resources.

The initial formats associated with the project evolved as new architectures, quantizations, and parameters emerged. The problem was that some changes could create incompatibilities and force applications to interpret different structures.

GGUF introduced a clearer key-value pair structure for metadata. This allows new information to be added without necessarily disrupting the functionality of applications that don't need it. The result is a a more flexible container, ready to evolve.

Another advantage is that the information needed to recognize and load the model can be transmitted along with its weights. This reduces problems caused by missing external configurations, incorrect vocabularies, or parameters that don't match the downloaded model.

Even so, compatibility isn't guaranteed forever. If a new architecture appears, the engine used must incorporate support for it. That's why a recent GGUF file might not work in an older version of llama.cpp, Ollama, or LM Studio.

How does quantization of a GGUF model work?

Quantization involves representing the model's weights using fewer bits. An original model might store a large portion of its parameters in FP16 or BF16, while a quantized version uses 8, 6, 5, 4, 3, or even 2-bit representations for many of its tensors.

Reducing precision decreases both the file size and the memory required during inference. It can also improve speed when the hardware efficiently processes that type of quantization. In return, Some of the model's fidelity is lost.

GGUF does not automatically adjust quantization based on your computer. You will typically find different files for the same model, and you will need to download the one that best suits your RAM or VRAM.

These are some of the most common options:

• Q4_K_M: It usually offers a good balance between size, memory consumption, and quality. It's a reasonable choice to start with.
• Q5_K_M: It retains slightly more fidelity than Q4_K_M, but generates larger files and increases memory usage.
• Q6_K: It provides high quality when the equipment has sufficient memory and does not need to reduce the model so much.
• Q8_0: It maintains a quality very close to the more precise versions, although its memory savings are less.
• IQ4_XS: It seeks to reduce the size below other four-bit alternatives while maintaining reasonable quality.
• IQ3 and IQ2: They significantly reduce consumption, but the loss of quality may be more evident.

There is no universally best quantization. The result depends on the architecture, model size, task, and hardware. A large model quantized to three bits may outperform a much smaller one at Q8_0, although it may also produce less stable responses than a four- or five-bit quantization of the same model.

Therefore, it's not always correct to dismiss two- or three-bit versions. They can be useful when the alternative is not being able to load the model. However, for work where accuracy is paramount, it's usually preferable to use two-bit or three-bit versions. stay around Q4_K_M, Q5_K_M or higher levels.

How much memory does a GGUF model need?

The file size provides an initial reference point, but it doesn't represent all the memory required. During inference, space must also be reserved for the KV cache, context, computation buffers, and other engine components.

For example, an 8GB GGUF file doesn't guarantee it will run correctly on a GPU with exactly 8GB of VRAM. The program will need additional memory, and if it doesn't have enough, it will have to split part of the model between the GPU and RAM or run it using the CPU.

To calculate whether a model will work on your computer, you must take into account:

• The size of the GGUF file.
• Available RAM and VRAM.
• The configured context length.
• The size and quantization of the KV cache.
• The layers that will be processed by the GPU.
• The applications that remain open simultaneously.

As a general rule, it's advisable to leave some margin above the file size. If the model doesn't fit entirely on the GPU, llama.cpp and other tools allow you to adjust the file size. download only a portion of its layers to the graphics card and process the remaining ones using the CPU.

GGUF versus GPTQ, AWQ and other formats

GGUF is not the only way to distribute quantized models. There are also methods and formats such as GPTQ, AWQ, EXL2, and bitsandbytes, each with different engines, accelerators, and use cases.

One of GGUF's main strengths is its ability to work in CPU, GPU, or hybrid configurationsThis makes it especially useful in home computers, systems with little VRAM, ARM devices, and systems that use unified memory.

GPTQ, AWQ, and EXL2 are typically geared towards execution using specific GPUs and specialized engines. While they can offer excellent performance on some systems, they don't always provide the same flexibility in allocating resources between the processor, main memory, and graphics card.

The choice depends on the application you'll be using. If you're working with llama.cpp, Ollama, or LM Studio, GGUF is usually the most practical option. If you have a powerful NVIDIA GPU and are using a specialized engine, another method might provide better results in certain configurations.

Recommended tools for using GGUF files

You don't need to directly manage all the parameters of llama.cpp to get started. There are applications that simplify downloading, loading, and configuring the models.

LM Studio

LM Studio It provides a graphical interface for searching for available models in Hugging Face, downloading different quantizations, and interacting with them locally. It can also start a server with endpoints compatible with the OpenAI API.

The application allows you to select how much context to use, how many layers to send to the GPU, and other engine parameters. If you experience excessively high response times, you can apply different adjustments to Improve performance when LM Studio is running slowly.

Ollama

Ollama It is designed to download, organize, and run models using simple commands. It uses components from the llama.cpp ecosystem and allows you to work with both its library and GGUF files from other sources.

Modelfiles allow you to configure the base model, system message, template, and parameters such as temperature or context. If you want to use your own file, you can follow our guide for Load a GGUF model using a Modelfile.

llama.cpp

llama.cpp It is the reference engine for working with GGUF. It offers terminal tools, an HTTP server compatible with different OpenAI API endpoints, and numerous options for optimizing CPU, GPU, and memory usage.

It can use acceleration technologies such as CUDA, Metal, Vulkan, SYCL, and HIP, depending on the system and the installed build. It also allows running the model by distributing its layers across different devices.

Jan and GPT4All

Applications such as Jan and GPT4All They also allow you to download or import compatible models and use them through a graphical interface. These are interesting alternatives for those looking for a simple experience without sacrificing local execution.

Metadata, tokenizers, and split models

A GGUF file doesn't just contain quantized weights. It can also store information about the architecture, number of layers, context size, vocabulary, special tokens, and conversation template.

This metadata allows the engine to identify how to interpret the model. However, an incorrect template or an outdated version of the program can still cause unexpected responses, repetitions, or errors when loading the file.

Although GGUF is designed to facilitate distribution in a single file, large models can also be published in multiple parts. In that case, names indicating the fragment number appear, for example:

modelo-Q4_K_M-00001-of-00003.gguf
modelo-Q4_K_M-00002-of-00003.gguf
modelo-Q4_K_M-00003-of-00003.gguf

You need to download all the fragments, keep their names, and save them in the same folder. Otherwise, the engine won't be able to rebuild the model correctly.

GGUF, LoRA and adjusted models

Models tuned using techniques like LoRA can be converted or distributed for use within the GGUF ecosystem. llama.cpp allows the application of specific LoRA adapters during inference, and tools also exist to merge them with a base model before quantizing it.

However, it is important to differentiate between several concepts. LoRA and QLoRA are methods related to model training or tuning., whereas GGUF is a format primarily oriented towards storage and execution.

QLoRA allows you to fine-tune models using a quantized representation to reduce memory consumption during training. Once that process is complete, the result can be merged, converted, and quantized to create a compatible GGUF file, provided the architecture is supported.

Similarly, the connection to frameworks like LangChain doesn't come directly from GGUF. Integration is done through engines and servers like llama.cpp, Ollama, or LM Studio, which expose the model via an API or a compatible library.

How to choose the right GGUF file

In repositories like Hugging Face, it's common to find many versions of the same model. To choose correctly, first check that the architecture is compatible with the version of the application you're going to use.

Next, check these items:

• Number of parameters: A 14B model needs considerably more memory than a 7B or 8B model.
• Quantization: Q4_K_M is usually a good starting point for most teams.
• File size: It must fit in the available RAM or combination of RAM and VRAM.
• Model type: Check if it is a base, instruction, chat, vision, or specialized version.
• Admitted context: Do not automatically configure the maximum if you have limited memory.
• Conversation template: Use the recommended one to avoid poorly formed answers.
• Excerpts: Check if the model is split into multiple files.

If you're unsure between versions, start with Q4_K_M. If the model has memory headroom, you can try Q5_K_M or Q6_K to maintain a bit more quality. If it still doesn't load, use lower quantization, reduce the context, or offload fewer layers to the GPU.

GGUF has helped make local artificial intelligence much more accessible. Its combination of metadata, quantization, efficient loading, and compatibility with different types of hardware It allows the execution of models that a few years ago would have required considerable infrastructure.

The format does not eliminate computer limitations nor does it guarantee that any model will function correctly. However, choosing an appropriate architecture and quantization allows you to experiment with wizards, programming tools, and specialized models without constantly relying on external services.