- Ability to integrate custom Hugging Face models that are not in the official Ollama registry.
- Flexibility to choose different quantization levels depending on the available VRAM memory.
- Advanced support for loading fragmented GGUF models across multiple files.
- Ability to run private models by configuring SSH keys.
If you enjoy tinkering with artificial intelligence, you've probably noticed that Ollama's official catalog sometimes falls short, and you need a very specific model that's only available in GGUF format. Fortunately, Ollama allows you to import these files manually, opening the door for you to use any quantization you find out there, especially those that the Hugging Face community uploads.
For beginners, the process can seem a bit intimidating because the documentation sometimes doesn't explain the steps clearly enough. However, once you understand that it all boils down to creating a configuration file called ModelfileThe process becomes a piece of cake and you can have your custom AI up and running in minutes.
Download and prepare the model
The first thing you need to do is go to Hugging Face and search for the model you're interested in. You'll see there are a lot of quantized versions, like the Q4_K_M, Q5_K_M or Q8_0Don't worry too much about the names; basically, the lower the number, the less RAM or VRAM the model will use, although it will lose a little accuracy. Choose the one that best suits your hardware and download the .gguf file.
Once you have the file on your computer, I recommend place it in a fixed folder And make sure the filename doesn't contain spaces or special characters. If the name is a tongue twister, change it to something simple to avoid errors when typing paths in the terminal.
If you're a Windows user, you can leave the file in your Downloads folder, but make sure to know the exact route (for example, C:\Users\YourName\Downloads\model.gguf), since it will be the key piece for Ollama to find the model.
Creating the Modelfile and registering it in Ollama
This is where the magic happens. You need to create a plain text file. You can open Notepad and type a single line: from «C:\path\to\file\model.gguf»It is vital that the path is correct and enclosed in quotation marks if there are spaces. Once written, save the file with the name Modelfile (without the .txt at the end), although if you have trouble removing the extension, save it as Modelfile.txt and then manually rename it in the folder.
Now it's time to take action using the console. Open CMD or your favorite terminal and run the command ollama create your-model-name -f ModelfileAt this point, Ollama will read the instructions, process the GGUF file, and create a new layer on your local system. All that remains is to wait for the process to finish and for you to see the success message.
If you're curious about how the official models are configured, you can use the command olivama show –modelfile llama3.2This will allow you to see how the system prompt and templates are organized, which is super useful if you want to customize your AI's behavior beyond simply importing the file.
Advanced methods and automation
For those looking to work faster, it's possible to run models directly from Hugging Face without creating a manual Modelfile. If the model is public, you can use the syntax rocama run hf.co/user/repositoryBy default, Ollama will try to use Q4_K_M quantization, but if you want a different one, you can select it from the GGUF viewer on the Hugging Face website and copy the generated code snippet.
In the event that you work with private modelsThe process requires an extra security step. You'll need to copy your Ollama SSH key (usually located in ~/.ollama/id_ed25519.pub) and add it to your Hugging Face account settings. Once you've done this, you'll be able to access your private repositories directly from the terminal without any issues.
Another common problem occurs when the model is so large that it is divided into several GGUF filesThis used to be a headache, but now the `create` command supports importing multiple files. You just need to list all the `.gguf` files in the `Modelfile` or pass them in the `create` command for Ollama to join them correctly.
Performance and customization settings
It's not enough for the model to load; we want it to respond correctly. If you notice the AI behaving strangely, you probably need to adjust the chat templateOllama attempts to detect it automatically from the GGUF file metadata, but if it fails, you can create a file called "template" in your repository or define it in the Modelfile using Go's template format.
You can also optimize the experience by creating a file called "params" in JSON format to adjust the sampling parameterssuch as temperature or top_p. This is fundamental if you want the AI to be more creative or, on the contrary, much more deterministic and rigorous in its responses.
Integrating external GGUF files allows you to bypass the limitations of the official library, letting you choose the exact quantization for your VRAM and customize the system prompt. Whether by manually creating a Modelfile, running directly from hf.co, or managing fragmented and private models, Ollama has become a flexible tool for deploying any LLM locally with minimal effort.
I am a technology enthusiast who has turned his "geek" interests into a profession. I have spent more than 10 years of my life using cutting-edge technology and tinkering with all kinds of programs out of pure curiosity. Now I have specialized in computer technology and video games. This is because for more than 5 years I have been writing for various websites on technology and video games, creating articles that seek to give you the information you need in a language that is understandable to everyone.
If you have any questions, my knowledge ranges from everything related to the Windows operating system as well as Android for mobile phones. And my commitment is to you, I am always willing to spend a few minutes and help you resolve any questions you may have in this internet world.