Not every homelab post ends with a working setup. This one’s about a model that, as far as I can tell, cannot currently run on my hardware through any tool I have access to — not because of a config mistake, but because of a real gap between two separate projects that neither one has closed yet.
I’m writing it up anyway, because “here’s the thing that didn’t work and exactly why” is worth just as much as another bake-off post, and because I burned real hours chasing this before the actual root cause became clear.
The Goal
I wanted to test Llama 3.2 Vision (11B) at Q4 and Q8 quantization, same head-to-head format as my Q5/Q6 Gemma 4 comparison — real prompts, real numbers, see where accuracy actually holds up.
Should have been simple. It wasn’t.
Attempt 1: Ollama’s Official Library
Ollama lists llama3.2-vision:11b and llama3.2-vision:11b-instruct-q8_0 right in their library. Pulled both, no issues on the download side. Tried to run either one:
Error: 500 Internal Server Error: llama-server process has terminated: exit status 1: error loading model: unknown model architecture: 'mllama'
Same error, every prompt, both quant levels. Not a fluke.
Turns out: Ollama’s new engine (v0.30.0+) doesn’t support the mllama architecture yet. Straight from their own release notes: “llama3.2-vision is not yet supported. If your stack uses Llama 3.2 vision specifically, this is a hard block.” I was on v0.30.10. Confirmed via an open GitHub issue from someone who hit the identical error after updating.
Not a me problem. A right-now problem with the current Ollama build.
Attempt 2: Bypass Ollama, Go Straight to llama.cpp
Fine — Ollama’s engine is really just a fork of llama.cpp under the hood, so the obvious move is to skip the middleman and run llama.cpp directly. Built it from source with full ROCm/HIP support for my RX 9060 XT (gfx1200 target), clean compile, zero errors, confirmed working binary.
Pulled the community GGUF (Q4_K_M + separate mmproj vision projector, since Llama Vision splits those into two files), pointed llama-server straight at both.
E llama_model_load: error loading model: unknown model architecture: 'mllama'
Same exact error. Different tool, same wall.
The Actual Root Cause
This is the part that made the whole thing click: llama.cpp’s mainline project has never supported the mllama architecture at all. Ollama’s vision support for this specific model was always running on a private internal fork with patches that were never upstreamed. There’s a long-running, unresolved community discussion about whether llama.cpp will ever add native support for it.
So here’s where that leaves things:
- Ollama’s old engine supported it, via their private fork.
- Ollama’s new engine, as of v0.30.0, dropped that support.
- llama.cpp itself, upstream, never had it in the first place.
Two tools, two different reasons, same dead end. There’s currently no active path to running this specific model locally through either one.
What Actually Would Fix It
- Ollama re-patches mllama support back into their new engine (their call, no ETA)
- llama.cpp’s maintainers merge native mllama support (open community debate, no ETA)
- Downgrade Ollama to a pre-0.30.0 release — technically works, but risks breaking whatever newer models/features you’re relying on today. Didn’t consider this worth the tradeoff for one model.
None of those are something I can fix from my end tonight. Filing this one under “wait it out.”
What Didn’t Go to Waste
The two runs I could actually complete tonight (Gemma 4 Q5 vs Q6) turned up a real, useful finding on their own: Q6 partially caught the sudo nuance on my “how do I check GPU/VRAM usage” tie-breaker question where Q5 didn’t — but both flatly fabricated a wrong definition for my own company’s internal IPO terminology, confidently, with zero hedging. Quantization level didn’t touch that gap at all. That’s a fine-tuning/RAG problem, not a “throw more precision at it” problem — worth remembering next time a model sounds confident about something it has no way to actually know.
Key Takeaways
- If a model architecture-specific error shows up (“unknown model architecture: X”), check whether your version of the tool actually supports that architecture before assuming it’s a local config problem — release notes and GitHub issues are the fastest way to find out.
- Ollama and llama.cpp are related but not interchangeable; a model working in one doesn’t guarantee it works in the other, especially for anything with a private/forked implementation.
- Not every model on paper is runnable in practice, even with correct files, correct quant levels, and a clean build. Sometimes the honest answer is “not right now.”
- A failed test still produces real data if you were rigorous about how you ran it. Tonight’s dead end doesn’t erase the real Q5/Q6 finding that came out of the same session.
JR Lorenzo · built with Hugo, served from the homelab