Getting deeper into local inference

26 September 2026

I’ve reached the point with my personal LLM usage—Q&A, coding, sysadmin, web research—that I no longer need a cloud provider like ChatGPT or Claude. A local model is working just fine.

Two recent developments have made this possible. One is the release of Qwen3.8-27B. This is the first model I’ve used offline that meets all the criteria: intelligence, token rate, and tolerable VRAM requirements.

Even though it’s a “small” model it’s well-developed for xhigh reasoning. What this means is that on large or complex coding tasks it’s not going to solve it right away. It will descend into long streams of thought where it spends tens of thousands of tokens pondering every aspect of the problem. It does this some number of times, and then when enough computation has occurred, it abruptly stops and emits hundreds or thousands of lines of code—code that typically works. This is great example of graceful degradation. If you’re not smart enough to answer instantly, just take more time to arrive at it. I don’t have “big jobs” too often in my personal coding and there are few time constraints, so I’m quite happy to send it off for a few hours to figure things out.

On “low” reasoning, on the other hand, it stays nice and snappy. This is perfect for Q&A and looking for information on the web. One model suits both modes of operation, which is super convenient.

The other thing that made this possible (no surprises here) was acquiring some new hardware, upgrading from my old desktop to a high-end PC laptop. I say “laptop” but it’s more like they took a honking great GPU and built a portable chassis around it. What I have is an RTX 5090 laptop edition (considerably less powerful than the desktop version) with 24 GB of VRAM. Even with Windows using a little, this is enough memory for a really solid llama.cpp deployment: Q4 model, 131072 context with KV at Q8, MTP for ~40% speed boost, and image input support. This yields around 60–30 tok/s output and 1500–1000 tok/s prefill depending on how deep I am in the context.

Please understand that it remains an economically foolish decision to buy hardware just for inference. If you don’t want to spend Claude/Codex kinds of money look at OpenCode Go. Local tokens are basically always going be slower and more expensive, and your usage duty cycle is going to be lower than providers hosting lots of customers. Obviously some people are going to do it anyway because they really like privacy or really dislike Sam Altman or whatever, but maybe try to buy hardware that can serve multiple purposes. Now I can play Forza Horizon 6 at a solid 4K/60fps. Bet you can’t do that with your DGX Spark?

Over the time I’ve been playing with local models my priorities have changed. My original assumption was that more parameters is better, that I would be happiest with a mid-tier model like DeepSeek V4 Flash running on unified memory on Apple Silicon, accepting a lower token rate of maybe 10–20 tok/s. Patience is a virtue, right?

It turns out my revealed preference is faster token rates and less smart. Coming from Claude/Codex what I want is minimum 30 tok/s, ideally 40+ to feel comfortable. This is way better both for interactive chats and for watching it work through some coding problem to make sure it’s on the right track. Any slower than this, it becomes hard to shut out the thought that I’m wasting my precious time on this planet watching matrix multiplication take place. As a result, I stopped using my Mac for inference and gave all that RAM to Docker. NVIDIA hardware remains king if you want speed. It absolutely tears through prefill, it’s a delight to watch.

The trade-off wasn’t as severe as I thought. When I was using ChatGPT for non-coding tasks, most of the time the information I needed didn’t actually come from the model’s training. Usually I want up-to-date info with references I can verify. I’d ask a question and the first thing it would do is run a web search or two, then summarise what it found to match my particular query. It turns out you don’t need a superintelligence to do this. On the other hand, a speedy 4B model would struggle to process the information and keep its story straight. What we need is something more reliable but still pretty fast… maybe…27B? Yes, I think a 27B model would do great here.

You’ll notice that I haven’t said anything about work usage of LLMs. I’ll be blunt: if you work in an LLM-positive workplace, local inference will not be satisfactory unless you spend tens of thousands of dollars on equipment. In a recent test I put the same code review query into both Sol 5.6 (high) and Qwen3.8-27B (xhigh), a complicated diff of a couple of thousand lines spanning two related PRs. Both models identified one minor issue, and each model discovered a different major issue (action at a distance, following code outside the main diff). The difference is that Sol did it in about 10 minutes while Qwen took over an hour, and that fully monopolised my single GPU. If tokens on tap are what you need, wait until things get much faster.

Now you understand what I’m doing and why, let’s have a look at my setup.

The main pieces are:

• llama.cpp running unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M on my new Windows PC.
• SearXNG (actually a home-grown golang port of the core functionality to make it easier to run on Windows). Having access to good search results is critical both for coding and chat. Today I’m using free-access providers and it’s working okay but I will definitely consider using the Kagi or Brave Search API.
• Oh-my-pi (OMP), a blinged-out version of the coding harness Pi. I have this inside a full Linux VM (not WSL) for isolation—10% because I’m worried about an LLM stuff-up, 90% because I’m worried about malware in library dependencies and I don’t want that on the OS with all my personal data.
• Open WebUI, a webapp that looks and works similarly to the ChatGPT web interface. It has logins, chats, a persistent database for your past conversations, automations, and a bunch of other functionality that I haven’t touched.
• Tailscale links together the servers and also enables my smartphone to privately access Open WebUI. All of the services are configured to listen directly on their Tailscale IPs so this stays entirely private to me.

You might wonder why Open WebUI is sitting off on some dedicated server. The answer is simply that I had some spare capacity on that server and I couldn’t be bothered working out how to run it on my PC. Other benefits include reducing RAM use on the PC (32 GB is already tight) and still being able to browse the webapp on my phone if the PC is asleep.

You might wonder why the search proxy is on the PC rather than the dedicated server. The answer is that as part of their measures to handle bots, some search engines treat requests coming from datacentre IPs disfavourably. This way the queries come from my residential IP. Arguably I'm obliged to move to a paid provider here. Selfishly, I’ll admit it’s nice for privacy to have my searches blend in with the rest of my ISP’s CGNAT traffic rather than having every single one of them tied to my particular API key.

I’ve been running this setup for about a week and it’s working great. The local Qwen helped me with some parts of the installation like deploying a new proxmox VM running Open WebUI, and building/testing that “mysearx.exe” port.

If that wasn’t enough, I believe there are good reasons to expect this hardware setup to get significantly better in the future, just through model and software improvements.

It is anticipated that Qwen4 will be released in the coming months. In Qwen3.8-Flash-Next they previewed a technology where they use a separate N-gram embedding layer. In this scheme, all of the intelligence doesn’t have to be resident in VRAM. You can keep this lookup table separately in normal system RAM or even on a fast SSD and boost the effective parameter count with essentially no slowdown in token generation. The big question is whether we will get a new open weights 27B release taking advantage of these newest developments.

Further ahead, research is continuing apace into “latent reasoning” methods. To handwave considerably, think about the verbose English gibberish produced by Qwen3.8 when it’s reasoning its way to a solution. If this output isn’t intended for me to read, why is it in English? It’s a computer program—intuitively, surely it could more efficiently compute its way to the same result using some compact binary representation of the same intelligence? There are many practical problems to solve and I’m not aware of any major deployments of this technique. Give it a couple of years—maybe we’ll see a leap forward in what we can get out of the same hardware, using this or any other novel inference efficiency technique.

While putting this new system together I learnt several things and ran into a bunch of minor issues along the way. In no particular order:

• Try not to look at Qwen’s reasoning traces too much. For a human reader they are infuriating. It will keep telling itself that it’s going to write out the source code, and I quote, “this time for real”, and then it will suddenly realise there’s some detail it hasn’t considered yet. This repeats many, many times.
• Qwen3.8-27B has a quirk with its chat template where while it advertises reasoning levels “low”, “medium” and “xhigh”, the only two effective modes in practice are low and xhigh. If you set it to medium it will act like it’s on xhigh.
• At first I aimed for 256k context with Q4 K/V. It started to lose its way and get into loops as the context grew. Q8 has been much more dependable and 128k is plenty most of the time.
• Unlike a standard OpenAI endpoint, Qwen3.8 needs all the reasoning to be replayed back to it in the requests. This is known and generally supported by consumer software but you often have to turn it on explicitly. Without that reasoning you get poorer responses and poorer caching
• In Open WebUI, correctly replay the reasoning by going to Connections > Edit your OpenAI API connection > Advanced > Provider and ensuring the dropdown says llama.cpp.
• OMP is trickier—it has some heuristics around whether a model is “local” and if you use a Tailscale IP as an endpoint this is detected as “non-local”. Even if it knows it’s talking to Qwen3.8, it chooses not to send the reasoning traces in its requests. Force it with replayReasoningContent: true. Confusingly, the UI mostly shows the context use as reported by llama.cpp, while locally it’s tracking context use with reasoning included. This causes weird bugs where it appears to compact (run auto-handoff) while it’s still 10% or more before the configured compaction point.
• Open WebUI needs a special tweak to put Qwen into low reasoning mode. Go to Models > Edit your Qwen3.8 model > Show Advanced Params. Then add a new one with name chat_template_kwargs and content {"enable_thinking": true, "reasoning_effort": "low"}.
• OMP has a few features that send additional prompts to your model beyond what you type, which can be disruptive when you have only one slot and it potentially ejects your entire working session from KV cache. The worst offender is the automatic session title generator. There’s a command line option to turn it off, or I use the envvar PI_NO_TITLE=1. Less cache-damaging is the “idle recap”, which you can turn off under settings. Also turn off Async Compaction so it doesn't compete with your regular work as your context gets close to full.
• If you use --sleep-idle-seconds on llama.cpp then it will automatically free up your VRAM for other purposes when there haven’t been any LLM requests for a while.

For reference, here are my OMP config.yml and models.yml, and the full command line I’m currently using for my llama.cpp instance: Gist

I assume that one day there will be turnkey solutions that provide most of this as a convenient bundle but today you’re kind of left to wire it all up yourself. It’s actually quite fun—unlike most things in computers it doesn’t just work or not work. It’s a proper heavy workload where you can tweak knobs to make things faster or slower or trade off different properties against each other. Maybe it works but it’s suboptimal and you won’t notice unless you’re paying attention. You need to follow recent developments or you might not know that you could use something like speculative decoding.

I’ve never been a car person and never quite understood how “working on the car” constituted a hobby but with local LLMs I feel like I’ve discovered the digital equivalent. Good times.

Serious Computer Business Blog by Thomas Karpiniec
Posts RSS, Atom