It’s affordable. They cost 69 USD each. Nice.

Versatile deployment modes. Out of the box you can build a standard AP/client scenario. You can also use 802.11s meshing, which I haven’t tested yet. You have the choice between NAT and bridging, so you can make it so that all the connected devices think that they’re on the same ethernet segment. You can configure it both for providing internet access and for using it as a secondary network that sits alongside your other WiFi and cellular.

Versatile local-device connectivity, especially ethernet over USB. Interestingly, these dongles do not even attempt to make themselves known to the OS as a WiFi HaLow adapter. The dongle itself acts as the host on the HaLow network and then your device is bridged over regular WiFi/Ethernet. You have three choices: connect to the dongle’s local 2.4 GHz WiFi AP; connect to the dongle’s RJ45 ethernet port; or use ethernet-over-USB via the dongle’s USB-C port.

Let’s focus on ethernet-over-USB for a moment because this is super interesting. I’ve tested this on a MacBook, an iPhone 17, and an Android Pixel 7. The Apple devices both did two things: they powered up the radio, and they added it as a wired ethernet device with zero configuration on the phone. This shows up as “Ethernet” in the screenshot above. This is significant so I’m going to spell it out in detail. If you would like your iPhone to communicate over a kilometre or two, you can plug in a device to its regular old USB-C port, and it will work. It does not need any batteries or charging of its own because it can get power from the iPhone. It does not require any pairing or user interaction. It shows up as an additional IP interface, which means that it does not displace any existing connectivity you have via WiFi, Bluetooth or cellular. Because it is a normal IP interface, it Just Works in every app that does LAN networking.

For anyone with an iPhone this is kind of amazing. Unfortunately, it did not work on the Android. For whatever reason my Pixel doesn’t come with an appropriate driver to pick up the Ethernet connectivity. It did, however, appear to power up the radio.

I mentioned before that these devices kind of default to being your primary network, where there exists some upstream gateway that is going to grant you both an IP address and internet access. This is not required. For my tests so far I haven’t gone full mesh and I’ve stuck with the simple “AP” and “STA” roles, where one radio is an AP, and the other connects as a client. With some config tweaks this coexists perfectly well with existing WiFi. I’m more interested in having HaLow as a secondary layer where you’re going to be using HaLow for a strictly P2P role with greater range.

Out of the box, one of the Heltec boxes acting as an AP will advertise itself as the default route. This is a problem if it is not actually connected to an upstream network with internet access, and you would prefer that clients used their existing WiFi or cellular connectivity to access the internet. Ultimately you can do whatever you need—under the hood it’s a MediaTek MT7628 paired with a Morse Micro MM6108 HaLow radio, running OpenWrt which is eminently hackable. You even have the option of flashing your own firmware if you want. The good news is that Heltec’s supplied user interface is sufficient to configure the DHCP server so that it will allocate IPs but not try to route unrelated traffic.

Navigate to Network > Interfaces and choose Edit on ahwlan.

Then go to DHCP Server > Advanced Settings and add a couple of extra DHCP options. The idea here is that a bare 3 will prevent it from advertising a default route, and a bare 6 will stop it advertising itself as a DNS server. This way you are free to talk to other devices across the HaLow LAN in addition to the networks you already had.

The other important thing to know is that these devices do not support wired ethernet and ethernet-over-USB simultaneously. You can toggle between these modes by briefly pressing the little button using the supplied metal widget. Everything is explained by the colour of the LED. For an AP, cyan means wired ethernet and purple means USB. For a station, green means wired ethernet and blue means USB.

The cool thing is that once you get this config locked in, it’s locked in. You can plug it in to a Mac or an iPhone and DHCP does the rest. You need only wait a minute or two for it to boot up.

I intend to put these radios through their paces—expect more posts about them!

This won’t be a surprise to anyone who’s been using LLMs in 2026. There are no special tricks and you don’t have to overthink it[1]. We really are at the point where you say to any modern model “find the bugs” and it finds the bugs. It’ll even fix them too. Isn’t that magical? I may be slinging slop but darned if my library isn’t substantially better for it.

The only findings I had to reject were cases where I’m actually being sloppy on purpose. For example, some platforms have edge cases where a network’s hardware address might not be readable. I choose to leave it as a String with possible placeholder, since it kind of sucks to make it an Option for the normal case. Similarly, in principle it’s possible for an interface watcher to start failing in the middle. I choose to swallow the errors. Virtually everything was validated when the watch was first created, and transient errors will be handled gracefully by incorporating changes into the next update. Because I do this, you don’t need to check for an error every time and the code is lovely and clean. LLMs don’t enjoy this kind of corner-cutting (nor do certain humans) but I put my foot down. If there’s evidence that this causes problems in the real world then I’ll reconsider.

The majority of the bugs were the sort of things where I think “oh yeah I could see that being a problem” but it wasn’t particularly likely to come up for most users most of the time: race conditions, handling panics over FFI, unusual interface types, things like that. Pre-LLM, these are the sorts of things that would come up as intermittent faults on the bug tracker and I’d have to spend hours poring over the code and vendor docs to figure out what went wrong. I feel like I’ve done a kind of speed-run on that hardening and now it’s ready for more users[2].

That said, the most significant bugfix in this release was not detected by LLM inspection, and I’m a little surprised that it wasn’t. It fell out by accident when I added IPv6-specific integration tests and I realised that if I changed an IPv6 address on Linux it wasn’t picked up by the library at all.

The history of this bug is mildly amusing. On Linux the standard way to subscribe to IP address changes is to open a NETLINK_ROUTE socket and use group bits to indicate what event types you care about. I needed to specify the constants RTMGRP_IPV4_IFADDR and RTMGRP_IPV6_IFADDR, which I took from the nix crate. At this point, everything was correct.

Later I was trying to make this work on Android (which is a kind of Linux) and the build failed because those constants were not present in the Android version of the nix crate. To make the compiler happy I switched those constants to local definitions:

const RTMGRP_IPV4_IFADDR: u32 = 0x10;
const RTMGRP_IPV6_IFADDR: u32 = 0x20;

Unfortunately for me, this was wrong. If you look at the relevant kernel header, you can see that I copy-pasted the correct value for IPv4, then for IPv6 I used the value of the line directly underneath it, which is actually RTMGRP_IPV4_MROUTE. IPv6 really needed the value 0x100.

For whatever reason, my LLMs didn’t see this. Naively I’d have expected RTMGRP_IPV6_IFADDR and 0x20 in the same line to be a glaring, self-contained error, if you’ve been trained on lots of code where this would have been set to 0x100. Yet it didn’t, so we found the bug the boring, traditional way. (Tests.)

The best part of this bug is that as soon as I got Android compiling, I learnt that modern Android restricts netlink sockets and I was going to have to build an entirely different watcher mechanism involving their Java API. Thus I subtly broke the code for no gain whatsoever.

So there we have it. Enjoy the new release. If you’re maintaining a public library, at this stage you really have nothing to lose by asking an LLM to find the bugs. (Except for maybe getting put on strange lists.)

1. If you really want to know I used a mixture of DeepSeek v4 Flash and Pro and GPT-5.6 Luna and Sol, but it really doesn’t matter on a small codebase like this one. ↩︎
2. If crates.io downloads mean anything, usage seems to be increasing, which is gratifying. ↩︎

Ostensibly /dev/world is a conference for developers on Apple platforms, but it’s always been a little broader than that, and this year was no exception. The keynotes were about career and life. We had adjacent technical topics like Kubernetes and security, and much more AI than previously.

Nic Wittison speaking about agentic AI development

One of my favourite parts was a series of presentations by students at RMIT. Some of them have done a short intensive course on app design and development—only a few weeks!—while others are studying it as a four-unit minor. These young people did a fantastic job. They found a human-focused problem where an app could help, did their research, and tried their best to develop an app that really meets the target user where they are. It also happened to be student groups who showed the most sophisticated engagement with AI/ML at the conference—creating custom forecasts and dramatically improving the performance of a small Qwen model for local inference via DoRA fine-tuning.

Toby Duffy is the creator of PreviewAR, an app used by many TV and movie producers when scouting outdoor locations where they intend to film. They can take photos and videos with custom 3D models inserted into the scene using Apple’s augmented reality framework, ARKit. He gave a fascinating talk about the problems that arise when creating large-scale AR scenes outdoors, noting that most of the existing samples and work assume you’re in a room. He said that one support request involved a user having trouble placing a model for a full-sized stadium in their scene—to my disappointment, I later confirmed that this wasn’t someone assessing the viability of the Hobart stadium.

What I’d really like is a Woolworths AR application that helps me find the aisle with the weird ingredient on my shopping list. A man can dream.

Josh Deprez spoke about Kubernetes, reframing it as a common language for describing the infrastructure and services you need when running highly available distributed software. The shared understanding this creates is almost more useful than the specific software. They used examples from their home lab of Raspberry Pis running K3s, hosting a web service over multiple nodes. Over lunch I inquired about high availability—apparently they are now using a system called MetalLB in which the Pis advertise BGP routes to the service IP, so if one node goes offline then new requests follow one of the remaining routes. BGP on a home LAN. Wild.

Josh Deprez speaking about Kubernetes

William Taylor gave a wonderful talk about using Swift for sharing business logic across iOS, Windows/C# and WASM at Drawboard. Like most other companies in this situation, they reasoned for a long time that C++ was probably the best fit. Unlike most other companies, they were sufficiently committed to their dislike of C++ to hold out until one day Swift became viable, with the official support for WASM landing in 6.2 last year. This is familiar territory to me—I’ve been using Swift on Linux and embedded Swift in WASM recently at work. It’s cool to see that it’s solving problems for other people too. In the past I’ve used Rust extensively for cross-platform logic, but unless you really need Rust’s performance or libraries, I can see Swift being a more developer-friendly language, especially if your team doesn’t already know Rust.

Now, let’s talk about AI. There were four or five talks in the main schedule that dealt with this topic substantively, with a range of opinions. I saw Claude/Codex running on a few people’s laptops (including my own) across the two days. It was a common topic in conversations. As far as I noticed, everyone managed to stay cool and normal about things, which is good. While I imagine a few delegates would prefer if AI just disappeared, I’m grateful that /dev/world has evolved into a conference where folks can be enthusiastic (usually with caveats), share worries out loud, experiment and develop with the technology on various axes, and examine the implications for cybersecurity. It’s a contentious topic, and having humans in the same physical space inspires a certain amount of good-faith decency that rarely extends to microblogging. There aren’t many venues for Australian software developers to have that in-person experience, so I’m glad this was one.

On a more critical note, I recognised the (undeclared) work of Claude in the slides of at least three presentations. I’m not out to embarrass anyone in particular, but suffice to say the end result would have been better without it. I’d much rather see extremely simple slides in the presenter’s own voice than a clichéd visual design, overcomplicated infographics, or Claude’s verbal tics.

For those who did want to hear about AI, we had plenty of choice. Ace Authors gave a great technical overview of the requirements and parameters of local inference, taking this all the way up to running MLX models over a cluster of Macs. It turns out this requires a lot of network bandwidth. Nic Wittison described his newfound productivity, noting that you only get slop if you allow slop and explaining how his experience as an engineering manager and structured tools like Open GSD help him get good outputs. Harry Ng explored how you teach and assess programming in a world where AI is able to write the code. Louis Cremen highlighted how AI can enable attacks both by finding existing vulnerabilities and by becoming a vulnerability itself when it can be tricked into performing actions against its owner’s interests. Cryptographic context injection is wild. Daniel Steinberg touched on AI in his keynote, in which he reminded us all to breathe and make time for the humans in our lives. Always good advice.

It was Nic who had the quotes of the day.

“If you feel it just makes slop, use your expertise and stop letting it make slop.”

“If fixing mistakes is cheap, it might be worth reframing what your tolerance for mistakes is.”

Great event, everyone. See you next time, I hope!

Obligatory Tim in Judit’s keynote

Over the years PRs have become a ceremony serving multiple functions: catching bugs by getting two or more sets of eyes on it, helping new hires understand the wider codebase and house style, being aware of what your teammates are working on, among other things. Many of these outputs are… just not as important any more. Thanks to LLMs, there are faster and more efficient ways to catch simple bugs, understand the codebase, and track your teammates.

I don’t think we should get rid of PRs. They still have a purpose but it has changed, and therefore the way we conduct PRs should also change.

In this post I want to share the main changes in how I operate on LLM-heavy repos. Obviously everybody’s still figuring things out. I expect in a couple of years much will have changed again.

Reading the code

Until recently my basic position has been that everyone needs to read the code. There are AI enthusiasts who say that’s unnecessary. I have a lot of respect for antirez, who has made this case, so I take the possibility seriously. Despite these voices and despite increasingly powerful models, I don’t think we’re quite there yet. The code is the ground truth and as of 2026 I believe we still need some connection to the code to be responsible and effective product developers.

I know full well that if you ask Claude Code to build a JSON API endpoint that runs a SQL query and outputs the results as JSON, it’s just going to do it right. It’s not going to mess that up. You have it add automated tests, you have it add documentation, you know it’s going to be good.

But I’m not reviewing that code. And now I’ve got that feeling of guilt: if I haven’t reviewed the code, is it really responsible for me to use this in production?

That was Simon Willison on 5 May. Nowadays I’d go further—for a given well-specced coding task, 5.6 Sol will write it about as neatly as I would, with much greater awareness of edge cases. It won’t get tired and skimp on either manual validation or unit tests. It will do it in a few minutes rather than take me all day. For major segments of my work output I literally trust its code more than my own. It’s a bit rough to admit, as someone who’s been paid to write code for around 18 years, but here we are.

I like to think of it like Waymo—there is a point at which the computer will statistically produce fewer defects than a human doing the same task, and I feel we’re reaching that. I haven’t even tried Astra yet.

Today, for a team with LLMs, I think the pragmatic approach is somewhere in the middle: read some of the code[1]. For a given PR to merge I would like to see:

1. The author has asked their own LLM to do a comprehensive self-review before taking it out of draft[2].
2. The author has read and understood the key parts of the code. Does the diff fix the bug? What functions implement the new feature and where are they called?
3. The author has skimmed the auxiliary code and tests. Has it edited some unrelated module? Have 300 lines of code sprung up to cover some case that isn’t all that important?
4. The reviewer understands the problem, the situation before and after, and has had an active conversation with their LLM about the suitability of this branch.

In particular, I don’t necessarily think reading the code is a good use of the reviewer’s time and energy. Certainly they should ask their own LLM to do an adversarial review and find code-level flaws, but that’s the less important part.

Often as a reviewer I will have conversations like this with my chatbot[3]:

<Human> Review this PR. Explain the problem from a product or user-facing perspective, give me a refresher on relevant context, explain what was changed and why that solves the problem. Highlight any bugs or code quality issues that I should bring up.

<GPT> … [This is the problem, this is how it was solved, the code looks pretty good] …

<Human> Okay so what if the user did X? Wouldn’t it kind of suck in that scenario

<GPT> [traces it through] Yes you’re right, this solution would handle that poorly because […]

<Human> What if we injected this data at this earlier point? Would that fix it?

<GPT> […] Yes

Now I have a high-value comment to put on the PR.

Obviously this is just a pseudo-conversation but you get the idea. The highest value I can bring as a reviewer is to catch architectural and product-level missteps. Today, this awareness generally sits outside the context of the LLMs that are writing the code so that’s the human’s job[4].

I would go further and say that the architectural/product review was always the most valuable part of the review. Previously, reading the code was usually a prerequisite to be able to comment well on the architecture. If you were lucky there was a good PR description but it’s rare that it’s complete and clear enough to cover all relevant discussions.

When an LLM ingests a PR diff and the surrounding context, that session has become a live and interactive PR description. You can ask for extra detail about the bits you care about. You can check what assumptions have been made. You can learn exactly what it is that you want to know about the PR so that you can bring your expertise to bear as to whether it’s a good idea to merge it. It’s almost unfair to expect the author of the PR to produce some static text upfront that serves the same need for all possible reviewers. Better to just put a clear summary on the PR and then the reviewers can have their LLM pull up whatever context they need.

This is a significant change to what we’ve considered “a good PR review” for a long time. The best news is that if you accept this framing, you need not feel burdened to read thousands of lines of LLM-written code as a reviewer. Why should you spend hours repeating what GPT can do in moments? It’s miserable and not sustainable. It’s also unnecessary for most kinds of software. Yay.

Discussion questions no longer useful

As a reviewer I would often ask a question where I didn’t necessarily expect a code change. Sometimes it was protecting my own time when something was inadequately clear, or relied on contextual knowledge which the author already has in their head: “Can you explain how this part works exactly?” or “Will this cover case X?” Most of the time it would just be a shared learning exercise. Maybe it would result in a simplification or an extra code comment.

In a team with LLMs, if I do this I’m simply wasting the author’s time. It’s almost certain that I can ask my own LLM and find the answer. If the answer is unsatisfactory and there’s something wrong with the code, then it’s no longer a question—it’s something to fix.

Faster to patch than to explain

A recent blog post by Niklas Gruhn instructed: “Don’t be a meat proxy”. This is good advice that I still stick to. If for some reason I’m showing someone raw output from Codex or Claude then I tend to be apologetic about it.

This respectful urge actually creates a problem in PR reviews.

1. Reviewer runs their LLM and it identifies a bug
2. Human interprets the finding, looks at the code, agrees it’s real
3. Human writes a summary in their own words, possibly starting from a draft description from their LLM, then posts it as a comment
4. PR author reads the comment, understands it to some degree
5. PR author pastes the comment into their LLM which then fixes the bug

Oh no, we’ve created a double meat proxy.

This is polite in terms of human-to-human interaction but clearly it would have been more efficient like this:

1. Reviewer runs their LLM and it identifies the bug
2. The human interprets the finding, agrees it’s real, and asks the LLM for a patch
3. They push the patch onto the branch

Most people would consider it bold, even rude for a reviewer to go in and just add more commits, but maybe not now. Apart from avoiding the double meat proxy, I suspect if an LLM is writing most of the code anyway then maybe there’s less preciousness about the exact code style and having “pure” ownership of the PR. Thanks to LLMs, painful merge conflicts are a thing of the past[5]. Maybe it’s a healthier dynamic generally if PRs are collaborative by default?

So it is now likely that when I review a PR, instead of leaving it hanging with 5 comments to address, I might just push a few commits along with an approval. The author gets to merge faster. Yay.

To be clear, I don’t think this is a universal shift. What I’ve described here may work in one project but it’s not what I would do by default if I went and opened a PR on some open source project. Unless that project is clearly aggressively vibed, I’d be very uncomfortable asking a stranger to deal with code that I haven’t read in detail. And yes, if a project asked that code be written entirely by hand then I would do exactly that. It’s fun, after all.

1. I’m talking here about the of code where there’s kind of a limited blast radius for errors. If you’re building medical devices or aeroplanes then you already know you shouldn’t be listening to bloggers like me. ↩︎
2. There are lots of anecdotes flying around about “a model won’t find its own mistakes”. That’s nonsense. You might find more bugs by trying a different model, but even an undirected “hey code review our working branch for issues” will frequently turn up valuable things that you want to fix before pushing. Obviously it would be nice if the model didn’t type out the bugs in the first place but that’s the reality of the tooling today. ↩︎
3. I have one “skill” in Codex, and it’s a longer version of this prompt. The real one includes some snark about Claude-generated PR descriptions. ↩︎
4. There are lots of companies working on lights-off factories where LLMs are responsible for all the specs and planning and long-term development. Again, I don’t think we’re there yet. Let’s optimise for the tools that are effective today. ↩︎
5. People talk about this surprisingly rarely. Do y’all realise how good we have it that now there is software that can intelligently untangle dozens of conflicting hunks across thousands of lines of code, taking into account the semantic intent of each commit?? ↩︎

Two recent developments have made this possible. One is the release of Qwen3.8-27B. This is the first model I’ve used offline that meets all the criteria: intelligence, token rate, and tolerable VRAM requirements.

Even though it’s a “small” model it’s well-developed for xhigh reasoning. What this means is that on large or complex coding tasks it’s not going to solve it right away. It will descend into long streams of thought where it spends tens of thousands of tokens pondering every aspect of the problem. It does this some number of times, and then when enough computation has occurred, it abruptly stops and emits hundreds or thousands of lines of code—code that typically works. This is great example of graceful degradation. If you’re not smart enough to answer instantly, just take more time to arrive at it. I don’t have “big jobs” too often in my personal coding and there are few time constraints, so I’m quite happy to send it off for a few hours to figure things out.

On “low” reasoning, on the other hand, it stays nice and snappy. This is perfect for Q&A and looking for information on the web. One model suits both modes of operation, which is super convenient.

The other thing that made this possible (no surprises here) was acquiring some new hardware, upgrading from my old desktop to a high-end PC laptop. I say “laptop” but it’s more like they took a honking great GPU and built a portable chassis around it. What I have is an RTX 5090 laptop edition (considerably less powerful than the desktop version) with 24 GB of VRAM. Even with Windows using a little, this is enough memory for a really solid llama.cpp deployment: Q4 model, 131072 context with KV at Q8, MTP for ~40% speed boost, and image input support. This yields around 60–30 tok/s output and 1500–1000 tok/s prefill depending on how deep I am in the context.

Please understand that it remains an economically foolish decision to buy hardware just for inference. If you don’t want to spend Claude/Codex kinds of money look at OpenCode Go. Local tokens are basically always going be slower and more expensive, and your usage duty cycle is going to be lower than providers hosting lots of customers. Obviously some people are going to do it anyway because they really like privacy or really dislike Sam Altman or whatever, but maybe try to buy hardware that can serve multiple purposes. Now I can play Forza Horizon 6 at a solid 4K/60fps. Bet you can’t do that with your DGX Spark?

Over the time I’ve been playing with local models my priorities have changed. My original assumption was that more parameters is better, that I would be happiest with a mid-tier model like DeepSeek V4 Flash running on unified memory on Apple Silicon, accepting a lower token rate of maybe 10–20 tok/s. Patience is a virtue, right?

It turns out my revealed preference is faster token rates and less smart. Coming from Claude/Codex what I want is minimum 30 tok/s, ideally 40+ to feel comfortable. This is way better both for interactive chats and for watching it work through some coding problem to make sure it’s on the right track. Any slower than this, it becomes hard to shut out the thought that I’m wasting my precious time on this planet watching matrix multiplication take place. As a result, I stopped using my Mac for inference and gave all that RAM to Docker. NVIDIA hardware remains king if you want speed (896 GB/s memory bandwidth vs 614 for my Mac). It absolutely tears through prefill, it’s a delight to watch.

The trade-off wasn’t as severe as I thought. When I was using ChatGPT for non-coding tasks, most of the time the information I needed didn’t actually come from the model’s training. Usually I want up-to-date info with references I can verify. I’d ask a question and the first thing it would do is run a web search or two, then summarise what it found to match my particular query. It turns out you don’t need a superintelligence to do this. On the other hand, a speedy 4B model would struggle to process the information and keep its story straight. What we need is something more reliable but still pretty fast… maybe…27B? Yes, I think a 27B model would do great here.

You’ll notice that I haven’t said anything about work usage of LLMs. I’ll be blunt: if you work in a workplace enthusiastic about using LLMs, local inference will not be satisfactory unless you spend tens of thousands of dollars on equipment. In a recent test I put the same code review query into both Sol 5.6 (high) and Qwen3.8-27B (xhigh), a complicated diff of a couple of thousand lines spanning two related PRs. Both models identified one minor issue, and each model discovered a different major issue (action at a distance, following code outside the main diff). The difference is that Sol did it in about 10 minutes while Qwen took over an hour, and that fully monopolised my single GPU. If tokens on tap are what you need, wait until things get much faster.

Now you understand what I’m doing and why, let’s have a look at my setup.

The main pieces are:

• llama.cpp running unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M on my new Windows PC.
• SearXNG (actually a home-grown golang port of the core functionality to make it easier to run on Windows). Having access to good search results is critical both for coding and chat. Today I’m using free-access providers and it’s working okay but I will definitely consider using the Kagi or Brave Search API.
• Oh-my-pi (OMP), a blinged-out version of the coding harness Pi. I have this inside a full Linux VM (not WSL) for isolation—10% because I’m worried about an LLM stuff-up, 90% because I’m worried about malware in library dependencies and I don’t want that on the OS with all my personal data.
• Open WebUI, a webapp that looks and works similarly to the ChatGPT web interface. It has logins, chats, a persistent database for your past conversations, automations, and a bunch of other functionality that I haven’t touched.
• Tailscale links together the servers and also enables my smartphone to privately access Open WebUI. All of the services are configured to listen directly on their Tailscale IPs so this stays entirely private to me.

You might wonder why Open WebUI is sitting off on some dedicated server. The answer is simply that I had some spare capacity on that server and I couldn’t be bothered working out how to run it on my PC. Other benefits include reducing RAM use on the PC (32 GB is already tight) and still being able to browse the webapp on my phone if the PC is asleep.

You might wonder why the search proxy is on the PC rather than the dedicated server. The answer is that as part of their measures to handle bots, some search engines treat requests coming from datacentre IPs disfavourably. This way the queries come from my residential IP. Arguably I'm obliged to move to a paid provider here. Selfishly, I’ll admit it’s nice for privacy to have my searches blend in with the rest of my ISP’s CGNAT traffic rather than having every single one of them tied to my particular API key.

I’ve been running this setup for about a week and it’s working great. The local Qwen helped me with some parts of the installation like deploying a new proxmox VM running Open WebUI, and building/testing that “mysearx.exe” port.

If that wasn’t enough, I believe there are good reasons to expect this hardware setup to get significantly better in the future, just through model and software improvements.

It is anticipated that Qwen4 will be released in the coming months. In Qwen3.8-Flash-Next they previewed a technology where they use a separate N-gram embedding layer. In this scheme, all of the intelligence doesn’t have to be resident in VRAM. You can keep this lookup table separately in normal system RAM or even on a fast SSD and boost the effective parameter count with essentially no slowdown in token generation. The big question is whether we will get a new open weights 27B release taking advantage of these newest developments.

Further ahead, research is continuing apace into “latent reasoning” methods. To handwave considerably, think about the verbose English gibberish produced by Qwen3.8 when it’s reasoning its way to a solution. If this output isn’t intended for me to read, why is it in English? It’s a computer program—intuitively, surely it could more efficiently compute its way to the same result using some compact binary representation of the same intelligence? There are many practical problems to solve and I’m not aware of any major deployments of this technique. Give it a couple of years—maybe we’ll see a leap forward in what we can get out of the same hardware, using this or any other novel inference efficiency technique.

While putting this new system together I learnt several things and ran into a bunch of minor issues along the way. In no particular order:

• Try not to look at Qwen’s reasoning traces too much. For a human reader they are infuriating. It will keep telling itself that it’s going to write out the source code, and I quote, “this time for real”, and then it will suddenly realise there’s some detail it hasn’t considered yet. This repeats many, many times.
• Qwen3.8-27B has a quirk with its chat template where while it advertises reasoning levels “low”, “medium” and “xhigh”, the only two effective modes in practice are low and xhigh. If you set it to medium it will act like it’s on xhigh.
• At first I aimed for 256k context with Q4 K/V. It started to lose its way and get into loops as the context grew. Q8 has been much more dependable and 128k is plenty most of the time.
• Unlike a standard OpenAI endpoint, Qwen3.8 needs all the reasoning to be replayed back to it in the requests. This is known and generally supported by consumer software but you often have to turn it on explicitly. Without that reasoning you get poorer responses and poorer caching
• In Open WebUI, correctly replay the reasoning by going to Connections > Edit your OpenAI API connection > Advanced > Provider and ensuring the dropdown says llama.cpp.
• OMP is trickier—it has some heuristics around whether a model is “local” and if you use a Tailscale IP as an endpoint this is detected as “non-local”. Even if it knows it’s talking to Qwen3.8, it chooses not to send the reasoning traces in its requests. Force it with replayReasoningContent: true. Confusingly, the UI mostly shows the context use as reported by llama.cpp, while locally it’s tracking context use with reasoning included. This causes weird bugs where it appears to compact (run auto-handoff) while it’s still 10% or more before the configured compaction point.
• Open WebUI needs a special tweak to put Qwen into low reasoning mode. Go to Models > Edit your Qwen3.8 model > Show Advanced Params. Then add a new one with name chat_template_kwargs and content {"enable_thinking": true, "reasoning_effort": "low"}.
• OMP has a few features that send additional prompts to your model beyond what you type, which can be disruptive when you have only one slot and it potentially ejects your entire working session from KV cache. The worst offender is the automatic session title generator. There’s a command line option to turn it off, or I use the envvar PI_NO_TITLE=1. Less cache-damaging is the “idle recap”, which you can turn off under settings. Also turn off Async Compaction so it doesn't compete with your regular work as your context gets close to full.
• If you use --sleep-idle-seconds on llama.cpp then it will automatically free up your VRAM for other purposes when there haven’t been any LLM requests for a while.

For reference, here are my OMP config.yml and models.yml, and the full command line I’m currently using for my llama.cpp instance: Gist

I assume that one day there will be turnkey solutions that provide most of this as a convenient bundle but today you’re kind of left to wire it all up yourself. It’s actually quite fun—unlike most things in computers it doesn’t just work or not work. It’s a proper heavy workload where you can tweak knobs to make things faster or slower or trade off different properties against each other. Maybe it works but it’s suboptimal and you won’t notice unless you’re paying attention. You need to follow recent developments or you might not know that you could use something like speculative decoding.

I’ve never been a car person and never quite understood how “working on the car” constituted a hobby but with local LLMs I feel like I’ve discovered the digital equivalent. Good times.