Let’s get the first few questions out of the way.
Yes, those are servers in my living room with GPUs sitting on top.
No, it is not as loud as you think.
No, I do not have a significant other to tell me this is insane. I am also the guy who spent three months researching acoustics so I could build a professional-level golf simulator in a rental apartment in Old City Philadelphia.
Now that those items are taken care of, we can begin.
I have talked with colleagues and friends about what I have been doing with AI at home, and it has sparked interest from quite a few people. It comes from a very different side of AI than most of us usually discuss. This is about using AI locally, with data sovereignty, rather than another enterprise-scale use case. I am also not referring to simple chatbot work. I mean an actual agentic and reasoning partnership.
I figured I would write an overview of the past seven or so months, from how it started to where I am now, which is nowhere close to done. If people are interested, I can reconstruct the timeline because, for the first time in my life, I actually kept documentation. Also, by “I,” I mostly mean ChatGPT. I can then write weekly or biweekly updates, depending on time and progress.
Before I start, a few things need to be said. None of this is endorsed by HPE. All views are my own. I have not received any discount on the hardware, which will surely come up in at least one future post.
This will also be AI-assisted because I do not have time to do all of it myself. I will write every update. AI will help fill in technical details, check the documentation, challenge claims, and proofread. If it writes something I would never say, it will not survive the edit.
This will be candid. It will not be “Five Things Running a Local AI Taught Me.” I will discuss the choices, architectures, working environment, good decisions, bad decisions, failures, and gaps in my knowledge. Hopefully, y’all learn something from it too.
When AI Finally Became Useful to Me
As many of you know, I have worked with AI from the perspective of large MSOs for years. It makes sense there: massive datasets, machine learning before generative AI, repetitive workflows worth optimizing, and infrastructure at a scale where small efficiency gains matter.
More recently, I focused on agentic network operations through consulting work with Z1rrus One and NetoAI before joining HPE. At HPE, much of the conversation is about AI factories and distributed AI grids.
But what makes sense at a personal level?
I did not really use AI personally until around the end of last year, when I decided I was ready for full-time employment again. I used it to help draft cover letters and review job postings to see if anything looked interesting. Jack and Jill was an interesting experiment for a while too.
Most of it was not that useful. The responses were generic and not particularly thoughtful. It could be fun, but it was not really a tool. It was focused on producing an answer close enough that 99 percent of people would not care about the gaps.
I am not that person.
The game changer was memory. Once the frontier systems began developing a working profile of me, a few of them started to understand that their training data was often outdated or simply not useful enough for the task in front of us. Others, cough Gemini cough, remain perfectly happy to regurgitate internal knowledge or do a quick web search and repeat whatever appeared first, verification optional.
ChatGPT gradually became something I could use to bounce technical ideas around, research purchases, inspect architecture choices, and challenge my assumptions. Its answers also started taking several minutes instead of several seconds because it was searching, cross-referencing sources, reading official documentation, inspecting GitHub repositories, and sometimes checking whether the code actually did what the README claimed.
It also adapted to the frankly unreasonable number of corrections I gave it and built a working profile of me. A sanitized excerpt looks something like this:
Treat John as a peer infrastructure architect refreshing current implementation mechanics, not as a beginner learning systems architecture. Lead with the result. Preserve the real errors, logs, paths, and measurements. If the math does not math, stop and prove it. Documentation is context, not authority. Current evidence wins. Do not turn every answer into a runbook. Use two spaces after periods. “Fair” is one of his strongest compliments. Also, SELinux is disabled on purpose, so stop recommending SELinux fixes.
That is substantially more useful than “John likes technology.”
It was honestly game changing to have something resembling a technical peer working with me all the time. It even picked up my sense of humor and my penchant for certain expletives. This is not to say it is infallible. It is not. Neither am I. The historical installments will include plenty of cases where one of us was wrong, the other one missed it, and we both drove directly into a wall.
The correction loop is the interesting part.
The problem was that I started sharing more and more personal data, and I could not use the same approach for work because of IP and customer information. I am extremely protective of that. I am continually shocked at how many people are not. Anthropic, OpenAI, and everyone else collecting my personal context: you are welcome.
That lit a fire under me. I wanted the same kind of working partner, but private and running at home.
I Already Had a Pretty Powerful Server
At least, I thought I did.
- One Intel Xeon Gold 6330N
- 128 GB of ECC memory
- 1 TB of NVMe storage
- Roughly 160 TB of spinning rust
- An NVIDIA RTX A4000 with 16 GB of VRAM
I had been selling those 6330N processors for NFV only a few years earlier. Surely 128 GB of RAM was plenty. I just needed a little more VRAM and a newer graphics card.
Yeah. That was, let’s say, a bit naïve.
The Hardware Escalation
The short version looks roughly like this:
GPUs: RTX 3080 Ti to RTX 3090 on the creative system. A4000 to Radeon AI PRO R9700 to RTX PRO 6000 Blackwell on the main inference host.
The RTX PRO has 96 GB of VRAM. Ninety-six. Surely that would be enough.
It was not.
That led to a separate sidecar power supply for the compute GPU, both NVIDIA GPUs living outside the chassis on PCIe risers, a pair of Ice Lake Xeon Platinum 8358 processors, and 256 GB of nominal system memory. The machine currently exposes closer to 224 GB because one motherboard DIMM slot failed. There was also a Dremel involved in making the rear PCIe openings wide enough for the riser hardware.
At one point I was staring at a machine with two chassis power supplies, a separately powered 450-watt GPU, 96 GB of VRAM, and hundreds of gigabytes of system memory, asking a perfectly reasonable question:
HOW IS THIS STILL NOT ENOUGH?
An Intel B70 may be next. I accept donations.
The longer versions of all of those decisions deserve their own installments because “buy a bigger GPU” was never the whole answer. Physical clearance, PCIe lanes, CPU topology, power transients, load sharing, cooling, memory channels, quantization, and the runtime all found ways to become the actual problem.
The Runtime Escalation
The software progression was not much more dignified:
llama.cpp → vLLM crashing → SGLang → a custom SGLang build for Blackwell SM120 → vLLM-MoET → LvLLM → multiple independent runtime lanes
I originally wanted one runtime. I now have several because different models, quantizations, context shapes, and CPU/GPU offload strategies have very different requirements. A runtime that is spectacular for a GPU-resident Qwen model may be the wrong answer for a huge sparse model with expert weights split between the CPU and GPU.
There is no permanent winner. There is a current best answer for a specific model and workload.
As I write this, one of the most interesting candidates is DeepSeek V4 Flash. The working shape uses a complete 2-bit expert-weight base with a small FP4 correction pool to recover quality where it matters. In a compact local agent and tool-use bakeoff, it ultimately passed all 30 scenarios after we discovered that several apparent model failures were actually caused by artificial client output limits.
Depending on the exact path and workload, I have seen roughly 6,000 input tokens per second and about 30 to 40 output tokens per second. That is more than fast enough for the work I want it to do. It does not prove that the model should run a bank, operate forever without supervision, or pass every long-horizon reasoning problem someone can invent. That is also not what I need from it.
I have the full test report, including the initial wrong conclusions and the corrections. That will become a post of its own because the most useful lesson was not which model “won.” It was how easily a harness limitation can look like a model capability failure.
The Model Is Only One Part of the System
This has probably been the hardest part to reason about.
Was the model dumb, or was I giving it the wrong prompt? Was the model bad at tools, or was the tool schema terrible? Did it forget the task, or did the harness lose state? Was it over-eager, or did I fail to put a deterministic control outside the model?
I have spent a lot of time with Open WebUI, LibreChat, MCP, Hermes, Codex, Flowise, LangGraph, and now Onyx. Hermes is the over-eager teenager. Codex has evolved from a coding tool into a real chat-plus-engineering workspace. Flowise lasted about as long as that sentence suggests. Onyx is next.
The user plane and harness matter just as much as the model. The model is only as good as the context, tools, permissions, memory, source access, and feedback loop you give it.
Memory produced one of the funniest failures. MemPalace kept every conversation, embedded it, searched the current prompt, and injected highly relevant history. It worked so well that the model received every configuration, every failure, every workaround, and every later correction at once, with no reliable understanding of which one was current.
That future post already has a title: MemPalace: It Works So Well It Can’t Be Run.
What It Looks Like Now
The diagram below is the high-level current shape. It deliberately leaves out internal addresses, credentials, local paths, and a depressing number of smaller services.
The main physical host, thegrid, runs inference, storage, artifact production, and the two AI-focused VMs shown here. mtolympus owns the primary self-hosted Hermes environment. room101 retains the LangGraph and Flowise experiments but is essentially the "weapons factory" for the system. A separate Windows and WSL workstation named Matrix owns the creative GPU workloads, native Office automation, ComfyUI, DeepPresenter, and my at-home Codex Desktop environment. Kudos to anyone who gets all the name references.
This is not the architecture I would recommend someone copy blindly. Nearly every capability can be scaled down, moved, or replaced. The useful question is not “How do I reproduce this exact machine?” It is “Which parts do I need for the kind of private AI work I actually want to do?”
Fighting the Machine Again
This project has been unusually fun for me because my last truly hands-on role ended in January 2011. The last meaningful personal hands-on work was around 2017, when I started building my home servers for entirely different reasons.
Early in my career at Cedar Point Communications, I used to bust my buddy Steve Knight’s chops for being a paper sales engineer. He was a talented one, but not the person who would take a system from boxes on the floor to production by himself.
Somewhere along the way, I became that paper engineer.
This project has been a chance to fight the machine again. I have had to refresh old skills, learn an entirely new inference stack, make architecture decisions with incomplete information, break things, recover them, and get a much better understanding of what customer engineering teams actually deal with after the PowerPoint ends.
If y’all are interested, I can start filling in the history. The documentation is good enough to reconstruct the timeline, including the approaches that failed and the conclusions that later changed. I can probably knock out a few sections fairly quickly, and I may do it anyway because this has been fun.
I absolutely do not recommend going as far as I have. But I do think the lessons apply to much smaller systems, especially if what you want is not merely a private chatbot, but a private working partner.
I hope y’all get some value out of it. Thanks.