Why Matthew Berman Says Local AI Doesn't Add Up Yet, and What It Would Take to Make It Work

Berman opens with a blunt position: local AI makes no sense to him right now. He argues that telling the average person to cancel a ChatGPT subscription in favor of running AI locally would be a mistake, because local AI is expensive, hard to set up, and not as good as the frontier models.

He adds that he is not speaking as someone who lacks the means. He says he reaches for ChatGPT or Claude when he needs to get real work done, even though he is technical and has substantial computing resources. He also criticizes much of the local-AI discussion for focusing on which setups people have running rather than what they actually accomplished with them.

Berman says that if local AI fails, the consequences are a bigger problem than a single product choice. He argues he should be the easiest person to convince, since he has a large amount of compute, is very technical, and covers this topic for a living. He says he can spend as much time as needed to run local AI optimally.

He names two motivations: control over his data, and not wanting to be at the mercy of frontier labs that might cut him off one day. He states plainly that he really wants local AI to work and will use it every day if it does, and asks viewers to keep that in mind as he presents the criticisms and the math that follow.

Berman sets out the distinction before doing the math. Running a model locally requires an open-source, open-weights model, but open-weights models can also be run in data centers without being local. He defines local AI as running the model on your own home device, and notes that if you disconnect from the internet, a local model can still answer questions.

He explains that open-weights AI means the lab has released the model's formula and weights publicly, so anyone can use it. Large providers such as AWS, Google, and the neoclouds load these models onto their GPUs and serve them to customers. Berman says the best open-weights models sit about one generation behind the absolute frontier, and that many strong ones come from Chinese labs. Even so, he stresses that they are difficult to run: the strongest need racks of GPUs, and serving a few people can require hundreds of thousands to millions of dollars of hardware.

Berman says the first reason he is excited about local AI is data sovereignty, which he expects many people do not care about as much as they should. Using hosted AI such as ChatGPT or Claude means sending your data to the company providing the service.

He notes that people have already become used to sending data to big tech through Facebook, Instagram, Twitter, Google Docs, and Gmail. He says his own life is largely online and that he is generally comfortable sending his data to the cloud. He admits a part of him dislikes that and would prefer to keep all his data locally, but that it is not a major issue for him, and he thinks he underappreciates how important data sovereignty is.

Berman then pauses for a sponsor segment for JetBrains Air. He says vendor lock-in is his biggest annoyance and describes Air as letting users bring any agent, such as Claude, Codex, or Gemini, and run several in parallel. He ties this to the same concern: being dependent on a single company's platform.

Berman describes a second concern: access to AI models is controlled by the labs, which decide what users can do with their models and when. He says he has been frustrated by providers changing their rules, and says OpenAI has historically been better about letting subscribers use their subscription wherever they want, which he appreciates. He adds that if a lab's business incentives diverge from how users need to use AI, it will cut them off, which he describes as simply how capitalism works.

He concludes that data sovereignty and unbridled access are important for individuals but critically important for enterprise companies. He notes that most of what he discusses, especially local AI, will apply to individuals.

Berman says another benefit of local AI is that it works without internet access. For himself, he says this is not a big deal: he has consistent internet access when out of the house, cellular service, and Starlink if he goes somewhere remote. He describes a scenario where he loses internet and needs his AI as an edge case of an edge case.

He says he made a video a couple of years ago arguing that storing a model locally on a hard drive is a good protective, prepper-style measure, and he still thinks so. But he notes that in that scenario you probably don't need the absolute frontier model and don't care about speed, only that you can get information at some point.

Berman says he likes the idea of owning his intelligence and names one of his biggest fears about AI as the concentration of power. He argues that if everyone depends on a frontier lab for intelligence, power would concentrate to a degree that makes him very uncomfortable. He likes the idea of people generating their own intelligence at home, which he says would spread out the power a few people hold now.

He also addresses cost. He says that getting rid of a ChatGPT or Claude subscription removes monthly payments, which is nice, but that thinking local AI is cheaper than paying $20 or $100 a month to ChatGPT or Claude is simply false, and he promises to show the math.

Berman says he is trying to optimize for getting his work done, and that local AI is not doing that for him at present. To reach intelligence close to the frontier, he explains, you need some of the largest open-weights models in existence, which he describes as trillion-parameter models that basically do not run on consumer hardware.

Quantizing a model shrinks it so it fits on a desktop, he says, but at a small cost in quality. Even then, the output is limited in tokens per second, and parallel use is limited more because a single GPU must run inference for five or ten agents at once. He says this does not work with the current state of local AI unless you are effectively building a data center in your house.

He notes that racks of DGX Sparks or several 5090s could run a fairly beefy model. Even then, he says, you would not get the speed or total token output of a frontier model or lab, though it is close. That route requires spending a lot of money and time, and he calls the learning curve massive and the biggest issue at present.

Berman says many people praising local AI on social media talk about which model they can get running on their machine, but never mention speed. He asks how many tokens per second people are actually getting, and says that matters. At around 15 to 30 tokens per second with one agent, he says it is basically not usable beyond very simple question and answer. At 70 to 100 or more tokens per second, he says it is quite usable, but only with one agent, so running multiple tasks in parallel would spread that speed across them.

Berman says one benefit of local AI is that it can run 24 hours a day, and that it arguably should, because the upfront purchase of a powerful computer and massive GPU is a capital expense. To spread that cost over three to five years, you need as many tokens flowing through the hardware as possible. He calls that a really cool benefit, since you probably could not run a ChatGPT or Claude subscription around the clock without running out of quota and having to pay for a $200 or $500 a month plan.

He then turns to what he calls the biggest problem: local AI is basically for hobbyists. He says he is technical, has built and taken apart computers for much of his life, and has been an engineer for many years, yet local AI is still difficult for him. He says he enjoys some of the tinkering, but is much more focused on what he gets done, and that the ends do not justify the means. He wants to spend most of his time producing work rather than getting the tools ready to produce it.

Berman says he wants local AI to work so badly that he put his money where his mouth is by investing in LM Studio, a company built around helping people get the most out of local AI. He says it is by far the easiest product to use, but still not dead simple, since hardware differences across machines complicate things.

Berman walks through several hardware options. At about $5,000 for an RTX 4090, he says a compact Qwen 3.8 27B runs at 25 to 40 tokens per second, the lower bound of usable. He says that is fine for one task, but as AI assistance spins off multiple agents for one task, you basically cannot run them all.

He then covers an RTX 5090 with 32GB, which he says costs about $8,500 and is hard to find. He says it can run the same model at about 40 to 70 tokens per second, usable but only single-threaded. For an RTX 6000 Ada, which he describes as a prosumer or enterprise-grade chip at about $8,300, he gives 50 to 130 tokens per second for Qwen Coder Next, but again only one task at a time.

For an RTX Pro 6000 Blackwell at about $16,000, he cites 95 to 245 tokens per second on the coding specialist and 75 to 190 tokens per second on a GPT OSS model, which he notes is two years old. But the cutting-edge open-weights GLM 5.3 flash, he says, cannot run unquantized on any of these devices without sacrificing intelligence. He points out that you would be spending $16,000 that might last three or four years, and asks whether it would pay off compared with $100 a month for ChatGPT or Claude, which need no setup or maintenance and keep getting better, cheaper, faster, and offer more quota.

Berman says a $16,000 GPU that lasts three or four years makes the return on investment hard to justify against $100 a month for a closed subscription that requires no setup or maintenance. He adds that the labs keep improving their models, making them cheaper and faster, and giving users more quota.

He then discusses a Mac Studio. He says that while there is a multi-month waiting period for one, a model can run at about 25 to 50 tokens per second on a compact model, and that on the 256GB or 512GB configurations, which he prices around $10,000 and $15,000, you get about 20 to 55 tokens per second on the frontier-scale model. He calls that usable at best and notes that heavy users running multiple agents would struggle.

Berman presents a cost-versus-quality chart he assembled. For each model, he takes the top open-weights model that fits on a consumer computer and amortizes its hardware cost over three years, then compares it with what it costs to pay ChatGPT or Claude.

On the chart, Qwen 3.8 27B costs about $144 a month, assuming a $4,000 computer that lasts three years, which he says could last longer. It runs at about 50 tokens per second with an intelligence index score of about 34 on the Artificial Analysis index. Claude Haiku 5.5, which Anthropic had just released, costs about $27 a month, runs at 235 tokens per second, and scores about 10 points higher on the same index.

Berman explains that the best place on the chart is as high and as far left as possible, meaning the highest quality at the lowest price. He calls that the Pareto frontier and says the only models on it are closed models. He says the labs are offering incredible, inexpensive, fast models.

Berman says closed apps are convenient: you open the app and it works, it is synced across your devices, it works at home or elsewhere, and it keeps getting better. He lists Claude Code, Claude Co-work, and Claude Design as examples of tools that keep growing. He says he has been a critic of Anthropic but is stating facts here, and that this is a very difficult model to justify replacing with local AI.

He gives reasons the math is off. First, the labs are not making it easier for local AI to win. They are not campaigning against it, but they absorb the GPUs and memory allocation needed for massive data centers. That has created a supply constraint and sharply raised consumer GPU prices. He says a 5090 that once would have cost about $3,000 to $4,000 now sits near $7,000 to $8,000 for the same GPU, so the cost of running local AI has roughly doubled. But model quality has not doubled with it: the model quality is the same, and the tokens per second you get from that chip are the same, so the economics simply get worse.

Second, at least in America, he says starting an open-source AI company is very hard. Building a frontier model requires billions of dollars, and the business model is unclear: you release the model free, and hyperscalers or neoclouds can serve it without making the upfront investment. He says that leaves little economic incentive for open-source AI to reach the absolute frontier, and especially for local AI, because open-source companies need to serve inference themselves to earn money.

Berman asks who local AI is for today. He says he would love it to be for him, since he fits the profile of someone who would want local AI, and he has it running. He enjoys getting the model set up, figuring it out, tinkering, and fine-tuning it to his needs. But for real work, he still goes to closed-source models.

He says many people enjoy local AI mainly for the setup process, which he compares to overclocking. Overclocking, he explains, is something hobbyists do to squeeze more out of their computer by adjusting many settings to speed up their CPU and GPU. People who overclock are generally excited by the process itself rather than by the results. He says he has done it for years and is all for it, but that the people who are really into local AI are into getting it set up, loading the biggest model they can, and trying to get the fastest tokens per second from it. He says he has not seen many people talk about the actual work they get done with it.

Berman says local AI is for hobbyists today, and he wishes it were for more than that because he really wants it to work. He says the concentration of power happens with AI when people are not able to control their own intelligence, and urges viewers to understand how big a deal this is. His advice is to own your intelligence, own your data, and practice data sovereignty, rather than depending on another company to dole out intelligence to you.

He then proposes how local AI might work. He says the first step is for the operating system companies, Microsoft, Apple, and Google, to get their act together. They are best suited because they generally own the hardware and the operating system, and can handle hybrid AI well. He argues that to make local AI popular, people should not have to know they are using it. Sensitive data, for example, could fall back seamlessly to a local model if the cloud goes down or the internet is cut, and that would serve the general audience rather than tinkerers.

Berman says he is very bullish on this hybrid approach. As small models get better, faster, and smaller, and chips become bigger, better, and more efficient, local AI will become more capable, so more of the workload could move there.

He points to the enterprise, where he says the total volume of tokens is skewing heavily toward open-source models. However, he notes that the revenue is still going mainly to closed-source labs such as OpenAI and Anthropic. He thinks a similar pattern could appear with local AI.

Berman suggests a future of local routing across devices. On Windows, a Mac, or an Android phone, he imagines not typing tasks into a chat window but an ambient AI that runs in the background and gets things done all day, built seamlessly into the operating system interface. He says he would not have to think about whether a task runs locally or in the cloud, because it would be routed appropriately: whenever a task can be done with a local model, it is, and anything more goes to the cloud.

He says he is super bullish on AI assistance using local models more and more, because the tasks most people use AI for do not require the best possible model. He adds that giving computers their own environment is still a challenge. He points to products such as Muse, GrokBot, and Dot as compelling because their environments run in the cloud, persist around the clock, and keep each bot separated and secure. He ends by saying he really wants local AI to work but does not understand how it works today, and points viewers to his review of the latest open-source model.