Local AI: its moment has arrived, but not for everyone

Local AI is no longer science fiction: installing Qwen on your PC comes with a catch

Five minutes to subtract two columns. That was the experience of someone who installed a local model and ended up uninstalling it. At the opposite extreme, another machine runs Qwen at 140 tokens per second with a high-end GPU. Local AI is no longer a lab rumor, but the question of whether it's worth it has only just begun.

The trigger is the arrival of Qwen 3.8, whose benchmarks once again put the Chinese industry in the lead. With Ollama and the downloaded weights, anyone can skip Claude's monthly fee. The price: thousands of euros in hardware and an electricity bill nobody fully quantifies. There, the cloud still wins.

Real-world performance: from marvel to ridiculous

The figures circulating are dizzying: a 9070 XT with a model quantized to 3 bits reaches 140 tokens per second, though it drops to 15 when the context fills up with 262,144 tokens. For automated tasks, a 7B Q4 model runs on a MacBook M2 while its owner sleeps. Even a 2013 Mac Pro works with a custom-built vector architecture.

But not everything is wins. The same tool that handles complex processes takes five minutes on a simple subtraction. It's not magic: it's young software and uneven configurations. The gap between the benchmark and what happens in your tower remains huge.

Privacy versus cost

The strongest argument for going local isn't savings: it's privacy. No monitoring, no subscriptions. But anyone who just wants quick answers will stay in the cloud. "Better to rent than to buy," the pragmatists conclude. Local AI is a bet for enthusiasts, not a mass-market product. At least until phones ship with chips capable of running it.

With finetunes on Hugging Face already surpassing the general intelligence threshold, the temptation grows. But how many are willing to wrestle with Python and their electricity bill to avoid the monthly fee?