Local AI: the gap between marketing and real teraflops

Local AI: what fits on your computer and what is pure marketing

Running a language model on your own computer without an internet connection is possible. Running ChatGPT is not. That distance —between what the industry sells as "local AI" and what actually fits on a home machine— underpins an 84-reply discussion that starts with a terse "AI without internet" and ends up uncovering a semantic war over what training means, what a model is, and why the fine print of the big platforms matters more than their demos.

The first reply to the question is a blunt verdict: "Pure garbage". But the thread soon changes course. LM Studio, Ollama and the Gemma and Qwen models come up as real tools that run on a laptop with 16 GB of RAM. The technical point that keeps coming up: rely on VRAM, not on CPU or conventional RAM. On Macs with M-series chips, unified memory does that job, though not without controversy.

What you can actually do today with a local AI

It's not all smoke. The most concrete case in the discussion is a program that checks the BOE (Spain's official state gazette) daily, filters by keywords and passes the text to a local AI on a 16 GB MacBook for a second screening. The AI returns a JSON with what is probably of interest to the user and what to do about it. The scraping is traditional; the model only classifies. And it works.

That's the pattern that repeats: repetitive, simple tasks with a closed output format. Web scraping, filtering, classification. For that, today's small models are "more than enough". The awkward question comes from another participant: if the same thing can be done with a scheduler and two prompts to ChatGPT, what does a local agent add? The answer is privacy and the absence of per-token cost, but also quality: local models fail more than a carnival shotgun the moment you ask them anything open-ended.

The language trap: training is not what they tell you

This is where the thread gets interesting. A technically savvy participant complains that training no longer means what it used to miccionan. Training used to be backpropagation, weight tuning, thousands of GPUs and millions of euros. Now anything that affects the output is called training: orchestration, prompts, fine-tuning of the last layers. "It's borderline fraudulent", he concludes.

The distinction matters because it explains why a local AI will never be a Gemini. Commercial models don't run on your phone: your phone sends a request to a data center with GPUs worth tens of thousands of euros each. The comparison used is brutal: fitting a PlayStation 5 game into a Spectrum. You can simulate something that looks like AI, but it's another dimension.

The hardware: unified memory, modularity and the end of the tower

The discussion drifts toward architecture. Today's Macs are practically a system on chip: everything together, powerful cooling, multi-phase power delivery. A traditional PC separates RAM, CPU and GPU with buses that introduce latency. One participant argues that unified memory is "a piece of shit" compared to segmented memory, and that it's only fast because of bus width and frequency. His analogy: a diesel engine can pull more than a petrol one, but not because it's diesel.

The debate then turns to the future of the modular PC. If manufacturers start putting out Apple-style machines, or Nvidia-style ones but cheaper, capable of running local AI and games, the classic tower stops making sense. Repairability suffers, but the universal power supply and removable storage remain the great arguments of the traditional format.

The M4 chip hack and the promise of the Neural Engine

One of the most striking final contributions is a news item: Apple's M4 chip has been hacked to unlock its hidden power. The Neural Engine, designed to run pre-trained models, was pushed beyond its limits through reverse engineering. A researcher developed his own intermediate language to communicate with the neural engine, bypassing Core ML and Metal. The result: 15.8 teraflops for training AI locally.

If that is confirmed and becomes widespread, the calculus changes. Not because you'll train a GPT in your living room, but because the hardware you already own can do more than its maker allows. The question is whether Apple will tolerate it or shut it down in the next update.

The verdict: useful, limited, and with an expiry date

The conclusion drawn from the whole exchange is that local AI is good for what it's good for, and not for what the industry promises. For automating specific tasks with sensitive data, it's a real tool. As a replacement for the big models, no. And the day home machines can run decent models without being a mega-corporation, we probably won't see it: as one participant jokes, by then we'll have starved to death.

What is clear is that the gap between the narrative and the teraflops isn't closed with YouTube tutorials. It's closed with hardware, and hardware has a price, wattage and physical limits that no prompt can bypass.