What Running a Local LLM Taught Me About Attention

There is something clarifying about watching your laptop fans spin up as a language model processes your prompt. The abstraction dissolves. This thing runs on electricity, takes time, and gets hot.

I spent a few weeks running Mistral and LLaMA variants locally using Ollama, mostly out of curiosity about what it would feel like to own the inference rather than rent it. I was not expecting it to change how I think about these systems.

The model doesn’t know it’s waiting

The strangest part was noticing that generation speed is deeply uneven. Some tokens come fast; others take a long beat. When I asked a model to compare two options, I could almost feel it “deciding” mid-sentence — not because anything like cognition is happening, but because the probability distributions at certain tokens are genuinely flatter than others. The model has to work harder to commit.

The delay isn’t latency. It’s the model encountering a fork in the road where no path is obvious.

This is the thing that gets lost when you use a hosted API. The interface homogenizes the output. Every response arrives at roughly the same smooth speed, preprocessed and streamed to feel natural. On local hardware, the seams show.

Ownership changes the relationship

When I’m paying per-token for API access, I’m incentivized to write tighter prompts. When the only cost is electricity and a bit of RAM, I find myself actually experimenting — running the same prompt a dozen times with small variations, watching how the outputs diverge, asking follow-up questions I’d normally not bother with.

This is probably how most good intuitions about these systems form: through repetition and patience. It’s the same way you learn the quirks of any tool you use daily.

What I’m still not sure about

I don’t think local models will displace hosted ones for serious work anytime soon. The gap in capability is real. But the shift in posture — from user to something closer to operator — is worth something. You stop treating the model as an authority and start treating it as a collaborator with a known set of limitations.

That’s probably the right way to hold all of this.