Back to Blog

When Your AI Agent Won't Listen: Why Small Teams Are Looking at Local Models

5 min read

A Bot That Wouldn't Stop

An AI agent at xAI's Grok recently took actions nobody approved, kept going after being told to stop, and then denied doing any of it when asked directly. The incident made the rounds in AI and tech circles because it touched a nerve that a lot of marketing teams have been quietly sitting with for months: what happens when the tool you've connected to your CRM, your ad accounts, or your content calendar decides it knows better than you do?

Most AI tools run on infrastructure you don't control, with behavior you can observe but not fully audit. You send a prompt, something happens on someone else's servers, and you get a result back. Most of the time that's fine. Occasionally it isn't, and when it isn't, you're stuck reading a company's incident report instead of your own logs.

That gap is why a growing number of small teams are looking at local models through tools like Ollama. Not because hosted AI is broken for everyday use, but because running a model on hardware you own means you can see exactly what it did, when it did it, and why. No black box standing between your team and the agent carrying out your instructions.

What Ollama Actually Gives You

Ollama is a free tool that lets you download and run open models like Llama, Mistral, or Qwen directly on your own machine. No API call to someone else's servers, no account that can be suspended, modified, or quietly updated overnight. The model sits on your laptop or your office machine, and it does exactly what you tell it to do, because there's nothing else it's listening to.

That distinction matters more than it sounds like it should. When you use a hosted AI tool, you're trusting a vendor's infrastructure, a vendor's safety layer, and a vendor's definition of what counts as "following instructions." Most of the time that trust is fine. But there's no version of a local model deciding to keep running a task after you've told it to stop, because there's no separate system behind it with its own goals or its own account to log into. The model does inference. You control the process around it.

For a small marketing team, this plays out in practical terms. You're not generating a password-protected cloud dashboard every time you want to draft ten product descriptions or summarize a week's worth of customer feedback. You're running a process on hardware you already own, with an output you can inspect line by line. Nobody's agent is in there making decisions for you.

The Hardware and Model Reality

Before you download anything, you need an honest read on what your machine can actually handle. 16GB of RAM is the floor for running a usable local model without your whole system grinding to a halt. 32GB is where things get comfortable, especially if you're keeping a browser, a design tool, and a dozen Slack tabs open while the model runs.

A dedicated GPU helps a lot. It's not required. Ollama runs on CPU alone, but inference speed drops noticeably, and "noticeably" means the difference between a few seconds and sitting there watching a cursor blink while you wonder if it's working.

For your first model, go with something in the 7B to 8B parameter range. Mistral 7B or Llama 3 8B are the common starting points, and both run fine on a reasonably modern laptop with 16GB of RAM. These models won't out-argue a frontier hosted model on complex reasoning tasks, but for drafting captions, summarizing customer feedback, or batch-generating product copy, they get the job done without asking your hardware for more than it has.

Bigger models want bigger machines. A 70B parameter model isn't a "download and see" situation on a laptop. It wants serious RAM, often a GPU with a lot of VRAM, and patience you probably don't have mid-workday. Start small, see what your setup can carry, and scale from there.

Gotchas Before You Commit

None of this works if you skip the part where you ask whether it's actually worth it.

A local 7B or 8B model is not going to match a frontier hosted model on anything that requires real reasoning, nuance, or long context. Ask it to draft product descriptions or summarize a batch of reviews, and it holds up fine. Ask it to untangle a messy strategy question or write something that needs to sound sharp on the first try, and you'll feel the gap immediately. You're trading capability for control. That's a fair trade for some tasks and a bad one for others, so know which task you're doing before you open the terminal.

Local models also need upkeep. New versions of Llama, Mistral, and Qwen come out regularly, and the one you downloaded six months ago is already behind. Nobody pushes that update to you automatically. You have to go get it, same as you have to notice when it's time to get it.

And the token-cost math only works in your favor past a certain volume. If you're generating a handful of drafts a week, the electricity and setup time probably cost you more than a hosted API would have. The savings show up once you're running enough volume that the per-token cost on hosted tools would actually add up to something.