/local LLMs

Why InferHaven runs your AI offline by default

Using a cloud API sends your repo to someone else. InferHaven flips that default: your code stays local, and the network is an opt-in, not an opt-out.

Ethan Ludwig 7 min read

When you point Cursor, Claude Code, or OpenAI Codex at the usual cloud APIs, your repo’s source can end up in someone else’s training corpus, and a model trained on it could hand pieces of your own code back to a stranger. The shape of the function you just wrote, the variable names that leak your domain, the comment with the bug ID in it: all of it passes through somebody else’s API. And as these models get more capable, the providers keep bolting on more filtering, more censoring, more guardrails.

You accept that quietly because the model is good and the loop from prompt to completion is fast. Sometimes that’s a fine trade. But, there are plenty of workflows it isn’t a trade you should be making at all. InferHaven flips the default. Out of the box the stack is offline. Inference runs against models on your own hardware. The network is something you opt into, not something you opt out of.

One command to a fully local stack

haven@laptop · ~/inferhaven-core 
$ docker compose up -d
[+] Running 4/4
  ✓ ollama       Started
  ✓ workspace    Started
  ✓ code-server  Started
  ✓ caddy        Started

$ haven pull qwen2.5-coder:14b
pulling manifest... downloading 8.5GB ✓

That’s the whole setup. Four services, one network, Caddy as the secure gateway in front of every request, SSH locked to your key.

The compromise the cloud forces you into

Don’t get me wrong: cloud coding assistants are great products. They’re also a trade a lot of teams take without ever really weighing it.

Hosted Cursor - Cline - Copilot

Default: cloud

  • Your code → vendor API
  • Telemetry on by default
  • Per-token billing
  • Model choice locked to vendor catalog
  • Outages take your tooling down
InferHaven (self-host)

Default: local

  • Your code stays on disk
  • Zero telemetry, ever
  • Flat hardware cost
  • Any GGUF or safetensors model
  • Offline-capable; no vendor dependency

We don’t just sell privacy. We ship privacy by default.

— An InferHaven essential

When you want the cloud, you still get it

Offline by default isn’t offline only. Drop your ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY, or OPENROUTER_API_KEY into the workspace and the bundled harnesses (Claude Code, OpenCode, Aider, QwenCode, Amp, Gemini, pi, Goose, Continue, avante.nvim) pick them up on their own. You decide which calls go where, per task and per harness, and you can run a dozen different models and providers without one project’s keys bleeding into another.

What’s next

This blog will be the running log of how InferHaven is built, what’s shipping, and the design decisions we made along the way. Float your boat up to the dock, clone the repo, run docker compose up -d, and let the lighthouse lead you to your haven.

— Ethan L.