Ollama

What Ollama is (plain English)

Ollama is a local model runner: pull a tagged model, run a daemon, call an OpenAI-compatible HTTP API. It wraps quantized weights (often GGUF) so laptop CPUs/GPUs can run instruct models without assembling a CUDA serving stack.

Perfect for dev loops, demos, offline experiments, and CI smoke tests. Not a replacement for vLLM / TensorRT-LLM at multi-tenant production scale.

Analogy: Ollama is Docker Desktop for LLMs — pull a tag, run a daemon, hit an HTTP port. Great for local product iteration. Terrible as your only plan for 1k concurrent users.

flowchart LR
  CLI[ollama pull / run] --> Daemon[ollama serve]
  App[Your app / SDK] -->|OpenAI-compatible HTTP| Daemon
  Daemon --> Weights[Local quantized weights]
  MF[Modelfile] --> Daemon

One-sentence definition you can defend in an interview

Ollama is a local daemon that serves quantized open-weight models behind an OpenAI-compatible HTTP API, optimized for developer cost and latency, not multi-tenant GPU utilization.

If you cannot say ChatClient, Modelfile pin, num_ctx, and graduate to vLLM when…, you are describing a toy CLI — not a product-ready local loop.

Interview cue: Local runners optimize for developer latency and cost, not cluster utilization. Say when you would graduate to vLLM.

The problem it solves

Without a local runner, every prompt tweak costs money and network. Without an interface, swapping local → hosted rewrites half your app.

What “local” does not mean

How it fits LLM apps

Ship rule: keep your client behind an interface (ChatClient). Swap base_url from Ollama → gateway without rewriting prompts.

flowchart TB
  App[App / agent loop] --> Client[ChatClient interface]
  Client --> Local[Ollama base_url]
  Client --> Hosted[OpenAI / Anthropic]
  Client --> GPU[vLLM OpenAI-compatible]

Interface contract (inputs / outputs / invariants)

Three metrics that matter locally: TTFT, tokens/sec, RAM/VRAM resident. Two degrade modes if happy path fails: fall back to hosted gateway, or fail closed with a clear “local daemon down” UX — never silent cloud spend.

Architecture (what runs on your machine)

1. Daemon listens (default localhost:11434).
2. Model blobs live in local storage; tags point at digests.
3. Runtime loads weights into RAM/VRAM; runs generate/chat.
4. API mirrors OpenAI chat/completions shape for easy clients.
5. Modelfile creates a new tag with baked SYSTEM / parameters.

sequenceDiagram
  participant App
  participant O as Ollama daemon
  participant G as GPU/CPU runtime
  App->>O: POST /v1/chat/completions
  O->>G: Prefill + decode
  G-->>O: Tokens
  O-->>App: SSE or JSON

Mental model vs “real” serving

Tags, digests, and reproducibility

Floating tags like llama3.2 can move. For evals and CI:

1. Pull once; record digest / exact tag used.
2. Prefer a custom Modelfile tag (shipai-lab) that pins FROM + parameters.
3. Document the pin in the repo next to the golden set.

flowchart LR
  Hub[Registry / library] --> Pull[ollama pull]
  Pull --> Blob[(Local blob store)]
  MF[Modelfile FROM + SYSTEM] --> Create[ollama create]
  Create --> Tag[shipai-lab:pinned]
  Tag --> Blob
  App -->|model=shipai-lab| Daemon
  Daemon --> Blob

Quantization in one breath

Ollama typically serves GGUF (or similar) quantized weights so a laptop can load an 7–8B-class instruct model. Quantization trades bits for RAM/speed. That is fine for scaffolding RAG and agents; it is not a fair bake-off against a full-precision frontier API unless you say so in the eval write-up.

Pair with Open-weight vs APIs when the interview asks “why not always local?”

How to use

ollama pull llama3.2
ollama run llama3.2 "Say hello in one sentence"
# API daemon (often auto-started on desktop apps)
ollama serve

# OpenAI SDK pointed at local Ollama
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
resp = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Explain RAG in 2 sentences."}],
)
print(resp.choices[0].message.content)

Minimal ChatClient sketch

from dataclasses import dataclass
from openai import OpenAI

@dataclass
class ChatClient:
    base_url: str
    api_key: str
    model: str

    def chat(self, messages: list[dict]) -> str:
        c = OpenAI(base_url=self.base_url, api_key=self.api_key)
        r = c.chat.completions.create(model=self.model, messages=messages)
        return r.choices[0].message.content or ""

local = ChatClient("http://localhost:11434/v1", "ollama", "shipai-lab")
# hosted = ChatClient("https://api.openai.com/v1", os.environ["OPENAI_API_KEY"], "gpt-4.1-mini")

Same call sites. Different env. That is the whole point.

Modelfile (stable system prompts)

FROM llama3.2
PARAMETER temperature 0.2
PARAMETER num_ctx 8192
SYSTEM You are a terse coding assistant for ShipAI labs.

ollama create shipai-coder -f Modelfile
ollama run shipai-coder

Pin custom tags in demos so classmates don’t fight default system prompts.

Collision rule: if the app also sends a system message, you now have two system layers. Pick one owner (usually the app) or document the merge order.

Useful knobs to know

When Ollama is the right tool

Walkthrough: local RAG loop

1. Pull a small instruct model + embedding path (or embed via sentence-transformers).
2. Store chunks in Chroma.
3. Point chat client at Ollama.
4. Retrieve → pack citations → generate.
5. When concurrency matters, keep Chroma; swap only the LLM base_url to vLLM/hosted.

flowchart LR
  Q[Question] --> Ret[Chroma retrieve]
  Ret --> Pack[Pack context]
  Pack --> Ollama[Local generate]
  Ollama --> Ans[Answer + citations]

Ship rule: measure retrieval (recall@k) separately from generation quality. A weak local generator can still validate your retrieve → pack path.

Walkthrough: cheap agent tool loop

1. Define 2–3 idempotent tools with strict JSON schemas.
2. Run ReAct locally against Ollama; log every tool I/O.
3. Cap max_steps and num_predict.
4. When tool schemas break, fix schemas before blaming the model.
5. Re-run the same trajectory against a hosted model via ChatClient to see quality delta.

Pair with Agents and ReAct and LangGraph and LangChain patterns once the loop is clear.

Walkthrough: CI smoke without cloud keys

1. Install Ollama in the job image (or cache binary).
2. ollama serve in background; wait for / health.
3. Pull a tiny pinned model (document digest).
4. curl chat/completions with a fixed prompt; assert non-empty assistant text.
5. Fail if env still points at paid cloud (OPENAI_BASE_URL guard).

This proves “client wiring works” — not model quality.

Alternatives

Production gotchas (even for “just local”)

• Model tag drift — llama3.2 can move; pin digests for reproducible evals
• Context length — default num_ctx may truncate long RAG prompts silently
• Resource contention — browser + IDE + model → thrash; set caps
• Not multi-tenant safe — one daemon, shared filesystem models
• API compatibility gaps — not every OpenAI feature exists; test tools/JSON mode
• GPU layers — misconfig → everything on CPU and “Ollama is slow” myths
• System prompt fights — Modelfile SYSTEM + app system message collide
• Eval unfairness — local 3B scores compared to GPT-class APIs
• Silent cloud spend — wrong base_url in .env during “local” runs
• Keep-alive thrash — unload mid-demo → cold-start pauses

flowchart TD
  Slow[Slow / weird local LLM] --> C{Check}
  C -->|CPU only| G[GPU layers / VRAM]
  C -->|Truncated RAG| X[Raise num_ctx]
  C -->|Wrong vendor| B[Verify base_url]
  C -->|Template fight| S[Modelfile vs app system]
  C -->|Cold start| K[Keep-alive / preload]

Failure modes checklist

1. Client hits cloud OpenAI by mistake (wrong base_url)
2. Chat template / Modelfile SYSTEM fights your app system message
3. Eval scores from local 3B compared unfairly to GPT-class APIs
4. Shipping Ollama in Docker as “prod” without capacity planning
5. Tool-calling assumed 1:1 with OpenAI; schema rejected at runtime
6. RAG packs exceed num_ctx; model answers from truncated head/tail
7. Multiple models kept warm → laptop OOM / swap death

OpenAI compatibility gaps to test

Do not assume parity. Explicitly test:

• Tool / function calling
• JSON / schema mode
• Streaming SSE event shapes
• max_tokens / context overflow errors
• Vision / multimodal if you need it
• Parallel tool calls vs sequential

Keep a compatibility matrix in the repo for your pinned Ollama version + model tag.

CI pattern that works

# Shape — job spins daemon, pulls tiny model, smoke-tests chat
ollama serve &
sleep 2
ollama pull tinyllama
curl -sf localhost:11434/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"tinyllama","messages":[{"role":"user","content":"ping"}]}'

Pin the model digest in CI docs. Fail the job if base_url accidentally points at cloud.

Guard against paid APIs in “local” jobs

# Shape — refuse if base_url looks like a vendor host
case "$CHAT_BASE_URL" in
  *openai.com*|*anthropic.com*) echo "refusing cloud base_url in local CI"; exit 1 ;;
esac

Resource budgeting on a laptop

Rough intuition: context memory grows with num_ctx × layers × precision; doubling context is not free. See KV-cache, prefill, decode for the serving-side version of the same idea.

Observability for local runs

Even on a laptop, log:

Promote the same fields into OpenTelemetry for LLMs when you leave localhost.

Debugging playbook (first hour)

Ship rule: verify base_url before you rewrite prompts.

Anti-patterns

1. Ollama as sole prod plan — no batching story for multi-user SLOs.
2. Floating tags in golden evals — unreproducible scores.
3. Dual SYSTEM owners — Modelfile + app fighting silently.
4. Fairness theater — scoring 3B GGUF against GPT-class as equal.
5. Skipping compatibility tests — tools work in OpenAI, blow up locally.
6. Unbounded agent loops on laptop — thermal throttle + swap.
7. Logging full prompts to shared CI artifacts — secrets in tickets/docs.

Security and privacy threat note

• Local ≠ air-gapped unless you control network and disk.
• Shared lab machines: model blobs and prompt caches can retain sensitive text.
• Binding the daemon beyond localhost without auth is an open LLM endpoint.
• CI logs of full prompts can leak API keys pasted into “test” messages.
• Catch layer: env allowlists for base_url, redaction in logs, and never expose :11434 on a public interface.

Noisy tool results or prompt injection still apply locally — schema validation and HITL belong in your agent loop, not in Ollama.

Hands-on next steps

1. Pull a small instruct model; hit /v1/chat/completions.
2. Build a Modelfile with fixed SYSTEM + temperature + num_ctx.
3. Point the same client at a hosted API; compare quality and latency.
4. Fill a one-page compatibility matrix for tools/JSON/streaming.
5. When concurrency matters, re-read the vLLM article.

Micro-project

1. Create shipai-lab Modelfile with pinned FROM, temperature, num_ctx, SYSTEM.
2. Call it from Python via OpenAI SDK behind ChatClient.
3. Run the identical messages against a hosted model behind the same interface.
4. Log TTFT + total latency for both; write a 5-line “graduate when…” note.
5. Add a CI smoke that refuses cloud base_url.

Interview whiteboard: local vs serve

Two columns:

Say out loud: “We keep ChatClient; only base_url and model id change.”

Interview prompts you should be able to answer

1. What does OpenAI-compatible buy you, and what does it not guarantee?
2. When do you graduate from Ollama to vLLM — name measurable triggers.
3. How do you make local evals reproducible across machines?
4. Why might RAG “lose” citations when num_ctx is too small?
5. How would you prevent CI from silently calling a paid API?

Common interview traps

• Claiming Ollama “is vLLM for laptops” (wrong optimization target).
• Ignoring quantization when discussing quality gaps.
• Designing multi-tenant SaaS on one shared local daemon.

Tradeoffs summary

Graduate-when criteria (example)

Ship the note, don’t keep it in your head:

1. p95 TTFT or tokens/sec cannot meet UX at N concurrent users.
2. You need tenant isolation or authenticated multi-user access.
3. Tool/JSON features you require are unreliable on the local pin.
4. Eval quality plateau is clearly model-capacity, not prompt/RAG.

Checklist

• ChatClient abstraction in place
• Modelfile tag pinned for demos/evals
• num_ctx sized for your RAG packs
• Compatibility tests for tools/JSON you rely on
• Written “graduate to vLLM when…” criteria
• CI cannot silently call paid cloud APIs
• TTFT / latency logged for local vs hosted bake-off
• Daemon not exposed beyond localhost without auth

End-to-end lab checklist (do this once)

• ollama pull + create custom tag from Modelfile
• Python OpenAI SDK chat against /v1
• Same messages via hosted path; table of TTFT + quality notes
• Tiny CI smoke with digest pin + cloud base_url guard
• One RAG or agent loop that only swaps base_url to leave local

Production readiness checklist

• Pin model digests / Modelfile tags in repo docs
• Compatibility matrix checked into git
• Resource caps documented for demo machines
• Explicit non-goals: no multi-tenant Ollama prod
• Observability fields ready to promote to OTel
• Fallback path (hosted gateway) defined for demos that must not fail live

Glossary

How it works end-to-end (request lifecycle)

1. Daemon up (ollama serve or desktop app).
2. Model tag resolved to local digest / blobs.
3. Weights loaded into RAM/VRAM (cold vs keep-alive).
4. Chat request arrives OpenAI-compatible JSON.
5. Runtime applies Modelfile parameters + messages.
6. Prefill + decode on CPU/GPU layers.
7. Stream or JSON response; optional unload after idle.

sequenceDiagram
  participant App
  participant Daemon
  participant Disk
  participant HW as CPU/GPU
  App->>Daemon: chat.completions
  Daemon->>Disk: resolve tag → blobs
  Daemon->>HW: load if cold
  HW-->>Daemon: tokens
  Daemon-->>App: SSE/JSON

Instrument TTFT separately from total latency — cold loads dominate laptop demos and confuse “Ollama is slow” narratives.

Modelfile as a product contract

Treat Modelfiles like Dockerfiles for behavior:

Ship rule: app-level system messages should not fight Modelfile SYSTEM. Pick one source of truth. For labs, bake lab instructions into the Modelfile; for products, keep SYSTEM minimal and put policy in the app.

Version Modelfiles in git. The custom tag (shipai-coder) is what clients pin — rebuild the tag in CI when the Modelfile changes.

Local embeddings + chat split

Many apps need:

• Chat model via Ollama
• Embedding model via sentence-transformers / Hub (or Ollama embed if you standardize)

Do not assume one Ollama tag does both well. Record both ids in traces. Rebuild vector indexes when the embedder changes — same rule as any vector database pipeline.

flowchart LR
  Docs --> Emb[Embedder service]
  Emb --> VDB[(Chroma / pgvector)]
  Q[Query] --> EmbQ[Same embedder]
  EmbQ --> VDB
  VDB --> Pack[Pack]
  Pack --> Chat[Ollama chat tag]

Agent loops on Ollama

Cheap local tokens are perfect for ReAct / tool-calling practice:

Pair with LangGraph and LangChain patterns or hand-rolled ReAct from Agents and ReAct. Graduate the same graph to vLLM by swapping base_url.

Networking and security (local is not “safe by default”)

Ollama on a developer laptop with repo tools attached is still a code execution surface if the model can call tools.

Observability for local runs

Even local demos deserve:

• model tag + digest
• num_ctx
• TTFT / total latency
• prompt/completion tokens (when API provides)
• whether GPU layers engaged

When you later compare to hosted APIs, these fields make the comparison honest.

Worked walkthrough: offline support demo

1. Modelfile shipai-support with terse SYSTEM + num_ctx 8192.
2. Chunk FAQs into Chroma.
3. Retrieve top-5; pack citations; generate via Ollama.
4. Script toggles LLM_PROVIDER=ollama|openai behind ChatClient.
5. Stakeholder demo runs airplane mode; same UI.

Anti-patterns (Ollama edition)

1. Shipping the laptop daemon as multi-tenant prod
2. Unpinned tags in published eval tables
3. Comparing local 3B scores to GPT-4-class without labeling model class
4. Ignoring num_ctx while stuffing long RAG packs
5. Assuming tool-calling parity with OpenAI
6. Running three 7B models concurrently on 16 GB RAM “for speed”

Debugging playbook

Memory, cost, and sizing intuition

Local tokens are “free” until you count engineer time and electricity — still cheaper than burning cloud during agent debug.

What “good” looks like in a design doc

Local-dev section should state: pinned tags/digests, ChatClient swap plan, graduate-to-vLLM criteria, CI smoke pattern, and explicit non-goals (no multi-tenant Ollama).

FAQ (Ollama)

Is Ollama the same as llama.cpp?
Related ecosystem; Ollama adds daemon + UX + model library ergonomics.

Can I fine-tune inside Ollama?
Not your primary fine-tune platform — train with PEFT/Hugging Face, then consider serving options.

Does OpenAI-compatible mean identical?
No. Test the features you need.

When do I delete Ollama from the architecture diagram?
You don’t — keep it on the dev swimlane. Prod swimlane shows vLLM/APIs.

Deep dive: reproducible class labs

Teaching labs fail when every laptop pulls a different tip. Recipe:

1. Publish Modelfile in the repo.
2. Document exact ollama create command.
3. Record ollama show --modelfile / digest in LAB_VERSION.md.
4. CI uses the same tag.
5. Rubric grades trajectory shape, not absolute model elo.

End-to-end lab checklist (do this once)

• Pull tiny + one 7B-class instruct
• OpenAI SDK against localhost
• Custom Modelfile tag
• ChatClient env swap demo
• Measure cold vs warm TTFT
• Write graduate-when criteria

Putting what / why / how together

Related

Guided Build & serve your SLM. Key tech: Hugging Face, OpenAI / Anthropic APIs, Chroma. Inference track for real serving: vLLM, KV-cache, prefill, decode. Core: Open-weight vs APIs, Agents and ReAct. Observability: OpenTelemetry for LLMs.