Open-source stack for an AI feature: vLLM, Ollama, and graph orchestrators

Own the path, rent the spikes

Not every feature needs a frontier API. Public OSS ecosystems (vLLM-class engines, Ollama-class local runners, LangGraph-style orchestrators) let small teams own latency, data path, and cost — if they pick a thin vertical slice.

This is a pattern guide, not an endorsement of one vendor’s roadmap.

Reference topology

flowchart TB
  subgraph Dev
    Ollama[Ollama / local GGUF]
  end
  subgraph Prod GPU
    vLLM[vLLM OpenAI server]
  end
  subgraph App
    API[Feature API]
    Orch[Graph orchestrator]
    Eval[Eval runner]
  end
  API --> Orch
  Orch --> vLLM
  Orch --> Vendor[Optional vendor API]
  Orch --> Tools[Tools]
  Eval --> Orch
  Dev -.->|same prompts| Orch

Principle: speak OpenAI-compatible HTTP everywhere so local and prod swap via base URL + model name.

When to use which runtime

Runtime	Best for	Watch-outs
Ollama / llama.cpp	Laptops, demos, CI smoke	Not multi-tenant scale
vLLM / TensorRT-LLM	GPU prod serving, batching, KV	Ops + GPU capacity planning
Vendor API	Spikes, vision, huge context	Data egress, cost, lock-in

Orchestration layer

Graph-style frameworks (LangGraph-class patterns) help when you need:

• Explicit states (retrieve → draft → verify)
• Checkpointing / durable execution
• Human-in-the-loop nodes
• Typed tool edges

Keep business logic in your modules; treat the framework as the state machine, not the product.

stateDiagram-v2
  [*] --> Retrieve
  Retrieve --> Draft
  Draft --> Verify
  Verify --> Draft: fail
  Verify --> [*]: pass
  Draft --> Human: low confidence
  Human --> Draft

Minimal “AI feature” vertical slice

1. One user-facing endpoint
2. One graph: retrieve → generate → validate schema
3. Local model for tests; vLLM or vendor in staging
4. Golden evals in CI (even 30 cases)
5. Traces + cost fields

Tradeoffs

Choice	Gain	Cost
All OSS	Control, privacy	GPU on-call
All vendor	Speed	Bill + egress
Hybrid route	Flexibility	Two failure domains
Heavy framework	Fast graphs	Upgrade churn

Failure modes

• Different tokenizers / templates between Ollama and vLLM → silent quality drift (pin chat templates)
• “Works on my 24GB laptop” ≠ prod concurrency
• Orchestrator retries amplify tool side effects
• Eval-free prompt edits across local/prod configs

What to ship this sprint

• OpenAI-compatible client wrapper with env-based base URL
• Docker compose: app + one local model runner
• Staging vLLM with a single quantized model
• Graph with max-step + timeout
• CI eval job on CPU small model

Staff framing

OSS is a control plane for learning and for regulated data paths. Use it to earn the right to scale — not as a religion against APIs.