Open-weight models vs closed APIs

The decision that shows up on day one

Every AI feature starts with: call a hosted API, or run an open-weight model (yourself or via a host like Together/Fireworks/Bedrock)? “Open source” is often misused — weights can be downloadable while licenses still restrict commercial use. ShipAI uses open-weight when we mean downloadable parameters, and closed API when the model only exists behind a vendor endpoint.

flowchart LR
  Need[Product need] --> Cap{Need frontier capability?}
  Cap -->|Yes| API[Closed API first]
  Cap -->|Maybe| Hybrid[API + open-weight fallback]
  Cap -->|Narrow task| OW[Open-weight / SLM]
  OW --> Data{Sensitive data?}
  Data -->|Yes| Local[VPC / on-prem / local]
  Data -->|No| Hosted[Managed open-weight host]

Mental model: three planes

Plane	Closed API	Open-weight
Capability	Vendor’s frontier stack	Your chosen checkpoint + post-train
Data path	Leaves boundary unless enterprise contract	Can stay in VPC
Ops	Mostly their problem	Yours if self-host (GPUs, batching, upgrades)

Most teams start on APIs, then pull narrow workloads in-house when evals + unit economics say so.

What you actually trade

Dimension	Closed API	Open-weight (self-host or managed)
Capability	Usually strongest general models	Catch-up varies by size; fine-tunes can win on narrow tasks
Latency control	Vendor SLOs + your region	You own queueing, batching, GPUs
Cost shape	Per-token; easy to start, spikes with traffic	CapEx/GPU or host fees; cheaper at high steady volume
Data path	Leaves your boundary unless enterprise contract	Can stay in VPC
Ops burden	Low	High if self-host (see Inference track)
Customization	Prompting, tools, light FT products	Full LoRA/QLoRA, custom serving
Multimodal / tools	Often best DX	Check template + tool support carefully

“Open source” confusion (engineer’s checklist)

1. Weights available? Hugging Face / model card download
2. License — Apache/MIT vs Llama-style community license vs research-only
3. Training data / eval claims — read cards; don’t assume
4. Tokenizer & chat template — wrong template → silent quality loss (Tokenization)
5. Serving path — Ollama for local, vLLM/Triton for throughput
6. Safety / refusal behavior — re-eval; do not inherit vendor assumptions

Hybrid gateway pattern

A widely used production shape:

flowchart TD
  Client --> GW[OpenAI-compatible gateway]
  GW --> Router{Route}
  Router -->|hard / multimodal| API[Closed API]
  Router -->|narrow / sensitive| Local[vLLM open-weight]
  Router -->|batch| Cheap[Smaller model]

One client SDK; many backends. Pair with Serving and streaming and Inference cost/latency routing.

A practical default for ShipAI graduates

• Prototype on a closed API (fast iteration, structured outputs, multimodal).
• Own the narrow task with LoRA + local/open serving when evals beat prompt-only (guided Build & serve your SLM).
• Gateway pattern — one OpenAI-compatible façade over API + local models.
• Keep evals portable — same gold set across backends.

Failure modes

Failure	Cause	Fix
“We self-hosted to save money”	Ignoring GPU idle + eng time	Model total cost of ownership
Silent quality drop	Wrong chat template	Golden set on every backend
License surprise	Assumed Apache	Read the license before ship
Capability gap	SLM for frontier task	Keep API fallback
Data leak anyway	Logs still ship PII	Privacy controls

Tradeoffs

• API-first — speed; vendor + data dependency.
• Open-weight-first — control; slower path to frontier UX.
• Hybrid — usually correct for serious products.

Glossary

Term	Meaning
Open-weight	Downloadable parameters (license may still restrict use)
Closed API	Model only via vendor endpoint
SLM	Small language model suited to narrow tasks
Managed host	Someone else runs GPUs for your open weights
TCO	Total cost of ownership including eng/ops

Micro-project

Pick one product feature and decide API vs open-weight with a one-page decision matrix (capability, data, cost, ops, exit criteria).

Related guided path

Open-weight vs closed API. Later: Build & serve your SLM, Deploy, cost, latency, observability, Inference track (vLLM).