Open-weight vs closed API

"Open source" is overloaded

LLM discourse collapses distinct concepts into one buzzword. Engineers who treat "open" as binary make bad architecture decisions — especially around license, data residency, and operational headcount.

ShipAI separates four terms you will use precisely:

Open weights ≠ open source end-to-end. You may download weights while the training data, eval suites, and safety filters remain proprietary. Your SLM module depends on picking a base model license you can fine-tune and serve for your intended use case (commercial vs. research, attribution requirements, field-of-use restrictions).

Callout — read the license, not the blog post: Meta, Mistral, Apache, and custom "community" licenses impose different obligations. A matrix row labeled "open" without naming the license is incomplete.

Closed API: when it wins

Strengths:

• Fastest path to quality — frontier models without owning GPUs.
• Vendor handles scaling, uptime patches, safety filters (imperfect but present).
• No weight storage, quantization, or CUDA debugging on your team.

Weaknesses:

• Data leaves your VPC unless you have enterprise/private deployment.
• Cost scales linearly with tokens — agent loops hurt.
• Behavior changes when vendor updates models; regression testing is your problem.
• Vendor ToS may restrict use cases (competitive products, certain verticals).

Fit signals: small team, need best quality now, prompts can leave the network under policy, variable traffic, no fine-tune requirement.

Open-weight self-host: when it wins

Strengths:

• Data residency — prompts stay inside your network.
• Cost at high sustained QPS can beat API (after ops amortization).
• Full control: fine-tune, distill, custom quantizations, air-gapped deploy.

Weaknesses:

• You operate inference (GPU pools, autoscaling, model registry, rollback).
• Base model quality may lag frontier APIs unless you invest in fine-tuning and eval.
• License compliance and security patching are yours.

Fit signals: strict privacy, predictable high volume, ML platform team exists, need custom behavior via LoRA/full fine-tune.

Fine-tune on open base (middle path)

Often the product answer is not API vs. raw self-host but:

1. Start with open-weight base (e.g., 7B–8B instruct model).
2. LoRA or full SFT on domain data (SLM module).
3. Serve via vLLM, TGI, or Ollama internally.
4. Still use closed API for hard queries or development velocity.

This hybrid appears in enterprise copilots: internal SLM for PII-heavy paths, frontier API for rare complex reasoning (with routing and eval gates).

Decision drivers (score these explicitly)

When you build the micro-project matrix, use these axes:

Traffic-light or 1–5 scores beat prose paragraphs — forces tradeoffs visible.

License risk (one sentence rule)

If you cannot quote the clause that permits your deployment, you do not have permission yet.

Ops preview (why Build & serve your SLM matters)

Self-hosting is not python app.py. Production inference involves:

• Model artifact storage and versioning.
• Quantization (INT8/INT4) for memory/latency tradeoffs.
• Batch vs. continuous batching.
• Health checks, rate limits, observability.

The SLM and deployment modules walk this path. This lesson's matrix prevents choosing self-host on ideology alone.

Worked example: support copilot traffic math (directional)

Suppose 200 engineers, 500 queries/day, average 2k input + 500 output tokens per query on a closed API at rough blended $2/M tokens (illustrative, not a price quote):

500 × 2500 tokens ≈ 1.25M tokens/day ≈ 37.5M/month
At $2/M → ~$75/month API inference alone

Same workload on a single small GPU instance (self-host) might cost $200–400/month fixed plus engineer time — cheaper only at higher QPS or when data cannot leave the VPC. The matrix forces you to plot your actual token volume instead of arguing "APIs are always expensive" or "self-host is always free."

Add fine-tune costs: GPU hours for LoRA, labeling, eval reruns. The SLM module makes those tangible.

Engineering problem (staff framing)

Model build-vs-buy: latency, cost, privacy, capability, ops. Wrong default ossifies architecture.

Diagram — API vs weights

flowchart TD
  Q{Weights on VPC?} -->|Yes| OW[Open weights + vLLM]
  Q -->|No| API[Closed API]
  OW --> Ops[You: GPU/quant/patch]
  API --> Vend[Vendor model / you: prompts+evals]

Precise definitions & mental model

Closed API vs open weights; licenses ≠ marketing "open".

Tradeoffs — when to use what

Failure modes (interview + on-call)

License violations; self-host 70B without batching; switch providers without evals.

Production & OSS practices

Hybrid common: frontier API for hard cases, local SLM for cheap QPS.

Deep dive (FAANG / OSS bar)

Push «open-weight-vs-closed-api» past tutorial depth: write the interface contract (inputs/outputs/invariants), list three measurable metrics, and name two degrade modes if the happy path fails. Add a short threat note: what an attacker or noisy tool result could do, and which layer catches it (schema, policy, HITL, or eval gate).

flowchart LR
  Contract[Interface contract] --> Metrics
  Metrics --> Degrade[Degrade modes]
  Degrade --> Threat[Threat + control]

Micro-project: Decision matrix

In m1/decisions/api-vs-weights.md:

1. Pick a fictional but concrete product — e.g., internal support copilot for 200 engineers, 500 queries/day, some containing proprietary stack traces.
2. Score closed API, self-host open weights (no fine-tune), and LoRA on open base across the drivers above (table with traffic lights or numeric scores).
3. End with a recommendation and the biggest risk if you are wrong (cost overrun, data leak, quality miss, ops burnout).
4. Include one sentence on license risk for your chosen open-weight option (name a real model family and license type, even hypothetically).

Checklist

• Matrix with explicit scores or traffic lights
• License risk called out in one sentence
• Recommendation states biggest risk if wrong