On-device / small models

On-device is a product constraint, not a vibe

Running small language models locally (phone, laptop, edge box) trades model capability for privacy, offline use, marginal cost near zero, and predictable latency without network round-trips. Product teams choose on-device when data cannot leave device, connectivity is poor, or cloud cost at volume dominates — not because 8B models beat frontier APIs on open-ended reasoning.

Connect to fine-tune module: your SLM is candidate for on-device deployment via Ollama, llama.cpp, MLX, or mobile runtimes.

Pick a constrained task

Narrow tasks where SLMs win:

• Intent classification / routing
• PII scrubbing suggestions
• Format conversion (JSON fix, regex-like extraction)
• Keyboard-style autocomplete with style guide

Avoid: multi-hop research, rare knowledge, long context synthesis.

Document task in ondevice/task_spec.md with success metric same as cloud baseline eval.

Hardware limits matrix

Record your test machine:

Resource	Your device	Implication
RAM	e.g. 16 GB	max quant size
GPU	Apple M2 / none	tokens/sec
Thermal	laptop sustained	throttle after N min
Battery	mobile target	inference budget

Measure tokens/sec, peak RAM, cold start (model load time).

Callout — Quantization tradeoffs: Q4_K_M saves RAM; watch quality drop on your task — re-run eval module comparison.

Quality vs cloud baseline

Run same golden subset (≥20 cases) on:

• Cloud API (frontier or mid)
• Local SLM (your Build & serve your SLM model or public small model)

Report pass rate, p95 latency, cost (cloud $ vs $0 local amortized hardware if ambitious).

Honest conclusion: "local wins on X%; cloud required for Y%."

Deployment sketch

Even for portfolio:

• Model artifact path + quant method
• Runtime command (ollama run, llama-server)
• Update strategy — model blob versioning
• Fallback to cloud when local confidence low (hybrid router)

Battery and thermal on laptops

Sustained local inference drains battery — document watts or % battery per 100 requests if mobile/laptop target. Product implication: on-device only when plugged in or user opts in.

Model update UX

App must download new quant blobs — sketch update flow with version check and rollback if new model fails eval on device smoke tests.

Federated learning (literacy only)

On-device sometimes pairs with federated ideas — one paragraph in hardware_limits.md distinguishing inference-on-device (this lesson) from training-on-device (research, privacy-sensitive) — avoid conflating.

Putting it together in practice

ShipAI treats this lesson as executable curriculum, not reading alone. Before marking complete, trace one real request through your portfolio stack and label where this lesson's concepts apply — even if the first pass is messy. Document what broke in the module README; that gap list becomes your next sprint.

Compare your implementation against the industry callouts cited earlier without copying their scale. Name one deliberate simplification you kept (mock auth, SQLite not Postgres, single-region deploy) and one simplification you refuse to ship without (no eval gate, no trace on mutating tools, no fail-closed guardrail on exfil cases). That contrast is what interviewers and graders look for.

Callout — Teach back: Explain this lesson's core tradeoff to a peer in five minutes without slides. If you cannot, re-read the failure modes section and add an example from your own run logs.

Common questions and misconceptions

"Is this overkill for a side project?" Side projects can skip pieces; capstones and production cannot skip knowing the pieces exist. You may waive cost accounting in v1 but your architecture diagram should still show where it would attach.

"Should I rewrite from scratch?" Extend what you built in prior modules — graders reward evolution, not parallel unused folders. Link file paths in your checklist.

"Which metric matters most?" The metric tied to user harm or revenue: policy violations, failed refunds, silent wrong answers — not vanity leaderboard scores.

Extension paths after the micro-project

After the micro-project passes smoke check, choose one extension aligned with your capstone pillar: tighten eval coverage, add a chaos or red-team case, or wire observability into SSE streams. Extensions belong in BACKLOG unless scope freeze explicitly includes them — avoids capstone death by optional polish.

Engineering problem (staff framing)

On-device SLMs win privacy/latency offline; lose capability/ops uniformity.

Diagram — On-device path

flowchart LR
  App --> Local[NPU/GPU local] --> Fallback[Cloud fallback]

Precise definitions & mental model

Quantization, NPU runtimes, sync of weights.

Tradeoffs — when to use what

Capability vs privacy/cost.

Failure modes (interview + on-call)

Battery blowups; silent cloud fallback leaking data.

Production & OSS practices

Explicit consent for cloud; versioned on-device packs.

Micro-project: Constrained local task

Ship:

1. Local inference script for narrow task.
2. Benchmark vs cloud on shared eval subset.
3. hardware_limits.md with measurements.
4. Hybrid fallback sketch if local confidence < threshold.
5. Capstone implication: on-device pillar yes/no with evidence.

Acceptance: numbers in report, not vibes; local run reproducible from README.

Checklist

• Narrow task spec with metric
• Local model runs on documented hardware
• Cloud vs local eval comparison table
• tokens/sec and RAM measured
• Hybrid fallback documented if applicable