Serve your SLM

Owning weights includes owning a serve path

A LoRA adapter in a folder is not a product. Serving turns base + adapter into an HTTP API your applications call — ideally OpenAI-compatible so code from the chat APIs lesson swaps endpoints with one config change.

Options for this course:

Pick one matching your hardware; document tradeoffs in README.

Callout — same client, different base_url: OpenAI(base_url="http://localhost:11434/v1", api_key="ollama") — verify with one script from m4.

Ollama path (developer default)

Create Modelfile:

FROM llama3.2:3b
ADAPTER ./m6/adapters/ticket_classifier_v1
SYSTEM Classify support tickets into billing, shipping, account, bug, other. Reply label only.

ollama create ticket-classifier -f Modelfile
ollama serve
curl http://localhost:11434/v1/chat/completions -d '{...}'

Convert HF adapter to GGUF if stack requires — follow tool docs for your base.

vLLM path (GPU throughput)

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.2-3B-Instruct \
  --enable-lora --lora-modules ticket=m6/adapters/ticket_classifier_v1

Hit http://localhost:8000/v1/chat/completions with model: ticket.

Tune --max-model-len to match training context.

Integration with existing prompt pack client

Refactor client from lesson 4.1:

def get_client():
    if os.getenv("USE_LOCAL_SLM"):
        return OpenAI(base_url=os.environ["SLM_BASE_URL"], api_key="local")
    return OpenAI()

Run same cases.jsonl through local SLM; compare outputs to teacher in eval harness.

Operational basics

Health: GET /health or lightweight completion on startup probe.

Concurrency: Ollama serializes some paths; vLLM batches — load-test if multiple users.

Model load time: Cold start seconds to minutes — note in OPS.md.

Versioning: Tag served artifact ticket_classifier_v1 aligned with adapter folder and report card.

Rollback: Keep previous Modelfile / LoRA module registered as ticket_classifier_v0.

Security on localhost

Even local servers bind 0.0.0.0 in some configs — firewall or bind 127.0.0.1 only. No auth on default Ollama; do not expose raw to internet.

Logging

Log request id, model version, latency, token counts. Redact user content if logs leave machine.

Callout — template at serve time: Ensure server applies chat template identical to training — some servers need explicit --chat-template flags.

Docker packaging for serve

Wrap Ollama or vLLM in Docker for reproducible demos:

# conceptual — pin image versions
FROM ollama/ollama:latest
COPY m6/adapters/ticket_classifier_v1 /adapters/
COPY m6/serve/Modelfile /root/Modelfile

Document GPU passthrough (--gpus all) vs CPU-only fallback. Course reviewers on Apple Silicon may use Ollama native without Docker — note both paths in SERVE.md.

Healthcheck in compose:

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"]
  interval: 10s

Load testing basics

Even a simple loop helps:

for i in $(seq 1 50); do curl -s -o /dev/null -w "%{time_total}\n" ... & done; wait

Record p50/p95 latency at concurrency 1 vs 4. Ollama may queue; vLLM batches — choose stack matching expected concurrency.

Adapter hot-swap

Run multiple LoRA adapters on one base in vLLM:

--lora-modules ticket=m6/adapters/ticket_v1,summarize=m6/adapters/summarize_v1

Client passes model name per request — useful when portfolio grows multiple narrow tasks without loading full bases repeatedly.

Document memory footprint: one base + two adapters vs two merged models — merged simplifies ops, adapters simplify experimentation.

Troubleshooting serve mismatches

When local outputs diverge from training notebook:

Keep a smoke_expected.jsonl with 3 inputs and gold labels — run after every serve config change.

Environment variables reference

Document in SERVE.md:

Consistent env naming lets m4 scripts run unchanged against local or cloud endpoints.

llama.cpp and Apple Silicon

On Mac, many students serve GGUF exports:

./llama-server -m ticket-classifier-Q4_K_M.gguf --port 8080

Convert HF adapter merge to GGUF via llama.cpp conversion scripts or Ollama import — document exact command chain in SERVE.md. Metal acceleration improves token/sec; still log p95 for batch=1 interactive calls.

CPU-only Linux fallback uses same GGUF with -ngl 0 — slower but valuable when no GPU available.

Streaming responses

OpenAI-compatible servers support stream: true — wire streaming in smoke client optional stretch. Streaming improves perceived latency for chat UIs; classification tasks often skip streaming for simpler parsing and easier golden-test assertions.

Engineering problem (staff framing)

Serving is the product: batching, KV-cache, OpenAI-compatible API, health checks.

Diagram — Serve path

flowchart LR
  Client --> Gateway --> Engine[vLLM/TGI/llama.cpp]
  Engine --> GPU
  Gateway --> Met[Metrics]

Precise definitions & mental model

Continuous batching, p50/p95 TTFT/TPOT, autoscaling.

Tradeoffs — when to use what

Throughput vs latency SLOs; quant quality.

Failure modes (interview + on-call)

OOM at context max; no queue limits; noisy neighbor GPUs.

Production & OSS practices

Load tests; circuit breakers; model warm pools.

Micro-project: Local OpenAI-compatible API

In m6/serve/:

1. Serve your adapter via Ollama, vLLM, or llama.cpp with OpenAI-compatible route.
2. smoke.py — reuse m4 client patterns against local endpoint for 3 cases.
3. SERVE.md — start commands, ports, hardware, cold start notes.
4. Optional: Dockerfile wrapping serve stack for reproducibility.

Checklist

• OpenAI client works with only base_url change
• Model version documented
• Smoke script passes
• Serve docs include stop, start, and rollback