Multi-tenant AI SaaS: isolation, quotas, and noisy neighbors

The isolation bug class

AI SaaS products inherit classic multi-tenant risks plus new ones: shared KV/prefix caches, shared vector indexes, and prompt logs that contain customer secrets. Pattern inspired by public B2B AI platform practice: assume every shared layer can leak until proven partitioned.

Isolation dimensions

flowchart TD
  Req[Request + tenant_id] --> Auth[Authn / Authz]
  Auth --> Quota[RPM / TPM / $ caps]
  Quota --> Data[Data plane isolation]
  Data --> Vec[Vector / files ACL]
  Data --> Cache[Cache key = tenant + hash]
  Data --> Model[Model route]
  Model --> Out[Response + audit]

Layer	Must isolate	Common miss
Auth	Tenant + role	Tool calls without re-check
Storage	Docs, threads, embeddings	Global collection “for speed”
Cache	Exact/semantic/KV prefix	Shared system-prompt cache across tenants
Models	Optional dedicated deploys	One noisy tenant starves GPU
Logs	PII redaction per policy	Full prompts in third-party APM

Rate limits that match LLM economics

Classic RPM isn’t enough. Prefer:

• TPM (tokens per minute)
• Concurrent generations
• $ / day ceilings
• Burst tokens with smoother

sequenceDiagram
  participant T as Tenant
  participant G as Gateway
  participant Q as Quota service
  T->>G: chat
  G->>Q: admit(tpm, concurrency)
  alt rejected
    Q-->>G: 429 + retry-after
    G-->>T: 429
  else admitted
    Q-->>G: ok
    G->>G: generate
  end

Return actionable Retry-After and remaining budget headers for well-behaved clients.

Noisy neighbor on shared GPUs

If you self-host:

• Per-tenant concurrency caps at the gateway
• Separate queues for interactive vs batch
• Optional dedicated replicas for enterprise SKUs
• Fair scheduling / weighted queues

If you use vendor APIs: still enforce your quotas — vendors’ limits won’t match your pricing tiers.

Vector and RAG ACL

Retrieval must filter by tenant (and finer ACL) before context packing. Post-hoc “please don’t cite other customers” in the prompt is not a control.

What to ship

1. tenant_id on every span, cache key, and DB row
2. Quota middleware with TPM + concurrency
3. Per-tenant encryption keys or at least logical separation with tests
4. Abuse playbooks (key leak → rotate + revoke)
5. Contract tests: tenant B query never returns tenant A chunks

Failure modes

• Semantic cache hit across tenants on similar HR policies
• Fine-tuned adapter trained on mixed-tenant data
• Background jobs using a god-mode service account into all indexes
• Cost attribution missing → can’t bill or cap

Design review questions

• Where do cache keys include tenant?
• What happens when tenant hits 100% of GPU concurrency?
• Can a compromised API key read another tenant’s traces?

Multi-tenant AI is distributed systems + privacy engineering — the model is the easy part.