Kafka for evented AI

What Kafka is (in an LLM product)

Apache Kafka is a distributed commit log: producers append events to topics; consumer groups read and process them independently. For AI systems it is the backbone for async RAG ingest, embedding backfills, eval sampling, and agent side-effects that must not die with a request timeout.

Cousins (Pulsar, Kinesis, Redpanda, NATS JetStream) share the mental model: durable ordered streams + fan-out. Learn Kafka once; the product patterns transfer.

flowchart LR
  Upload[Doc uploaded] --> T1[topic: docs.raw]
  T1 --> Parse[Parse workers]
  Parse --> T2[topic: docs.chunked]
  T2 --> Emb[Embed workers]
  Emb --> T3[topic: docs.embedded]
  T3 --> Idx[Indexer → vector DB]
  T3 --> Eval[Eval / sampling sink]

Interview cue: Draw ingest as a pipeline of topics with idempotent upserts. “Kafka” is the bus; the hard part is keys, versions, and replay.

The engineering problem

Synchronous “upload PDF → embed → answer” couples UX to GPU/CPU work and fails half-way with no replay. Event logs let you:

Kafka is ops weight. Earn it with volume, multi-consumer fan-out, or replay requirements — not day-one prototypes.

Architecture mental model

flowchart TB
  Prod[API / uploader] --> Broker[Kafka brokers]
  Broker --> G1[group: parsers]
  Broker --> G2[group: embedders]
  Broker --> G3[group: indexers]
  G2 --> GPU[Embed GPUs]
  G3 --> VDB[(Vector DB)]

Ship rule: partition by the entity you need ordered (tenant_id or doc_id). Random keys destroy per-doc ordering and make upsert races painful.

Ordering and hot partitions

Often: partition by tenant_id, include doc_id in the payload, and make indexer upserts idempotent so rare reorders are safe.

Topic design for RAG ingest

Keep payloads small: store blobs in S3/GCS; put refs + hashes on the wire. Large message bodies blow heap and kill rebalance latency. Pair with chunking and metadata and vector databases.

sequenceDiagram
  participant API as Upload API
  participant S3 as Object store
  participant K as Kafka
  participant P as Parser
  participant E as Embedder
  participant V as Vector DB
  API->>S3: Put blob
  API->>K: docs.raw ref
  K->>P: consume
  P->>K: docs.chunked
  K->>E: consume
  E->>K: docs.embedded
  K->>V: upsert vectors

Exactly-once is a lie — design for idempotency

Workers will reprocess (rebalance, crash, at-least-once delivery). Production patterns:

1. Upsert by stable chunk id into the vector store
2. Store processing version (embed model + chunker hash) with each vector
3. Commit offsets after successful side effect (or use transactional outbox)
4. Route poison messages to docs.dead — never infinite retry on bad PDFs

# Shape — idempotent index write
def handle_embedded(event: dict) -> None:
    chunk_id = event["chunk_id"]
    version = event["pipeline_version"]  # embedder@v3+chunker@v2
    upsert_vector(
        id=f"{chunk_id}:{version}",
        vector=event["vector"],
        metadata={**event["meta"], "pipeline_version": version},
    )
    # commit offset only after upsert succeeds

Optional: tombstone or delete old chunk_id:* versions after a successful migrate so the index does not grow forever.

How it fits agent products

sequenceDiagram
  participant U as User
  participant A as Agent API
  participant DB as App DB
  participant K as Kafka
  participant W as Side-effect worker
  U->>A: "Create Zendesk ticket"
  A->>DB: Write agent state + outbox row (same txn)
  A-->>U: Accepted
  A->>K: Publish outbox event
  K->>W: ticket.create
  W->>W: Call Zendesk (idempotent key)

When an agent “creates a ticket,” write an outbox event in the same DB transaction as agent state, then publish. Consumers perform the side effect. This avoids “LLM said success but API failed” half-states — pair with agents and ReAct and multi-agent orchestration.

Long-running multi-step jobs also fit: emit agent.task.requested → workers → agent.task.completed with budgets and OpenTelemetry trace_id on every message.

Backpressure and SLOs

Embedding spikes overwhelm GPUs and vector DBs. Operate on:

flowchart LR
  Lag[Consumer lag high] --> Scale[Scale embedders]
  Lag --> Shed[Pause batch topics]
  DLQ[DLQ growth] --> Halt[Halt + page humans]
  VDB[VDB write p95] --> Slow[Reduce batch size]

Interactive chat should not share the same GPU pool as nightly backfills without isolation (Ray or separate queues).

Schema evolution

Prefer JSON + explicit version for early products; Avro/Protobuf + registry when many teams share topics.

Failure modes

• Schema drift — version payloads; reject unknown fields loudly
• Keying mistakes — hot partitions on one tenant starve others
• Offset commit too early — lost embeds after crash
• Giant messages — blob-in-topic anti-pattern
• No DLQ — poison PDF loops forever
• ACL / PII on the wire — encrypt refs; never put raw regulated text in clear topics without policy
• Lag blindness — alert on lag and DLQ, not only CPU
• Missing trace propagation — cannot debug a doc across stages

Production checklist

1. Topic map + partition key + retention documented.
2. Idempotent upserts with pipeline_version.
3. DLQ + runbook for poison documents.
4. Separate consumer groups for interactive vs batch.
5. Lag + DLQ alerts before GPU util alerts.
6. Propagate trace_id / tenant_id on every event.
7. Replay drill: reset offsets for one partition in staging quarterly.

Alternatives

Micro-project

1. Design topics for doc-ingest → embed → index (table above).
2. Specify partition key + idempotency key for the indexer.
3. Add an outbox sketch for one irreversible agent tool.
4. Compare to a Ray Data backfill for the same corpus — three bullets each.

Related

Guided Agentic workflows & multi-agent and RAG ingest. Data: vector databases, chunking and metadata. Advanced Key Tech: Ray, OpenTelemetry for LLMs.