Ray

What Ray is

Ray is a Python-native distributed runtime for scaling AI workloads across a cluster: tasks/actors, data-parallel maps, multi-worker training, hyperparam search, and model serving.

When one machine is not enough — fine-tunes, embedding backfills, offline evals, multi-model serving — platforms reach for Ray (Anyscale, Databricks-adjacent stacks, many internal ML platforms).

flowchart TB
  Job[Driver / job] --> Train[Ray Train]
  Job --> Data[Ray Data]
  Job --> Serve[Ray Serve]
  Job --> Tune[Ray Tune]
  Train --> Workers[GPU workers]
  Data --> CPU[CPU/GPU map batches]
  Serve --> Replicas[HTTP replicas]

Interview cue: Ray is orchestration + distribution, not a decode engine. Often Ray schedules work; vLLM or TensorRT-LLM burns tokens.

The engineering problem

Without a cluster runtime you either: (1) hand-roll Kubernetes Jobs and pray for retries, or (2) keep everything on one box until the backlog is measured in days. Ray’s bet: same Python code, declared resources, object-store shuffle.

Architecture: pieces you will touch

flowchart LR
  subgraph cluster [Ray cluster]
    Head[Head node]
    W1[Worker]
    W2[Worker]
    W3[Worker]
  end
  Client[Driver script / CI] --> Head
  Head --> W1
  Head --> W2
  Head --> W3

Mental model: the driver submits work; workers execute with declared CPU/GPU resources; the object store shuffles intermediate results without forcing everything through the driver’s memory.

Tasks vs actors

Actors win for embedding / LLM wrappers: load weights once, call many times. Tasks win for pure CPU transforms where startup is cheap.

How it fits LLM apps

flowchart TB
  API[Product API] --> Serve[Ray Serve]
  Serve --> Emb[Embedding deployment]
  Serve --> Gen[vLLM deployment]
  Batch[Nightly job] --> Data[Ray Data embed backfill]
  Data --> VDB[(Vector DB)]
  TrainJob[Fine-tune job] --> Train[Ray Train]
  Train --> Registry[Artifact store / MLflow]

Pair batch paths with Kafka for evented AI when you need durable fan-out and replay; Ray Data shines when the corpus is already in object storage and you want an in-cluster map.

How to use — batch embed sketch

# Shape — embed a corpus with an actor pool
import ray

@ray.remote(num_gpus=0.25)
class Embedder:
    def __init__(self):
        self.model = load_model()  # load once per actor

    def __call__(self, batch: dict) -> dict:
        return {
            "vector": self.model.encode(batch["text"]),
            "id": batch["id"],
            "pipeline_version": "embedder@v3",
        }

# ds = ray.data.read_parquet("s3://bucket/chunks/")
# ds = ds.map_batches(Embedder, concurrency=8, batch_size=64)
# ds.write_parquet("s3://bucket/embedded/")

Write vectors to pgvector/S3; same job shape as Kafka workers, but orchestration stays in-process to the Ray cluster. Always stamp pipeline_version so re-embeds are idempotent upserts downstream (chunking and metadata).

Serve deployment graph (shape)

# Pseudocode — Serve graph in front of engines
# from ray import serve
#
# @serve.deployment(num_replicas=2, ray_actor_options={"num_gpus": 0.25})
# class Embed: ...
#
# @serve.deployment(num_replicas=1, ray_actor_options={"num_gpus": 1})
# class Generate:
#     def __init__(self, embed):
#         self.embed = embed
#         self.engine = connect_vllm()  # or HTTP to vLLM pool
#
#     async def __call__(self, request):
#         ...
#
# graph = Generate.bind(Embed.bind())
# serve.run(graph)

Keep auth, quotas, and OpenAI-shaped routing in a gateway; use Serve for the multi-model product graph.

Operational knobs that matter

flowchart LR
  Interactive[Chat / low latency] --> PoolA[Serve pool A]
  Batch[Nightly backfill] --> PoolB[Job / Data pool B]
  PoolA --> GPU1[GPU quota interactive]
  PoolB --> GPU2[GPU quota batch]

Ship rule: interactive chat and nightly embed backfills must not share one unconstrained GPU pool — isolate with separate Serve apps, queues, or cluster resource groups (cost/latency routing).

Serve vs vLLM (and friends)

Ray Serve can front vLLM replicas, do routing (small vs large), and A/B. For decode math, tune the engine — not Ray.

Failure modes

• Head node SPOF unless you design HA / managed Ray
• Version skew between workers — pin container images
• Driver as bottleneck — ray.get on huge result lists
• Object store blowups — materialize huge tables eagerly
• GPU fractional scheduling surprises — 0.25 × 5 ≠ free packing if fragmentation
• Silent actor OOMs — one bad batch kills a warm replica
• Missing trial/run ids — cannot join to OpenTelemetry or MLflow

Production checklist

1. Pin Ray + CUDA + model image digests per job.
2. Declare num_gpus / num_cpus explicitly — no “best effort.”
3. Separate interactive Serve from batch Data/Train quotas.
4. Stamp pipeline_version + git_sha on every written vector/artifact.
5. Propagate trace_id / run_id into logs and OTel.
6. Alert on job failure rate, object-store pressure, and GPU idle during backlog.
7. Rollback Serve = previous deployment graph revision, not “restart and hope.”

Alternatives

Micro-project

1. Sketch a Serve graph: embed → generate with resource annotations.
2. Run a tiny local map_batches embed on a parquet sample.
3. Compare the same backfill designed as Kafka consumers — write three bullets on when you’d pick each.
4. Guided Build & serve your SLM and Deploy, cost, latency, observability when leaving a single box.

Related

vLLM · Kafka for evented AI · Triton Inference Server · MLflow for LLMOps