Generative AI landscape

Generative ≠ one architecture

"Generative AI" is a product category, not a single model class. It describes systems that produce new content — tokens, pixels, waveforms, video frames, structured objects, or actions — rather than only classifying inputs. Under that umbrella sit radically different architectures, training objectives, and serving costs.

As an AI engineer, your job is often systems integration: retrieval, tools, evals, caching, routing, guardrails, and observability — with a generative model as one component (sometimes the smallest part by line count, rarely by dollar cost).

Confusing modalities causes expensive mistakes: proposing a text LLM for real-time video synthesis, expecting diffusion latency to match chat APIs, or ignoring ASR error propagation in a voice agent. This lesson gives you a high-level map so you can scope projects and read vendor marketing critically.

Modalities and typical model families

Text and code

Core pattern: autoregressive decoder-only transformers (GPT-class, Llama-class, Claude-class APIs). Training: next-token prediction on large corpora; post-training for instruction following.

Products: chat assistants, email drafting, summarization, code completion, SQL generation.

Systems extras: RAG for freshness and domain docs; tool calling for calculators and APIs; prompt caching for cost; eval harnesses for factuality.

Code is text with syntax constraints — often the same LM with different fine-tuning and IDE integration, plus static analysis in the loop.

Images (and image editing)

Core patterns:

• Diffusion models (Stable Diffusion, DALL·E class, Midjourney-class) — iterative denoising from latent noise; strong for creative generation.
• Autoregressive or hybrid approaches exist but diffusion dominates public tooling circa 2024–2026.

Systems extras: text encoders (CLIP-like) for prompts; ControlNet / IP-Adapter for structure; safety classifiers on prompts and outputs; GPU-heavy serving with queueing.

Audio: speech-in, speech-out

ASR (speech-to-text): Whisper-class models, streaming variants for low latency.

TTS (text-to-speech): neural vocoders, transformer TTS, sometimes LM-backed prosody control.

Music / SFX: separate generative stacks (not the same as chat LMs).

Systems extras: diarization, VAD (voice activity detection), latency budgets for real-time conversation, accent and noise robustness testing.

Video

Often compositions: diffusion or autoregressive frame models + temporal consistency modules; increasingly multimodal LMs for scripting/storyboarding. Serving cost and latency are extreme relative to text — many "video AI" products are batch/async, not chat-speed.

Multimodal LMs

Single model (or tightly coupled stack) consuming images, audio, or video frames and producing text or actions — GPT-4o-class, Gemini-class, open multimodal checkpoints.

Use cases: document QA over screenshots, visual customer support, chart reading.

Failure modes: hallucinated details in images, OCR errors on dense slides, jailbreaks via adversarial pixels.

System patterns vs. "just an LLM"

Pattern	What it adds	Example product shape
Raw chat LM	Single model call	Simple Q&A bot
RAG	Retriever + reranker + LM	Enterprise doc assistant
Tool/agent	LM plans; runtime executes tools	Booking assistant with API calls
Cascade	Small model routes; big model answers hard queries	Cost-optimized support
Generative + verifier	LM proposes; rules/model check	Compliance-sensitive drafts
Diffusion pipeline	Prompt encoder + UNet + VAE	Image generation SaaS

When you classify products in the micro-project, at least two rows should name agent/tooling or RAG, not stop at "it's GPT."

Callout — modality mismatch in interviews: If asked to design a voice support bot, mention ASR → LM → TTS latency chain and where errors compound. Text-only answers signal gap.

Cost and latency intuition (rough)

Text LLM APIs: milliseconds to seconds per request, priced per token.

Image diffusion: seconds to tens of seconds per image, GPU-bound.

Video: often minutes or heavy prepaid GPU.

Voice real-time: sub-500ms targets for natural turn-taking — constrains model size and forces streaming.

These constraints drive product architecture more than benchmark leaderboard ranks.

Scoping projects without modality confusion

When you propose a feature, ask four questions before picking a model:

1. Output modality — text, image, audio, video, structured JSON?
2. Latency class — interactive chat, batch overnight, or streaming partial tokens?
3. Grounding need — static model knowledge enough, or must retrieve private docs?
4. Verification — can a human review, or must outputs be machine-validated?

A "generate onboarding emails from CRM fields" project is text LM + templates + schema validation. A "generate product photos from SKUs" project is diffusion + brand style LoRA + content moderation. Mixing them in one sprint usually means neither ships well.

Structured generation counts as generative

JSON tool calls, SQL, API request bodies, and UI component trees are generated artifacts even when they are not prose. Agent modules treat structured output as first-class — with parsers, retries, and golden fixtures exactly like the setup module taught.

Callout — read vendor diagrams skeptically: Marketing slides flatten stacks to one "AI" box. Your product table should unpack retriever, reranker, LM, safety filter, and post-processor when present.

Engineering problem (staff framing)

GenAI is a layered system (UX→orch→RAG/tools→model→infra→evals). Wrong layer bets waste quarters.

Diagram — Stack layers

flowchart TB
  UX[Product UX] --> Orch[Orchestration]
  Orch --> RAG[RAG/tools/memory]
  RAG --> FM[Foundation model]
  FM --> Infra[Serving/GPU]
  Orch --> Eval[Evals/guardrails]
  Eval --> UX

Precise definitions & mental model

Foundation models, orchestration, differentiation via data+eval+workflow.

Tradeoffs — when to use what

Bet	Speed	Moat
Closed API	Fast	Weak
Open weights	Medium	Ops+custom
Train scratch	Slow	Rarely justified

Failure modes (interview + on-call)

Thin wrappers; multi-agent before single-tool works; late modality.

Production & OSS practices

Swappable model interfaces; unit economics per successful task.

Deep dive (FAANG / OSS bar)

Push «generative-ai-landscape» past tutorial depth: write the interface contract (inputs/outputs/invariants), list three measurable metrics, and name two degrade modes if the happy path fails. Add a short threat note: what an attacker or noisy tool result could do, and which layer catches it (schema, policy, HITL, or eval gate).

flowchart LR
  Contract[Interface contract] --> Metrics
  Metrics --> Degrade[Degrade modes]
  Degrade --> Threat[Threat + control]

Micro-project: Map 10 products → model type

Create m1/landscape/products.md with a table:

Product	Modality	Likely core model pattern	System extras (RAG/tools/etc.)	Confidence

Fill 10 rows covering diverse examples:

• At least one ChatGPT-class assistant.
• At least one Midjourney / DALL·E / Stable Diffusion-class image tool.
• At least one coding assistant (Copilot, Cursor-class).
• At least one voice bot or meeting transcription product.
• Others: video gen, music, search+gen hybrid, enterprise RAG copilot, etc.

Be honest when guessing from public info — mark high / medium / low confidence. Add one paragraph at the bottom: which row would you most likely mis-scope without this modality map, and why?

Checklist

• 10 rows filled in m1/landscape/products.md
• At least 3 different modalities represented
• At least 2 rows call out agent/tooling or RAG, not just "an LLM"