Text-to-Image Generation with Diffusion Models

autoregressive-image-generation designed an unconditional generator: no prompt, just "produce a plausible image," using a VQ-VAE tokenizer and a decoder-only Transformer over visual tokens. This subject is the interview question people actually mean when they say "design DALL-E" or "design Stable Diffusion": generate an image from a text prompt, at a quality bar high enough to ship as a consumer product. It reuses that subject's vocabulary — FID, Inception Score, the super-resolution cascade — and it is the deepest subject in this track's generative-image sequence for a reason: it introduces diffusion training and CLIP/CLIPScore, which the two subjects that follow it (personalizing image generation with DreamBooth/LoRA, and text-to-video generation) reference rather than re-teach. If you only fully internalize one subject in this sequence, this is the one.

The angle interviewers reach for is almost always the same: "walk me through designing a text-to-image system." That's a deceptively large prompt, and the candidates who do well are the ones who resist diving straight into U-Net diagrams and instead work the same loop this track uses everywhere — clarify requirements, frame the ML task, prepare the data, choose and train an architecture, define sampling, evaluate, then assemble the production system.

Clarifying Requirements

Before any architecture talk, pin down the numbers that will justify every downstream decision:

• Resolution. A concrete target — 1024x1024 is a common anchor — stated explicitly, because (as in subject 1) it determines whether the model generates directly at that size or relies on a super-resolution cascade.
• Language and prompt complexity. English-only to start is a reasonable scope cut; a stated maximum prompt length (say, ~128 words) bounds the text encoder's context window.
• Data scale. Hundreds of millions of image-caption pairs — large enough that "clean everything by hand" is off the table and the data-prep pipeline has to be automated end to end.
• Latency. "Near real-time" for a consumer product usually means single-digit-to-low-double-digit seconds per image, not milliseconds — a materially looser budget than a ranking model, but still tight enough to make the sampling-step count (covered below) a real design lever, not an afterthought.
• Domain breadth. "Anything a user might describe" — landscapes, portraits, abstract art — rather than one narrow category, which is itself a reason this subject reaches for diffusion over the chunked-autoregressive approach from subject 1: broader domains reward diffusion's higher ceiling on image quality.
• Fairness and safety scope. State explicitly that the system needs bias mitigation (not skewing generated people by age, race, or gender) and content filtering (no violent, hateful, or explicit output) as first-class requirements, not a follow-up feature — this is what the inference-pipeline safety stages later in this subject exist to satisfy.