Case Study: Synthetic Training Data for Computer Vision

"Our detector fails on rare cases — night rain, occluded pedestrians, a new product on the shelf. Design the system that generates the training data to fix it" is a staple senior interview prompt at any company whose product depends on a computer-vision model that has to work in the long tail: autonomous-driving perception stacks, retail shelf-detection and planogram-compliance systems, and medical-imaging classifiers all share the same structural problem. The failure cases that matter most — the ones that cause a collision, a missed stockout, a missed tumor — are by definition the rarest ones in any real-world data collection process, and rarity compounds with cost: collecting a genuinely representative sample of "pedestrian partially occluded by a parked delivery van, at night, in rain" means driving fleets for months and hoping the scenario recurs, or it means putting a vehicle into a scenario deliberately to collect it, which for some scenarios is unsafe or illegal to do on public roads at all. Waiting for the real world to hand you enough labelled examples of the case that is currently hurting you is not a data strategy; it is a bet that the failure won't recur before you happen to collect enough of it.

This case study is deliberately positioned as this track's internal flywheel pattern, and that label is worth stating up front because it is the single fact that changes the shape of every step that follows. case-study-ai-product-photography-at-scale is batch-offline but still user-facing — a seller uploads a photo and gets a finished listing image back. case-study-virtual-try-on and case-study-generative-fill-inpainting-and-outpainting are interactive, sub-five-second, user-facing surfaces. case-study-image-super-resolution-and-restoration is an edge-vs-cloud split, again serving a human who is looking at the output. None of that applies here. Nobody ever looks at the images this system produces as a product; the only consumer of the generated pixels is a training job, and the only thing anyone downstream ever evaluates is whether a different model — the production detector — got better on real data it has never seen. That inversion is why the ROI story, the evaluation suite, and the system design all look different from every other case in this block, and it is worth naming explicitly and early rather than letting an interviewer discover it three steps in.

This subject follows the same seven-step case-study framework case-study-fraud-detection uses. It leans on text-to-image-diffusion-models for diffusion and conditioning vocabulary — classifier-free guidance and layout/mask conditioning are cross-linked there rather than re-derived here — and on generative-adversarial-networks-and-face-generation for the GAN-vs-diffusion comparison table this case's Step 3 argument is built directly on top of, and for the adversarial-loss and mode-collapse vocabulary that CycleGAN-style unpaired translation, this case's canonical GAN use case, is grounded in. Aim to deliver the full answer in about 40 minutes, then use the follow-up questions to pressure-test it.