Case Study: Generative Fill — Inpainting, Outpainting and Object Removal

"Design Photoshop-style Generative Fill: the user selects a region and either removes what's there, extends the canvas, or types what should appear" is the interview prompt behind one of the most visible generative-AI product launches of the last few years — Adobe added Generative Fill to Photoshop in 2023, Canva shipped comparable Magic-branded editing tools, and every marketplace and ad platform that lets sellers touch up product photos has a version of the same button. The business problem is two-sided: for creative tools (Adobe, Canva), Generative Fill is a retention and differentiation feature — the reason a subscriber renews instead of switching to a competitor with a worse editor; for marketplaces and ad platforms, it is operational leverage — a seller who used to pay a freelancer $10–20 and wait a day to remove a background distraction or extend a product photo to a new aspect ratio can now do it themselves in seconds, which means more listings get the visual polish that drives click-through and conversion. Both sides pay for the same underlying capability, and both sides are unforgiving about the two things that make this case study different from a generic "design a text-to-image system" prompt: the edit has to be interactive (this is a tool a person is actively using, not a batch job), and the edit has to be invisible — the whole point is that nobody looking at the result should be able to tell where the mask was.

This subject follows the seven-step case-study framework from ml-system-design-framework, the same shape case-study-fraud-detection uses. It leans on text-to-image-diffusion-models for diffusion vocabulary — classifier-free guidance and latent diffusion are cross-linked there rather than re-derived here — and on generative-adversarial-networks-and-face-generation for the GAN-vs-diffusion comparison table and failure-mode vocabulary, since this case is the first business case in the track where the honest answer to "GAN or diffusion?" is genuinely "both, for different parts of the same product."