Arguing the Cascade Against a Single-Model Proposal
A teammate on your Generative Fill team proposes simplifying the architecture: "We already have a solid mask-conditioned diffusion inpainting model for the generate-with-prompt path. Let's just use it for object removal too — pass an empty prompt when the user wants to remove something. One model, one service, less to maintain."
- Using the latency, quality, and diversity axes from the
GAN-vs-diffusion comparison table (
generative-adversarial-networks-and-face-generation), explain what this proposal gets wrong. - Removal requests are the majority of real Generative-Fill traffic. Quantify, at a back-of-envelope level, why this matters for the proposal's cost and latency impact, stating your assumptions.
- Is there any scenario in this product where routing a removal request through the diffusion path instead of the GAN path is actually the right call? If so, describe it; if not, explain why not.
1. What the proposal gets wrong
Removal is a narrow, well-defined task — make existing content disappear and extend the surrounding texture plausibly — with essentially one correct-enough answer per input, not an open-ended creative one. On latency, diffusion is inherently iterative (even with aggressive step reduction, still many sequential network evaluations) while a single-pass GAN inpainter costs the same as any ordinary inference call — for a task performed on every eraser-brush drag, that gap compounds across volume. On quality, diffusion's advantage is specifically its higher ceiling on broad, open-ended content; removal doesn't need that ceiling, since the target content (whatever should logically be under the mask, given the visible surroundings) is comparatively constrained. On diversity, this is the axis where GANs are structurally weakest and diffusion strongest — but removal is exactly the case where that weakness doesn't matter, because the product doesn't want ten different plausible removals, it wants one good one, fast. Passing an empty prompt doesn't change any of this; it just runs the more expensive, slower model on a task that doesn't need what makes it expensive and slow.
2. Back-of-envelope cost and latency impact
Stating illustrative assumptions: if diffusion inference costs on the order of a few GPU-seconds per batched request (even at reduced steps) versus a GAN inpainter's sub-200 ms on cheap commodity hardware or on-device, and removal requests are the majority of traffic — say, illustratively, 60-70% of all Generative Fill requests — then routing all of that majority-share traffic onto the GPU-bound diffusion path multiplies the cloud GPU fleet's required capacity by roughly the same factor, for a task that previously cost close to zero marginal cloud spend when served by an on-device or cheap-hardware GAN. The latency impact is separate but compounding: every removal request now also waits on a multi-step sampling process instead of returning near-instantly, directly hurting the interactive feel of the single most common action in the product.
3. Is there ever a case for routing removal through diffusion?
Yes, one plausible case: a removal request where the surrounding context genuinely can't provide enough information for a texture-completion model to produce a plausible fill — for example, removing a large foreground object that occupies most of a complex, structured scene (not a simple background), where "extend the surrounding texture" isn't well-defined because there's no clearly dominant surrounding texture to extend. In that narrow case, a diffusion model's stronger scene-level generative capability could produce a more plausible fill than a texture-completion-oriented GAN. This is the exception, not a reason to abandon the cascade generally — a well-designed intent router could detect "large, structurally complex mask with low-confidence GAN output" and escalate to diffusion as a fallback, rather than routing all removal traffic there by default.
Share this question