Intermediate
Open
Pro
Choosing an Architecture Shape for a New Product Feature
Your company is building a document-translation feature: users upload a document (up to ~20k tokens) in one language and get back a full, faithful translation in another. A teammate proposes using your existing decoder-only chat LLM (the one already deployed for the product's chat assistant) with a "translate the following document:" prompt, since "it's already a great LLM and reusing it means no new infra."
- Is that a reasonable default, or does the task's shape argue for something else? Walk through the reasoning, not just the conclusion.
- If you did consider an encoder-decoder model for this feature, what would you actually be trading off against the "just prompt the LLM you already have" approach?
- What would make you say "no, decoder-only prompting is clearly the right call here, don't overthink it"?
Share this question