From Hackathon Framework to Production Architecture
A startup built its first customer-facing AI feature in a two-week sprint using a general-purpose orchestration framework (chains/graphs of LLM calls, retrievers and tools) because it had the fastest path to a demo. The demo went well, the feature is now live, and traffic is growing. The on-call engineer just spent six hours debugging a latency spike and couldn't easily find the literal prompt and token count sent to the model for the slow requests — everything was buried inside the framework's chain internals.
- Explain why this outcome was predictable from the framework's core abstraction, not just bad luck.
- Would you recommend ripping the framework out? Justify your answer.
- Propose a concrete plan for what changes now, in priority order.
1. Why this was predictable
The framework's core value is composability — expressing an LLM application as a pipeline of chained steps with a large library of pre-built integrations. That composability requires layers: a chain wraps a prompt template, which wraps a model client, which wraps the HTTP call. Each layer is a place where the literal request sent to the model gets one step further from what's visible without deliberately tracing it. This is the standard trade-off, not a defect specific to this team's usage — "fast to build, harder to introspect" is exactly what a general-purpose orchestration abstraction buys you, and the six-hour debugging session is the bill coming due on the opacity side of that trade.
2. Rip it out?
Not necessarily, and probably not first. Ripping out a working, revenue-generating feature to satisfy a debugging complaint is a large, risky change for a problem that has a smaller fix: add observability at the framework boundary. The framework's ecosystem and velocity are still real advantages for whatever this team builds next; throwing them away because one incident was hard to debug over-corrects. A full migration to a hand-rolled orchestration layer becomes justified only if this keeps happening after instrumentation is added, or if the team hits a wall on custom retry/guardrail logic the framework's assumptions fight against.
3. Concrete plan, in order
- Instrument the boundary now: log the literal rendered prompt, token counts, latency and model/version per step (not just per chain run) into the trace store, so the next latency spike is a dashboard query, not a six-hour archaeology dig.
- Add a token/cost budget and alerting per request so a spike is caught before it becomes a six-hour incident.
- Identify the specific step(s) causing latency (usually retrieval, a slow tool call, or an unbounded generation) using the new tracing, and fix that step directly — this is very likely a model-routing, prompt-length, or retrieval-tuning problem, not a framework problem.
- Re-evaluate only the parts that keep causing pain. If a specific chain keeps being the source of incidents, replace that chain with hand-written code calling the raw API directly, while leaving the rest of the framework in place where it's still earning its keep (e.g., its vector-store integrations). This progressive replacement — not a rewrite — is the migration path senior teams actually use.
Share this question