Paths Subjects Questions Quizzes Pricing Search
Advanced Open Pro

Designing a Trace Schema for a Tool-Using Agent

You're building observability for an agentic support assistant that rewrites the user's query, retrieves from a help centre, calls tools (account lookup, refund), and streams an answer with citations. Support keeps escalating tickets where they can't explain why the bot called (or didn't call) a tool, or why an answer cited the wrong thing, because all you currently log is one line: timestamp, user id, final answer, latency.

  1. Design the trace schema as a tree of spans. What does each span type need to capture, at minimum, to make "why did it do that" answerable after the fact?
  2. What would you explicitly exclude or redact, and why?
  3. Give two concrete support tickets this schema would let you resolve in minutes that the current one-line log cannot resolve at all.
  4. How does sampling/retention policy differ between recent traffic and long-term storage, and why not just keep everything forever at full fidelity?

Share this question

← Back to LLM Observability and Evaluation practice

We use cookies for product analytics to improve OmniAtlas. See our Privacy Policy.