Intermediate
Open
Pro
Parallel Tool Calls and a Latency Budget
An agent turn needs three independent lookups before it can answer: the user's account plan, their last three orders, and current shipping-carrier status. Each tool call has a 300 ms round-trip, and each model turn (deciding what to call next) takes about 500 ms.
- Compute the approximate wall-clock time for this turn if the calls are made sequentially (one tool call per model turn) versus if the model requests all three in parallel in one turn.
- What has to be true about these three tools for the parallel version to be safe to build?
- A fourth lookup is added: "reserve the last item in stock for this order" immediately followed by "charge the customer's card." Should these be included in the same parallel batch as the three reads? Justify your answer.
Share this question