Classifying Tool Errors

When an agent calls a tool and it fails, the loop needs to know what kind of failure it was before it can react. Retrying a rate-limited call is smart; retrying a call to a tool that does not exist, or a call with malformed arguments, just burns budget on a failure that will repeat identically. This exercise is the small classifier that sits between a failed tool call and the retry logic.

The categories

Sort every failure into exactly one bucket:

• unknown_tool — the agent asked for a tool that is not in known_tools. No retry fixes a missing tool.
• transient — a timeout or rate-limit / service-unavailable condition. The call might succeed if tried again, so this is the only retryable category.
• validation — the arguments were malformed (invalid value, missing required field). Retrying the identical call repeats the failure.
• tool_error — the tool ran but reported a domain failure of its own.
• unexpected — anything that matches none of the above.

retryable is True for transient and False for every other category.

Task

Complete classify_tool_error(tool_name, error, known_tools) using simple rules on the error type and message:

1. If tool_name is not in known_tools, return ("unknown_tool", False) before inspecting the exception.
2. Otherwise look at type(error).__name__ and the lowercased str(error):
   • transient if the type is a TimeoutError / RateLimitError / ConnectionError / ServiceUnavailableError, or the message mentions a timeout, rate limit, “too many requests”, 429, 503, or “temporarily unavailable”.
   • validation if the type is a ValidationError / ValueError / TypeError / KeyError, or the message mentions “invalid”, “validation”, “required”, or “missing argument”.
   • tool_error if the type is a ToolError / RuntimeError, or the message contains “tool error”.
   • otherwise unexpected.
3. Return (category, retryable) where retryable is True only for transient.

Check the rules in that order — the first matching category wins.

Example

known = {"search", "calculator"}

classify_tool_error("web_search", ValueError("bad"), known)
# -> ("unknown_tool", False)

classify_tool_error("search", TimeoutError("request timed out"), known)
# -> ("transient", True)

classify_tool_error("calculator", ValueError("invalid operand"), known)
# -> ("validation", False)

classify_tool_error("search", RuntimeError("upstream tool error"), known)
# -> ("tool_error", False)

classify_tool_error("search", MemoryError("out of memory"), known)
# -> ("unexpected", False)

Constraints

• Return a (category, retryable) tuple; category is one of the five strings above.
• retryable is True only when category == "transient".
• Rule order matters: screen unknown_tool first, then transient, validation, tool_error, and finally unexpected.
• Match on both the exception’s type name and a case-insensitive message substring; do not call any network or model.

Approach

Screen the cheapest, most decisive rule first: if the tool name is unknown, no exception detail matters, so return unknown_tool immediately. Otherwise reduce the exception to two features — its type name and a lowercased message — and match those against ordered sets of type names and signal substrings, one category at a time. The first category that matches wins, and retryable is derived from the category rather than computed separately, so the two can never disagree.

Solution

from typing import Set, Tuple

def classify_tool_error(
    tool_name: str,
    error: Exception,
    known_tools: Set[str],
) -> Tuple[str, bool]:
    if tool_name not in known_tools:
        return ("unknown_tool", False)

    name = type(error).__name__
    msg = str(error).lower()

    transient_types = {"TimeoutError", "RateLimitError", "ConnectionError", "ServiceUnavailableError"}
    transient_signals = ("timeout", "timed out", "rate limit", "too many requests", "429", "503", "temporarily unavailable")
    if name in transient_types or any(s in msg for s in transient_signals):
        return ("transient", True)

    validation_types = {"ValidationError", "ValueError", "TypeError", "KeyError"}
    validation_signals = ("invalid", "validation", "required", "missing argument")
    if name in validation_types or any(s in msg for s in validation_signals):
        return ("validation", False)

    tool_error_types = {"ToolError", "RuntimeError"}
    if name in tool_error_types or "tool error" in msg:
        return ("tool_error", False)

    return ("unexpected", False)

Walkthrough

Tracing the five example calls against known = {"search", "calculator"}:

1. ("web_search", ValueError("bad")) — "web_search" is not in known_tools, so the very first guard returns ("unknown_tool", False) without ever reading the exception.
2. ("search", TimeoutError("request timed out")) — tool is known; name == "TimeoutError" is in transient_types (and the message also matches “timed out”), so ("transient", True).
3. ("calculator", ValueError("invalid operand")) — not transient; name == "ValueError" is in validation_types (message also has “invalid”), so ("validation", False).
4. ("search", RuntimeError("upstream tool error")) — not transient, not validation; name == "RuntimeError" is in tool_error_types and the message contains “tool error”, so ("tool_error", False).
5. ("search", MemoryError("out of memory")) — matches no type set and no signal, so it falls through to ("unexpected", False).

Only case 2 comes back retryable, which is exactly the one a backoff-and-retry wrapper should act on.

Complexity & notes

• Time is O(S) in the total number of signal substrings scanned (a small constant), and space is O(1) beyond the fixed rule sets. Effectively constant per call.
• Order is load-bearing. Transient is checked before tool_error on purpose: a RuntimeError("request timed out") is a tool_error_type and carries a transient signal, and you want it classified transient so it gets retried rather than failed permanently. The first-match ordering resolves that overlap deterministically.
• Deriving retryable from the category (only transient is True) keeps a single source of truth. Returning them as independent values invites drift where, say, a validation failure is accidentally marked retryable and the agent loops forever on an argument it will never fix.
• Real systems enrich this: HTTP status codes as first-class input, honoring a Retry-After header for rate limits, and a retry budget so even transient failures give up eventually. The substring matching here is a pragmatic stand-in for structured error metadata, which is what you would prefer when the tools expose it.