Token Thrift
  1. Ask
  2. Agents and governance
  3. Agents
  4. Your account
Started by Token Thrift

Agents / 3 public sources

How should you evaluate AI agent observability tools?

Add your answer

Loading replies.

Add your answer

Don't paste anything your employer treats as confidential. Before anything is posted, we take out names of people, your company, its clients and internal projects, plus places and exact numbers, and show you the result to check. Your raw text is never stored. To redact it, we send it to Anthropic's API, which doesn't train on API data by default.

How Ask protects you

  • You post under a random handle. No name or employer is shown, and your email is never shown to anyone.
  • Before a reply is saved, a model removes names, employers, clients, places and exact figures, a fixed leak check reads it again, and you approve the result. Your raw text is never stored. It is sent to Anthropic's API for this step. Redaction lowers the risk; it is not a guarantee.
  • Follow-up questions are passed on by email, and neither person sees the other's address. You can answer privately.
  • A handle's first reply is reviewed before it is published.
  • Replies that are never published are deleted after 30 days. Deleting your handle removes your email; published replies stay up as anonymous text.

More in the privacy policy and the FAQ.

What you get for replying

  • A digest of everyone else's answers on that topic, at most one email a day, one click to stop.
  • A count of replies readers marked helpful, shown next to your handle.
  • Credit by handle in the monthly roundups when your answer is used.

Tips to stay hard to recognize

  • Describe your company by size and industry, not by name, product or city.
  • Leave out unusual tool combinations or internal project details only your team would know.
  • Round numbers and timelines yourself, even though redaction also does it.
  • Wait a while before writing about something that was just announced inside your company.
  • Redaction doesn't change your writing style, so a coworker with context could still recognize you.

Summary of public sourcesWritten by our research agent from 3 public sources. Not first-hand experience.

Once agents call tools and touch real systems, teams want to see what each run did and why. A public devops thread, started by someone who said they work for a vendor in the space, listed three things to compare: tracing at the session level rather than single calls, how deeply each agent framework is covered, and whether agent steps link to traces from the services they touch. One reply ranked session traces and service links highest and token cost lowest, since cost can be rebuilt from provider invoices.

The most concrete test suggested was a blind replay: take 10 to 20 past incidents, run them through each candidate, and score whether it surfaced the right root cause, flagged harmful actions, and logged every step. OpenTelemetry's GenAI conventions give a vendor neutral shape to check against. They define spans for invoking an agent and for each tool call, with attributes for agent name, conversation ID and tool call ID, and keep tool arguments and results opt in. The conventions are still marked as in development, so coverage differs by framework.

Still open: how much prompt and tool output to keep given privacy and storage, and whether tools built on open conventions are worth the extra setup over a closed agent specific product.

Sources

  • Reddit: How should we evaluate AI agent observability?
  • Standard: Semantic conventions for GenAI agent spans | OpenTelemetry
  • Standard: Semantic conventions for generative client AI spans | OpenTelemetry