AI Agent Observability: How to Make Agents Improve Every Run

On October 12, DigitalOcean launched Agent Learning Services and Agent Data Services, a set of tools built around one idea: an AI agent in production should get better every time it runs. Its product chief, Vinay Kumar, framed it this way: "The question has changed from 'can we launch an agent' to 'does our agent get better with every run.'"

That shift is why AI agent observability has moved from the engineering backlog onto the business owner's desk. If you can't see what your agent did, step by step, you can't fix it, you can't prove it's working, and you can't safely swap the model underneath it. Here's what launched, what it signals, and what to set up whether or not you ever touch DigitalOcean.

TL;DR#

• DigitalOcean shipped a record, flag, test loop for agents (Insights, Signals, Evaluations & Simulations), now in public preview, plus a data layer in private preview.
• Agents rarely crash. They loop, stall or quietly give wrong answers, and normal uptime monitoring won't catch any of that.
• You don't need a new platform to start. You need step-level traces, a short list of failure signals and a test set built from real sessions.

What DigitalOcean actually launched#

The release describes Agent Learning Services as one loop made of three products:

• Insights records every step an agent takes. Forkast's analysis notes it's built on OpenTelemetry.
• Signals flags the sessions that matter: where the agent stalled, looped, lost the user or failed.
• Evaluations & Simulations tests a proposed change against realistic conversations built from those flagged sessions, before real users see it.

Around that loop sit two more pieces. An updated Inference Router picks the best-scoring model for each task based on cost, quality or your own evaluations, and reruns those evaluations as new models ship. Agent Data Services is the storage underneath: video, audio, documents, logs and agent sessions become searchable by meaning, with configurable redaction, per-customer isolation and opt-in use of data for training.

Availability matters here. The learning tools are in public preview. The router's optimizer and the data layer are private preview, and DigitalOcean's own fine print says preview services aren't supported for production workloads. No pricing or benchmarks were published.

Why AI agent observability matters more than the model you pick#

Traditional software fails loudly. A server throws a 500, an alert fires, someone fixes it. Agents fail quietly. Every API call returns 200 OK while the agent calls the wrong tool, retries the same search four times, or tells a customer their order shipped when it didn't.

Picture a support agent that checks order status. One day the shipping provider renames a field. The agent keeps answering confidently, using an empty value. Your dashboards look healthy. The only people who notice are customers, and most of them just leave.

That's the gap AI agent monitoring has to close. You need to see the reasoning path, not just the response time: which tools were called, with what inputs, what came back and what the agent decided next.

There's a second reason. Models change constantly. Without an evaluation set that reflects your real work, every model swap is a guess. With one, it's a measured decision you can make in an afternoon.

The improve loop, broken down#

Strip away the branding and the loop looks like this:

production session
      |
      v
trace: every LLM call, tool call, retrieval, decision
      |
      v
signals: stalled, looped, failed, user dropped off
      |
      v
flagged sessions become test cases
      |
      v
change prompt / tool / model, run evals and simulations
      |
      v
ship only if scores hold or improve, then repeat

What a useful trace captures#

At minimum: the user input, each model call with its prompt and output, each tool call with arguments and results, latency and token cost per step, and the final outcome. The OpenTelemetry GenAI work is defining shared conventions for this, though the agent-specific parts are still settling. Building on OpenTelemetry now keeps your options open later.

Managed platform or build your own#

We've written before about the cost of vendor lock-in. The same logic applies here: keep traces in an open format so you can move them.

Privacy is part of the design#

Traces contain customer messages, order details and sometimes payment context. Redaction has to happen before data lands in storage, not as a cleanup job later.

What this means for businesses running agents#

If you have an agent answering customers, processing orders or triaging tickets, three things follow.

First, budget for evaluation, not just the build. A working demo is the starting line. Most of the long-term value comes from the weeks after launch, when real sessions show you where the agent breaks.

Second, someone has to own the loop. Signals are useless if nobody reviews flagged sessions. That can be one person for an hour a week, but it has to be a named person.

Third, AI agent evaluation is what makes cost savings safe. Cheaper models will keep arriving. If you can rerun your test set in minutes, you can switch with confidence. If you can't, you'll either overpay for months or switch blind and find out from customers.

The industry direction is clear too. DigitalOcean says it plans to extend automated improvement to agent workflows, adding to today's human approvals. That's a roadmap item, and you can't trust an automated fix you can't inspect.

How to set up AI agent evaluation and monitoring: a checklist#

1. Trace every step. Log model calls, tool calls, inputs, outputs, latency and cost per session, ideally in OpenTelemetry format.
2. Redact at the source. Strip personal and payment data before it's stored.
3. Define 4 to 6 failure signals. Start with loops, stalls, tool errors, escalations to a human and sessions the user abandons.
4. Build a test set from real sessions. Take 30 to 50 flagged conversations, write down the correct outcome for each and keep them as regression tests.
5. Gate every change. Prompt edits, tool changes and model swaps all run against the test set before they ship.
6. Track a few business metrics. Resolution rate, human handoffs and cost per task tell you more than accuracy scores alone.
7. Review weekly. Look at the worst ten sessions, add new failures to the test set and fix the top issue.
8. Keep humans on risky actions. Refunds, cancellations and anything touching money stay behind an approval step until the data says otherwise.

How MagicMakers Lab approaches this#

When we build agents through our Agentic Development work, decision logging and human approval gates go in from day one, so we can replay the exact path when something goes wrong. It's the same discipline behind Framico, where 200+ orders a day ship with zero manual steps: you can only remove humans from a process when you can see everything the system does.

Key takeaways#

• DigitalOcean's launch puts the improve loop (record, flag, test) at the center of agent platforms.
• Agents fail quietly, so step-level traces matter more than uptime checks.
• Evaluation sets built from real sessions let you change prompts and models without guessing.
• Open formats like OpenTelemetry reduce lock-in while standards settle.
• The loop needs a named owner and a weekly review, or the data goes unused.

FAQ#

What is AI agent observability?#

AI agent observability is the practice of recording every step an agent takes, including model calls, tool calls, retrieved data and decisions, so you can see why it produced a given result. It goes beyond uptime monitoring. The goal is to spot loops, wrong tool use and silent failures, then use those sessions to test and improve the agent.

How is AI agent monitoring different from regular application monitoring?#

Regular monitoring checks whether services are up, fast and error free. Agent monitoring checks whether the agent made good decisions. An agent can return a successful response while calling the wrong tool or giving an incorrect answer. That's why agent monitoring tracks reasoning paths, tool results and session outcomes, not just latency and error codes.

How do you evaluate an AI agent before deploying a change?#

Build a test set from real conversations, especially ones where the agent failed, and write down the correct outcome for each. Run every prompt edit, tool change or model swap against that set and compare scores with the current version. Ship only when results hold or improve. Simulations can extend this to multi-turn conversations.

Do small businesses need AI agent observability tools?#

If an agent talks to customers or touches orders, yes, but it doesn't have to be expensive. A small team can start with structured logs in their own database, a handful of failure signals and a weekly review of the worst sessions. Paid platforms make sense once volume grows or several agents need tracking.

Running an agent in production and not sure what it's actually doing? We'll look at your setup, show you where it's likely failing quietly and outline the tracing and evaluation you'd need to fix it. Book a free audit.

Sources#

• DigitalOcean Launches Agent Learning and Data Services for Recursively Self-Improving AI Agents (DigitalOcean Investor Relations)
• Managed Trust: DigitalOcean's Bet on the Agent Improve Loop (Forkast)
• DigitalOcean Launches Agent Learning and Data Services (BigDATAwire)
• AI Agent Observability: Evolving Standards and Best Practices (OpenTelemetry)