Yoav Land
The Minds · AI observability and AWS

Don't go to production blind

By Yoav Land

CEO, INNO by YouCC ·

Observability is what takes AI from POC to production.

The gap between POC and production is not a technology gap. It is a trust gap, and trust is built on visibility. A failing model does not throw an error; it returns a fluent, confident and wrong answer. To go live, you need to see, measure and improve: end-to-end tracing, continuous evaluation, drift detection, a full audit of every action, and control over cost. Observability is not an add-on after launch. It is the condition for launch.

The distance between POC and production

Almost every organization in Israel has run a GenAI POC in the past two years. The demo worked, management was excited, and then the project stalled. Not because the model wasn't good enough, but because no one could commit that it would still work tomorrow.

I have seen this pattern repeat itself again and again, from startups to large enterprises. The gap between POC and production is not a technology gap. It is a trust gap, and trust is built on visibility.

AI doesn't fail like the code we write

When code breaks, it returns a 500 error and someone gets paged. When a language model fails, it returns a fluent, confident and wrong answer with a 200 status code. It is a silent failure, and classic monitoring will never raise an alert.

The list is long: hallucinations, answers that drift from company policy, RAG pipelines that retrieve the wrong document, agents stuck in loops burning tokens, and answers that gradually degrade until the model is no longer as good as it was (drift). Any one of these can sink a project, and all of them are invisible without the right tools.

Then there is cost. In the POC you paid for a thousand calls; in production you pay for millions. Real users asking real questions over time behave very differently from your POC test set. If you don't track cost per request, you will find out from the invoice.

Observability for agentic AI

Observability for AI agents is not a CPU and memory dashboard, and it is not just a log of prompts and responses. An agent receives a goal, plans, makes decisions and takes actions on its own against real systems. Knowing what it answered is not enough; you need to see how it reached that answer and what it did along the way. Observability here means continuously monitoring, in real time, the decisions, actions and reliability of the agents and of the system as a whole. It rests on five building blocks:

  • End-to-end tracing of the agent's path: Every run is recorded, from the goal it received, through planning and decision steps, to the final outcome. That includes retrievals from memory and the knowledge base, tool and API calls with their parameters and results, and handoffs between agents, with latency, tokens and cost logged at every step. This shows not only what the agent did, but why.
  • Continuous evaluation (evals) of the process, not just the outcome: Automated checks on live traffic, including LLM-as-a-judge applied to complete trajectories. Did the agent complete the task? Did it pick the right tool with the right parameters? Did it get there in a reasonable number of steps? Metrics include task success, tool-use accuracy, faithfulness to source, safety and compliance.
  • Drift and anomalous behavior: Alerts when agent trajectories start deviating from the expected pattern, such as loops, repeated retries, unexpected tool usage or decisions that change after a model update. Also alerts when questions, data, embeddings or responses from external tools drift from the baseline you built on.
  • Actions, permissions and audit: An agent doesn't just generate text; it acts. It sends emails, updates records and executes transactions. That requires a full record of every action taken on a user's behalf, visibility into which actions required human approval and what was blocked by guardrails, and an immediate alert on any attempt to act beyond its permissions.
  • Cost and performance: An agent stuck in a loop can burn through a budget in minutes. So measure cost per complete task, not just per model call, along with step count, end-to-end latency and token usage by agent, tool, feature, customer and model. Set budget caps and alerts on anomalies.

A chatbot that gets it wrong returns a wrong answer. An agent that gets it wrong also takes a wrong action: it updates a CRM record, emails a customer, approves a refund or runs a query against a production database.

An agent is not a single model call but a chain of decisions: plan, choose a tool, call an API, read the result and decide what comes next, sometimes dozens of steps per task. A correct final answer can hide a problematic path, and a mistake in step three cascades into everything that follows.

That is why observability for agents has to answer questions that did not exist before:

  • Why did the agent take this path?: A full trace of every step, including the decision, the tool selected, the parameters sent and the result returned.
  • Is it using tools correctly?: Detecting the wrong tool, wrong parameters, or repeated calls that signal a loop.
  • Is the path itself sound?: Evaluating the entire trajectory, not just the final answer, including step count, task success and staying within permission boundaries.
  • What does it cost?: An agent stuck in a loop can burn a full day's budget in minutes. Alerts on cost and on steps per task are not a luxury.
  • What happens when agents talk to agents?: In multi-agent systems, you need to follow a request as it moves between agents, not only within each one.

There is also a corporate governance dimension. Whoever lets an agent act on the organization's behalf must be able to know, after the fact, what it did and why. Observability is therefore also the audit and compliance infrastructure for agentic AI. An organization that cannot explain an agent's decision will not let it act on its own, and rightly so.

AWS as the foundation

The good news: if you build on AWS, you start with a strong foundation. Amazon Bedrock provides managed access to leading models, including Anthropic's models, together with built-in guardrails and evaluation capabilities. Amazon Bedrock AgentCore brings built-in observability for agents through Amazon CloudWatch, and Amazon SageMaker covers model monitoring for classic ML.

The cornerstone here is OpenTelemetry. AWS has adopted the open standard, so your traces are not locked into a single tool: collect once, send wherever you need.

The missing layer: dedicated AI observability

Still, native tools provide a foundation, not the full picture. Data science teams and AI engineers need an environment built around their questions: why did this answer fail, which types of questions produce hallucinations, and how did the new prompt version affect quality.

This is where dedicated AI observability platforms come in. They sit on top of AWS and integrate natively with Bedrock, SageMaker and agents. The best of them are built on OpenTelemetry and open source. They let you investigate any trace, run automated evals on live traffic, detect embedding drift and close the loop between production and development.

There is a business advantage too. When such a platform is available on AWS Marketplace, it is purchased through your existing AWS account. In many cases it also counts toward the organization's committed spend with AWS, so the question is not new budget but better use of existing budget.

Four steps from POC to production

  • Instrument from day one. Instrumentation goes in during the POC, not after it. If your demo has no traces, you won't know why it worked.
  • Define what "good" means. Agree an evaluation set with the business: quality metrics, an acceptance threshold and a maximum cost per request. Without it, there is no exit criterion for the POC.
  • Roll out gradually, with continuous monitoring. Shadow mode or a limited user group, evals on live traffic, and alerts on quality and cost, not just availability.
  • Close the loop. Every production failure goes into the evaluation set, and every prompt or model change is tested against it before it ships.

Those who measure, go live

The models are already good enough. What separates an organization stuck in experimentation from one that gets real value from AI is the ability to see, measure and improve. Observability is not an add-on after launch. It is the condition for launch.

At YouCC INNO we see this in every Data & AI project we support on AWS: the projects that go live are the ones that built visibility from day one. If your AI still lives in a slide deck, stop asking "is the model good enough?" and start asking "can we see what it is doing?"

More from The Minds