ROOT Cloud Native Day 2026 · Gandhinagar
USE OBSERVABILITY
THE CRITICAL IMPORTANCE OF LLM
OBSERVABILITY IN ENTERPRISE AGENTS
— from a single AI agent to a production multi-agent system
AP
Ashish Patel
Sr Principal Architect — AI · ML · DS @ Oracle
Sept 5 2026 · 45-min technical session · advanced devops & infrastructure track
obs-cli · session bootroot span
talk.preview12 spans · one trace
core ▸ root span opened — 45 minutes, one system (Meridian Claims), one trace. press → to open span 01.
01 The Mental Model
From AI Agent to Production Multi-Agent System — using a real claims-adjudication system as the example
① an LLM — text in, text out. no state · no tools · no decisions.
core ▸ one picture, five layers: llm → control loop → tools & memory → specialists → the plane that watches it all. (press → to build it)
02 The Problem
Why Traditional Observability Isn't Enough
Traditional Servicedeterministic
CLAIM SUBMITTED
▼
SERVICEknown logic · known paths
▼
ERROR ✗
▼
HTTP 500
fails loud — the pager fires in seconds, the stack trace names the line.
Meridian Claims · Same Requestnon-deterministic
tool called ✓decision reached ✓HTTP 200
decision: wrong
in health insurance, this isn't a UX bug — a wrongly denied claim with no reconstructable reason is a compliance and liability exposure, not just a bad answer.
core ▸ HTTP 200 ≠ correct behavior — the medical history agent called its tool correctly and still missed the exclusion.
03 Trace Anatomy
Anatomy of a Claim's Trace
The System · Policy Verification Agentwhat ran
The Trace It Emitswhat it recorded
SESSION CLM-9001 · 6 spans · 4.9 s
├ policy.run4.90sroot span · status OK · 1 llm + 1 mcp call
├ llm.decide0.9sqwen2.5:7b · 312 in / 88 out tok · local
├ mcp.check_coverage0.4sstdio · policy POL-1001 · procedure 99213
├ finding.recorded0.1scovered:true · deductible_met:true
├ user.tag0.02spatient_id → langfuse user · cost by patient
└ score.groundedness1.6sapp-managed judge · 1.0 · PASSED — quotes tool output
▸ hover a span — see exactly where it lives in the system. every field above came out of a real Langfuse trace, not a mock.
core ▸ every span is a decision you can interrogate: tokens, MCP calls, groundedness — parent → child, all the way down.
04 Distributed Intelligence
Multi-Agent: Where Failure Is Born
Supervisor + 3 Specialists + Decision Gatecycle 1
every arrow is a span boundary — the trace context travels with each handoff
Live Session · CLM-9004● replaying
the failure was born at one MCP call — the pipeline still reached a decision. only the trace tells you which node went quiet.
core ▸ three specialist agents, one decision gate — this is already "no longer a chatbot, a distributed system."
05 Signals
What Should We Observe?
①SYSTEM
FastAPI request latency · SQLite write path · SSE connection health
the world your dashboards already own — this axis never changed for agents
②LLM
940 tokens/claim · 4.9s p50 · qwen2.5:7b local, $0/call
local inference makes cost a non-issue here — latency is the real budget
③AGENT
which MCP tool fired · sequence order held · one Langfuse session per claim
the control loop is the application now — a wrong tool choice is a bug, not a stack trace
④QUALITY
decision_correct score · groundedness 0-1 · 4/4 eval dataset pass rate
the layer everyone forgets — a claim can look "done" and still be wrong
Where Each Lens Looks
1
2
3
4
① substrate · ② the brain · ③ the control loop & handoffs · ④ the output gate
TRADITIONAL+
LLM+
AGENT+
EVALS=
AI OBSERVABILITY
core ▸ four lenses — system, llm, agent, quality. miss one, and you are blind on that axis.
06 Landscape
The Tool Landscape — Five Layers
L5
EVALUATIONquality · regression · drift
Langfuse DatasetsLangfuse Scoresapp-managed LLM-judge
watches: correctness
before every deploy
before every deploy
L4
AGENT OBSERVABILITYtraces · sessions · users
Langfuse TracingLangfuse SessionsLangfuse Users
watches: the loop
one session per claim
one session per claim
L3
LLM GATEWAYrouting · fallback · cost
Ollama (local, primary)OpenRouter (fallback)
watches: every call
free-tier fallback chain
free-tier fallback chain
L2
TOOL INTEGRATIONMCP · stdio · structured data
MCP server (stdio)CrewAI MCP adapter
watches: tool calls
policy · history · fraud
policy · history · fraud
L1
APPLICATIONFastAPI · SQLite · Next.js
FastAPISQLiteNext.js portal
watches: substrate
everything runs on
everything runs on
The System Every Layer Watches
Langfuse is more than tracing here — Sessions group a claim's whole pipeline into one timeline instead of three disconnected traces. Users attribute LLM spend per patient, not just per app. Datasets + Scores turn "we fixed a bug" into "4/4 green," not a shrug. One honest caveat: the groundedness judge is app-managed, not Langfuse's native evaluator-rule engine — that migration is a known follow-up, not a finished feature.
core ▸ five layers, open standards at the seams — MCP for tools, Langfuse for the rest.
07 The Climax
Production Architecture — Meridian Claims
The Full System · One Trace, One Planestage 5 — everything on
GET /claims/{id} → findings + decision + Langfuse session URLone query plane — the portal, the claim, and the trace are always one click apart
①NEXT.JS PORTAL
SSE ─▶ live agent activity
claims queue · new-claim form · audit trail · every finding links its trace
②MCP TOOL SERVER
CrewAI agents ─▶ stdio MCP server
check_coverage · lookup_medical_history · fraud_score — real tool calls, real spans
③LANGFUSE CLOUD
agents ─▶ Sessions · Datasets · Scores
one session per claim · eval dataset re-run on every fix · continuous groundedness scoring
core ▸ the observability plane is a first-class citizen of the architecture — not a bolt-on dashboard. no docker required.
07a Meet The System
Meridian Claims — the actual system, named
core ▸ five real components, one project name — nothing on this slide is a stand-in for something else.
07b Now Watch It Run
Live Demo — Meridian Claims
switching to localhost:3000
file a real claim · watch the floating "thinking" widget name each agent live · watch the audit trail fill in with real MCP tool output · click "View trace ↗" into the real Langfuse session
✓ real MCP calls✓ real SSE stream✓ real Langfuse trace✓ no mocks, no slides
core ▸ everything on the last slide, running — this is the part no diagram can substitute for.
08 Closing the Loop
The Production Improvement Loop
Trace → Test → Deploy → Traceauto-cycling
a real eval run flagged claim CLM-9004 — expected human_review, got approved.
the fraud signal for that claim had no real reason to route it — a seed-data gap, not an agent bug.
one fix to the fraud-signal table → re-run → 4/4 green.
run_evals.py → 1 FAIL (CLM-9004) → fix mcp_server.py → run_evals.py → 4/4 PASS
↺ this happened during this project's build — not staged for the talk
core ▸ production failures become permanent test cases — the system gets safer with every incident.
09 The Takeaway
If You Remember One Picture
core ▸ agent = llm + loop + tools + memory · observability = traces + scores + evals → improvement.
10 Closing · Q&A
IF YOU CAN'T
SEE IT, YOU
CAN'T SHIP IT
// open floor — architectural questions · Ashish Patel
agents.pymcp_server.pytracing.pyseed_evals.py
scan ▸ connect on LinkedIn
linkedin.com/in/ashishpatel2604
linkedin.com/in/ashishpatel2604
QUEEN'S UNIVERSITY · GIFT CITY · GANDHINAGAR — CLOUD NATIVE DAY, SEPT 5 2026