As AI agents move from experiments to production in enterprises, security observability has become critical. Unlike traditional software systems, agentic platforms operate with probabilistic behavior and dynamic decision-making, which makes real time security monitoring essential. And that monitoring begins with a robust system for collecting Agentic data. Most of the agentic platforms capture traces, logs and metrics- Is that enough?
In this post, we explore two questions: What observability data is essential for threat detection—and how mature is the ecosystem now?
Observability Data for Security Use Cases
To effectively monitor AI agents, you need end-to-end visibility across the agentic pipeline. The following data are foundational for threat detection and incident investigation:
- Agent Metadata: Agent Name, Agent ID
- Interaction Context: Query, Response, Chain of Thought (CoT), System Prompt
- Tooling Details: Tool Name, Tool Command, Knowledge Source
- User Info: User ID, Username
- RAG Metadata: Citations, Document Links
- Timestamps: Query Timestamp, Response Timestamp
These parameters are needed for detecting threats like agent manipulation, overshared data access, tool poisoning and others. (For deeper dive into Agentic threats that can be detected with these data, check out our article: https://vinodvasudev.substack.com/p/why-agentic-ai-threats-could-eclipse)
How Easy Is It to Capture This Data Today?
The short answer: it depends on the platform. Agentic platforms are evolving and so is the case with their observability data. Here’s what we’ve learned-
- LangChain sets the benchmark with LangSmith, offering real-time capture of all key parameters and integration with open-source tools like Langfuse.
- Microsoft’s ecosystem is evolving:
- M365 Copilot supports real-time streaming via the AI Interactions API, but with limited parameters.
- Copilot Studio splits observability between Azure Insights (real-time, limited) and Dataverse (rich data, ~15-minute delay). Real-time access is on the roadmap.
- AutoGen leverages OpenTelemetry, making it compatible with multiple observability tools.
- No open-source integration yet for M365 or Copilot Studio.
- CrewAI supports Langfuse integration, in addition to its own AMP platform. Langfuse seems to be the more common integration option.
This table below distills our observations from a sample of leading agentic platforms
Long-Term Retention: A Missing Piece
One major gap across agentic platforms is long-term retention of observability data. For security teams, this is critical for historical analysis and compliance audits.
Our recommendation: Centralize observability logs from all agentic platforms in a data lake for long-term storage and analytics.
Final Thoughts
Security observability in agentic platforms is evolving rapidly. Platforms like LangChain have greater capabilities currently, while others are rapidly evolving to meet enterprise needs. If you’re deploying AI agents, make observability a first-class citizen in your architecture—it’s the foundation for threat detection and incident response.
(At Ackuity, we are building real time threat monitoring and incident investigation for AI Agents. This article captures some of our learning in collecting data from various platforms)