Most agentic platforms capture traces, logs and metrics, but security teams need more than that. To detect threats and investigate incidents, you need specific fields about the agent, the user, the tools and the data behind each interaction, captured in real time and kept long enough to audit.

This post looks at two questions: which observability data matters for threat detection, and how mature the platform ecosystem is today. It draws on what we learned collecting agent data from several platforms.

Why do AI agents need security observability?

Enterprises are moving AI agents from experiments into production. Agentic platforms behave probabilistically and make decisions dynamically, which traditional software does not, so security teams have to monitor them in real time. That monitoring starts with a reliable way to collect agent data.

What observability data does threat detection need?

You need visibility across the whole agentic pipeline. These fields are the foundation for threat detection and incident investigation:

  • Agent metadata: agent name and agent ID
  • Interaction context: query, response, chain of thought (CoT) and system prompt
  • Tooling details: tool name, tool command and knowledge source
  • User information: user ID and username
  • RAG metadata: citations and document links
  • Timestamps: query timestamp and response timestamp

With these fields you can detect threats such as agent manipulation, access to overshared data and tool poisoning. For more on the agentic threats this data can reveal, read our article on why agentic AI threats could eclipse earlier ones.

How easy is it to capture this data today?

It depends on the platform. Agentic platforms are changing quickly, and their observability data is changing with them. Here is what we found when this post was first published:

  • LangChain sets the benchmark with LangSmith, which captures all the key fields in real time and integrates with open source tools such as Langfuse.
  • Microsoft's ecosystem is still evolving:
    • M365 Copilot supports real-time streaming through the AI Interactions API, but with a limited set of fields.
    • Copilot Studio splits observability between Azure Insights (real time, limited) and Dataverse (rich data, with a delay of about 15 minutes). Real-time access is on the roadmap.
    • AutoGen uses OpenTelemetry, so it works with many observability tools.
    • M365 Copilot and Copilot Studio have no open source integration yet.
  • CrewAI supports a Langfuse integration in addition to its own AMP platform. Langfuse appears to be the more common choice.

The table below sums up what we saw across a sample of leading agentic platforms.

Table of agentic platforms and the observability tools that capture agent parameters: AI Interactions API for M365 Copilot, Azure Insights and Dataverse for Microsoft Copilot Studio, LangSmith and Langfuse for LangChain and LangGraph, and Langfuse for CrewAI

Why does long-term retention matter?

Long-term retention of observability data is a gap across agentic platforms. Security teams need that history for investigations and compliance audits.

We recommend centralizing observability logs from every agentic platform in a data lake for long-term storage and analytics.

Where does agentic security observability stand?

LangChain currently offers the most complete capture, and the other platforms are moving fast to meet enterprise needs. If you are deploying AI agents, design observability into your architecture from the start. Threat detection and incident response both depend on it.

How can observability data help stop an agent action before it runs?

Observability shows what an agent did. Ackuity, the Agent Execution Control Switch, uses the same kind of data to decide whether an agent action should run at all, and it makes that decision before the action executes.

Teams can start observe-only by pulling events from OpenTelemetry, Langfuse or LangSmith. To act on each action, they add an open source sidecar beside the agent, with no agent code changes. Before every action, execution control assembles the Agent Security Context Graph: 29 signals across six dimensions (user, agent, intent and goal, target system and data, tools and supply chain, and history), built outside the agent so the agent cannot edit it. Several of the fields listed above, such as agent ID, user ID, chain of thought and tool name, feed directly into those dimensions.

The decision comes back in 40 to 100 ms: Allow, Constrain, Human in the loop, Block or Terminate. See how execution control works on the Ackuity platform.

For a documented example of agents crossing infrastructure boundaries, read our OpenAI Hugging Face incident timeline and analysis, including the controls that limited the intrusion.