Monitoring Agentic Tool Use with Observability Data

Over the past few months, we’ve witnessed a growing concern in the agentic AI space: agents going rogue and executing harmful actions using the tools they’ve been granted access to. An example is the Replite agent incident, where an AI agent deleted a production database. (Read the story at- https://www.pcmag.com/news/vibe-coding-fiasco-replite-ai-agent-goes-rogue-deletes-company-database)

At Ackuity, we’re focused on building real time detection of threats and misbehavior of agents. One of our use case is to agent tool usage to detect and respond to malicious or unintended actions before they cause damage. We are using observability data of Agents along with logs from IT systems.

Let’s take a simple example: a LangChain agent designed to query an Azure SQL database. Under normal conditions, it runs harmless SELECT statements. But if compromised or misconfigured, it could start executing UPDATE, DROP, or DELETE commands—corrupting or erasing critical data.

This is where observability data becomes essential.

Using tools like LangSmith, we can capture detailed traces of agent behavior, including tool execution. For instance, we can detect when an agent uses the SQL_Database tool to run a query like DELETE FROM Employees WHERE Name = 'Emily Rivas'. This visibility allows us to flag and investigate suspicious actions immediately.

Similarly, for low-code agents built with Copilot Studio, we leverage telemetry from Azure AppInsights to monitor operations like record deletions in Azure SQL

Using observability data, we can do a range of detection and response around tool misuse:

1. Tool Usage Profiling: We build behavioral profiles for agents based on their typical tool usage. For example, if an agent usually interacts with an Email tool but suddenly invokes a SQL tool, it could be a sign of compromise—such as an XPIA-style attack.
2. Action-Level Anomaly Detection: Even within a known tool, we monitor for unusual actions. A database agent meant to read employee data should not be executing DELETE or UPDATE statements.
3. Risky Command Monitoring:  We flag high-risk operations like DROP, DELETE, or execution of malicious Terraform scripts. Below is a snapshot of an alert in Ackuity triggered by a record deletion in the database.

We believe that observability is key to securing agentic systems. If you're working on similar challenges or exploring new approaches to monitor agent behavior, we’d love to connect and exchange ideas.