AI agent security metrics and reporting
Track AI agent security with a small set of counts: agents found, share covered by a blocking control, actions blocked versus only alerted, credential age, time to revoke and unapproved tools. A monthly report lists each against last month, names open gaps by layer and carries the date each control was last tested.
Facts read on .
What should you measure?
Measure coverage, enforcement, credentials and response time. Coverage is the share of known agents under a control, enforcement is blocks against alerts, credentials is how old and how broad they are, and response is minutes to revoke.
NIST's AI RMF asks organizations to measure and manage AI risk, which gives these counts a governance home.
Set a target for each measure and an owner. A coverage target of 95 percent of managed endpoints with the hook installed is concrete, and a note that says improve coverage is not. Targets can start modest and tighten every quarter.
- Agents inventoried, and agents with an owner.
- Share of managed endpoints with the enforcement hook installed.
- Actions blocked, asked, and only alerted, by layer.
- Age and scope of agent credentials; share that are short-lived.
- Minutes from decision to revoked access.
- Unapproved MCP servers and tools found and closed.
Sources: NIST AI Risk Management Framework.
Where do the numbers come from?
They come from the vendor consoles and the logs you forward. Check which export format each product offers before you plan a dashboard.
Bay says events reach the customer SIEM in OCSF format, Operant says Endpoint Protector forwards telemetry to Splunk, Datadog, Elastic and Chronicle, and Cisco says decisions can go to Splunk or any OpenTelemetry backend. OpenTelemetry's GenAI conventions name agent spans, which helps you join data across tools.
If a vendor offers no export, ask for a report on a schedule or an API. Several vendors list SIEM or OpenTelemetry export in their documentation, and others list none found, which is a factor when you pick between products with similar blocking.
Sources: Bay: AI agent investigation; Operant AI: Endpoint Protector; Cisco: protect enterprise coding agents with DefenseClaw; OpenTelemetry GenAI semantic conventions.
What belongs on a monthly report?
A monthly report fits on one page: five counts against last month, the top three open gaps by layer, incidents with time to contain, and the date each control was last tested.
Add one line on policy changes and one on exceptions granted. Reviewers want to know what changed, not the whole inventory.
Keep a standing section on exceptions. Count the exceptions granted, who approved them and when each expires. An exception with no expiry is a permanent gap that nobody remembers.
Sources: NIST AI Risk Management Framework.
How do you count agents you did not know about?
Count discovered agents separately from approved ones, and track the gap. A rising discovered count with a flat approved count means shadow agents are growing.
Microsoft's release notes describe local AI agent discovery on Windows and macOS that reports each agent's configured MCP servers, which gives an endpoint-side count.
Set a rule for what happens to a newly found agent: assign an owner within a week, approve or block within two. Report how many sit past that deadline.
Which metrics mislead?
Alert counts mislead because a noisy rule inflates them and a silent gap hides. Pair every alert count with the block count and the coverage share.
Vendor dashboards also count only what the product covers. Compare against your own list of agents by layer.
Add a measure for the time between a vendor's feature release and your use of it. Agent security products change monthly, and unused controls count as no coverage.
Sources: Zero Trust overview, Microsoft Learn.
How often should you review the targets?
Review them quarterly, and after any new agent platform or a major vendor change. Keep the definitions stable so the trend means something.
Record each definition change with its date on the report.
Share a short version with leadership: coverage, blocks, incidents and the largest open gap, in plain words. Keep the detail for the security team.
Sources: NIST AI Risk Management Framework.
Sources
- NIST AI Risk Management Framework (nist.gov), read
- Bay: AI agent investigation (bay.io), read
- Operant AI: Endpoint Protector (operant.ai), read
- Cisco: protect enterprise coding agents with DefenseClaw (blogs.cisco.com), read
- OpenTelemetry GenAI semantic conventions (opentelemetry.io), read
- What's new in Microsoft Defender for Endpoint (learn.microsoft.com), read
- Zero Trust overview, Microsoft Learn (learn.microsoft.com), read