Reading Time: 3 minutes
It’s not computing properly, but as a gate for autonomous action it’s measuring the wrong thing, and it’s dangerously incomplete if used alone. The log normalization point in the post is the strongest part of it.
Where the 95% framing falls short
- Detection rate isn’t what governs autonomous action. False positives are. The 95% figure is about recall: how many real attacks it catches. Once an agent acts on its own, the risk comes from what it does to legitimate activity. A bank SOC might process tens of millions of events a day. Even a 0.1% false positive rate on 10 million events is 10,000 wrong actions daily, such as disabled accounts, isolated hosts, blocked payment flows, or locked-out branch staff. A 95% detector with poor precision is far more dangerous in production than an 85% detector that is almost never wrong when it fires.
- “Accuracy” is a misleading metric here. Attacks are a tiny fraction of events. A model that labels everything benign scores about 99.9% accuracy. If the post means detection rate, it should say so, and it still needs a precision or false positive requirement next to it.
- 100 test attacks is a thin sample. Catching 95 out of 100 gives a 95% confidence interval of roughly 89% to 98%. The test set also usually consists of known techniques, often replayed red-team scenarios, so it says little about novel tradecraft or adversaries who adapt once they learn how the agent behaves.
- It matters which 5% gets missed. Missing five commodity phishing attempts is very different from missing the one SWIFT operator credential compromise. The threshold should be weighted by severity and mapped to ATT&CK coverage, not treated as a flat count.
- A single gate treats autonomy as all-or-nothing. The better approach is to authorize autonomy per action, based on blast radius and reversibility:
- Enrichment, ticketing, and quarantining a suspicious email can be autonomous with modest confidence.
- Isolating an endpoint or forcing a password reset needs high precision and an easy rollback.
- Disabling a privileged or payment-system account, or blocking core banking traffic, should require a human approver no matter what the test score is.
- The benchmark should be relative to the current baseline. The real question is whether the agent beats your existing Tier-1 triage on both missed detections and false escalations, not whether it hits a fixed number.
- It treats approval as a one-time event. Model drift, new log sources, and attacker adaptation erode performance. A better pattern is to run the agent in shadow mode against live traffic, compare its decisions with analyst outcomes, then expand autonomy gradually. Live precision and recall should be monitored continuously, with automatic rollback if they degrade.
Where the post is right
The normalization and entity-mapping point is solid. Most “AI hallucinated a correlation” failures in SOCs come from inconsistent field names, missing identity resolution (one user appearing as three different principals), and legacy core banking logs with no clean schema. Using a common schema such as OCSF or ECS, plus an asset and identity graph, is a prerequisite for any agent to perform reliably. Without that groundwork, no threshold means much.
Bottom line
95% detection is a reasonable floor as one criterion among several. It becomes a problem when it’s presented as the gate. A defensible standard for a bank would combine:
- a precision or false positive ceiling,
- severity-weighted coverage,
- autonomy tiers by action impact,
- shadow-mode validation on live data,
- continuous monitoring with human accountability for high-impact actions.