What problem does it solve?
A detection is the rule or query that turns raw logs into an alert: a sequence of events that means somebody is abusing an account, an exploit hitting a web server, a tool that attackers use to move between machines. New attacker behaviour is published every day in threat reports and vulnerability disclosures, and each item has to become a detection in the language of each platform the security team runs, be tested against real traffic and be tuned until it stops producing false alerts.
Amazon describes the manual process for a new vulnerability. An analyst finds public proof of concept exploit code, studies it, writes a rule, writes queries to measure the rule's accuracy against traffic logs, tunes it to reduce false positives, repeats until it is good enough, and then sends it to another engineer for code review. The time that takes meant the team "had to carefully prioritize which vulnerabilities to cover first".
Platform changes add a second backlog. SAP Enterprise Cloud Services describes manual detection engineering that consumed analyst time, tuning rules to cut false positives as "an ongoing drain on resources" and limited MITRE ATT&CK coverage on a legacy SIEM, which made it hard to measure progress. mnemonic, which runs detections on several platforms, says that every new piece of attacker knowledge has to be read, placed in a taxonomy, expressed in the right query language, tested and checked against what already exists.
How does it work?
- Intake. The system takes a trigger: a newly disclosed vulnerability with example exploit code, a threat report from a feed, a gap in coverage against MITRE ATT&CK, or a plain language request from an engineer. Items are ranked, for example by exploit content and threat intelligence.
- Check what already exists. An agent queries the detection inventory (for example through an MCP server) to see whether the behaviour is already covered, and writes down the gap.
- Draft candidates. A generation step writes one or several candidate rules in the query language of the target platform, with metadata such as ATT&CK technique, data sources, expected false positives and tuning notes. Reusable skills or templates hold the house format.
- Evaluate and test. A separate judge model scores each rule, for example for how likely it is to miss the attack and how likely it is to match something that only resembles it. Rules are run against historical logs and known attack samples, and rules that fail go back to the generation step with the reason.
- Human review. An engineer reads the logic, the test results and the false positive estimate, edits it and approves. Merge, deployment and the rollout to production follow the normal peer review and change process.
- Tune from feedback. Alert outcomes from analysts feed back into recommendations to tighten or retire rules.
- Audience
- Employee facing
- Autonomy
- Copilot
- Adoption
- Emerging
- Channels
- Internal tools, API and system to system
What is it worth?
Benchmarks are computed from the public deployments below: one data point per organization per KPI, with who made each claim.
| KPI | Median | Reported range | Data points | Claimed by |
|---|---|---|---|---|
| False positive reduction | Too few to pool | 67% | 1 | 1 organization |
| Handling time reduction | Too few to pool | at least 60% | 1 | 1 vendor |
| Productivity gain | Too few to pool | 336% | 1 | 1 organization |
Value drivers: Employee productivity, Speed and cycle time, Risk and loss reduction.
Indicative value
A security team of an enterprise with two to six detection engineers and a SIEM
USD 20,480 to USD 491,400
Engineer time released from rule work, valued at cost per year
How this is calculated
Formula: engineers * shareOnRules * hoursPerYear * timeSavedShare * costPerHour. The low scenario uses every low input, the high scenario every high input.
| Input | Low | High | Basis |
|---|---|---|---|
| Detection engineers engineers, engineers | 2 | 6 | Editorial assumption for a large enterprise. Replace with your own headcount. |
| Share of an engineer's working year spent writing, translating and tuning rules shareOnRules, fraction of working time | 0.4 | 0.7 | Editorial assumption, replace with a time study from your own team. |
| Working hours per engineer per year hoursPerYear, hours | 1,600 | 1,800 | Editorial assumption that allows for leave and meetings. |
| Share of rule work the AI saves timeSavedShare, fraction of rule work | 0.2 | 0.5 | Well below the figures on this page, where Databricks reports a reduction of 60% or more at SAP Enterprise Cloud Services and Amazon reports a 336% productivity advantage. Both include a platform change or a purpose built system, and neither is an independent measurement. |
| Fully loaded cost of an engineer hour costPerHour, USD per hour | 80 | 130 | Editorial assumption, replace with your own fully loaded cost. |
What it leaves out: Counts engineer time only. It leaves out the cost of the AI and the test environment, the time to review every draft, the value of covering vulnerabilities that would otherwise get no rule, and the cost of a rule that misses an attack or floods analysts with alerts.
Who already uses it?
3 public deployments, strongest evidence first. Grades: A regulator or audit, B the organization itself, C vendor case study, D anonymous or estimate.
mnemonic
Norway · Technology and software · 2026
mnemonic, a Norwegian security services company, describes an agentic workflow that reads threat intelligence reports, lists the attacker behaviours worth detecting, checks its detection inventory through an MCP server to see what is already covered and drafts a detection in its common model, with platform specific logic and tests produced by reusable skills. Analysts review every draft before it is merged. The company says an attempt in 2023 failed because the models invented MITRE techniques, and it publishes no figures for the current workflow.
No outcome disclosed.
Amazon
United States · Technology and software · 2025
AWS security engineers built RuleForge, a set of specialised agents that ingest and prioritise newly disclosed vulnerabilities, generate a detection rule from example exploit code, score each rule with a separate judge model for sensitivity and specificity, test it in stages and return failures to the generation agent with feedback. A human reviews and approves every rule before deployment. Amazon reports that over the last four months of the year the team produced and validated rules 336% faster than by hand, and that the judge model cut false positives by 67%.
- Productivity gain: 336%, rule creation and validation over the final four months of the year
"At AWS, we built RuleForge, an agentic-AI system that generates detection rules directly from examples of vulnerability-exploiting code, achieving a 336% productivity advantage over manual rule creation while maintaining the precision required for production security systems and enhanced customer security."
Claimed by: organization - False positive reduction: 67%, generated rules scored by a dedicated judge model, compared with the generation model scoring its own rules
"Using a dedicated judge model reduced false positives by 67% while maintaining the same number of true positive detections."
Claimed by: organization
SAP Enterprise Cloud Services
Germany · Technology and software · 2025
SAP Enterprise Cloud Services runs a private cloud of more than 200,000 virtual machines. It moved its threat detection from a legacy SIEM to Databricks and uses Anvilogic's AI to translate detections from SPL to SQL, generate modular detection logic tagged with MITRE ATT&CK metadata, recommend tuning changes from historical patterns and validate rules against its own data before deployment. Databricks reports that detection engineering time fell by 60 to 80% and that rules are created and deployed five to six times faster.
- Handling time reduction: at least 60%, detection engineering time after moving to Anvilogic and Databricks
"Detection engineering time has been reduced by 60-80%, freeing analysts to focus on higher-value investigative work."
Claimed by: vendor
How do you implement it?
A model agnostic playbook: what to prepare, the order to build in, and what goes wrong.
Data you need
- An inventory of existing detections with owners, status and MITRE ATT&CK mapping
- Historical log data or replayable traffic to measure how often a rule fires and on what
- Labelled examples of true attacks (exploit samples, red team and purple team results)
- A feed of threat reports and vulnerability disclosures
- Past alert dispositions, so the system can learn which rules produce false positives
Systems to integrate
- SIEM or data lake where rules run (for example Splunk, Microsoft Sentinel, Elastic or Databricks)
- Code repository and CI pipeline for detections as code
- Threat intelligence feed or platform
- Ticketing or case management, for alert feedback
- MITRE ATT&CK data and the organization's data source catalogue
Complexity: High
A detection pipeline needs a detection inventory with metadata, a test bed of historical logs or replayed attack traffic, and a peer review process that already treats rules as code. Without those, the AI has nothing to check its drafts against, and the review load moves to the engineers. Each target platform adds its own query language and its own tests.
- 1
Put the detections in a repository first
Store every rule as code with metadata (owner, ATT&CK technique, data sources, status), a test and a review step. Amazon describes the best drafted rule entering code review, just as before, and mnemonic says an analyst reviews each draft before it is merged, so the review pipeline has to exist before the AI joins it.
- 2
Build the test bed
Assemble historical logs and a set of known malicious samples. Every generated rule must report how often it fires on normal traffic and whether it catches the sample, before an engineer sees it.
- 3
Start with a narrow class of rules
Pick one platform and one kind of detection, such as rules for newly disclosed web vulnerabilities or translation of existing rules between two query languages. Compare drafts with rules your team wrote for the same inputs.
- 4
Separate writing from judging
Use a second model or a set of checks to score the output of the first, as Amazon does, and send failures back with the reason. A generator that grades its own work tends to approve it.
- 5
Measure engineer time and rule quality together
Track hours per rule and the false positive rate of rules after release. A team that writes more rules and receives more noise has moved the work to the analysts.
- 6
Widen to tuning and gap analysis
Once drafting is trusted, let the system propose tuning changes from alert outcomes and list coverage gaps against ATT&CK. Keep the engineer's approval on every change that reaches production.
Guardrails
- No generated rule is deployed without review and approval by a detection engineer
- Every draft is tested against historical logs and attack samples before review, with the results attached
- Exploit code and threat reports are treated as untrusted input to the model, and the agent has no deployment rights
- Generated rules carry their source, the model and the version that wrote them, in the repository history
- Rules that fire above an agreed rate in test are blocked automatically
KPIs to instrument
- Engineer hours per production rule, by rule type
- Share of drafted rules approved unchanged, edited and rejected
- False positive rate of released rules after 30 days
- Time from vulnerability disclosure or threat report to a deployed detection
- Share of prioritized vulnerabilities and ATT&CK techniques with a tested detection
Human in the loop
The engineer reviews the logic, the tests and the estimated noise, and decides whether a rule ships. Amazon states that human judgment remains the final gate before production deployment and describes the analyst's job as shifting from authoring to review. In the manual process every rule went to code review by another security engineer, and Amazon says the best RuleForge rule still enters code review, just as before.
Common failure modes
- A plausible rule that is wrong
- A generated rule can misread a report, over generalise a behaviour or look right and flood analysts with false alerts. mnemonic says this is why every output is treated as a draft that needs validation. Test against real traffic, and measure the rate after release.
- A rule that is too narrow
- A rule built from a single exploit sample catches that sample and misses variants. Ask the judge step to score sensitivity as well as specificity, and add variants to the test set.
- Reviewers become the bottleneck
- When drafting is cheap, review queues grow. Limit the number of open drafts per engineer and rank them by risk, so the reviewers see the best candidates first.
- Invented framework details
- Models can invent ATT&CK techniques or log fields. mnemonic reports that models invented MITRE techniques in its 2023 attempt. Validate identifiers and field names against the framework and the data catalogue in a script.
What are the risks and rules?
EU AI Act
Minimal risk
Generating detection rules for an organization's own cyber defence is not an Annex III use, and recital 55 of the AI Act says components used solely for cybersecurity purposes do not qualify as safety components. The tier stays minimal while engineers approve each rule and the rules do not themselves decide on people, for example by scoring employees.
Guidance
- MITRE ATT&CK (MITRE, Global). The shared catalogue of attacker techniques that detections are mapped to; use it as the reference to validate technique identifiers and to measure coverage.
- OWASP Top 10 for LLM Applications (OWASP, Global). Lists prompt injection and excessive agency among the main risks of LLM applications, relevant when a model reads exploit code and threat reports and writes code that will run in production.
- NIST AI Risk Management Framework (NIST, North America). A voluntary framework for mapping, measuring and managing the risks of an AI system, useful for documenting the review and test regime around generated rules.
Controls to put in place
- Detections stored as code with a named owner and a change history
- Mandatory peer review and approval before any generated rule is deployed
- Automated tests against historical logs and attack samples, with a release threshold for noise
- Validation of framework identifiers and field names by script
- Monthly review of released rules for false positive rate and missed attacks
When it went wrong elsewhere
- mnemonic's first attempt to extract structured data from threat reports (2023). The security services company reports that its 2023 experiment with language models on threat intelligence reports was not good enough for production, because the models invented MITRE techniques, could not take long reports and gave inconsistent answers.
Frequently asked questions
- How much faster can AI write detection rules?
Amazon reports that its RuleForge system gave a 336% productivity advantage over manual rule creation over four months, and Databricks reports that SAP Enterprise Cloud Services cut detection engineering time by 60% to 80%. Both are the organizations' or vendor's own measurements. Measure hours per production rule in your own team before you set a target.
- Can AI deploy detection rules by itself?
The teams on this page do not let it. At Amazon a human gives the final approval, and mnemonic's analysts review each draft and decide whether to merge it. A generated rule can look plausible and still flood analysts with false positives, so deployment stays with an engineer and your CI process.
- How do you stop AI written rules from creating more false alerts?
Test each rule against historical logs before review and set a noise threshold for release. Amazon uses a separate judge model that scores each rule, and reports that it reduced false positives by 67% with the same number of true positives. That figure measures how the judge screens candidate rules during evaluation, not alerts after release, so keep measuring the rate after release.
- Is this the same as security alert triage?
No. Triage works on alerts that rules have already produced, while detection engineering writes and tunes the rules that produce them. The two connect through feedback: triage outcomes show which rules are noisy and which attacks were missed.
How to cite this page
Blits.ai AI Use Case Library, "AI for detection engineering and detection rule writing", last verified 13 October 2026, https://www.blits.ai/ai-use-cases/detection-engineering-assistant. Licensed under CC BY 4.0. Method: how we verify use cases.
Changelog
- 13 October 2026: First published