Detection Engineering Is a Software Discipline

Detection · June 12, 2026 · 8 min read

Somewhere in most security programmes there is a detection nobody trusts. It fires occasionally. When it fires, an analyst closes it. Nobody remembers who wrote it, what it was meant to catch, or whether it still catches it. It stays because deleting things feels riskier than leaving them.

That artefact is the natural end state of treating detections as configuration rather than as code. And detections are unambiguously code: logic that runs continuously in production, against adversarial input, where a silent failure means an attack proceeds unobserved. Every property that makes software engineering hard applies, and most of the practices that make it tractable are absent.

The failure mode is silence

A broken web service tells you it is broken. Requests fail, dashboards go red, somebody gets paged.

A broken detection does nothing at all. It sits in the console looking exactly like a working one. The log source it depended on changed its field names three months ago; the rule has matched nothing since, and matching nothing is indistinguishable from nothing bad happened. The two states produce identical output.

This is the central problem, and it drives everything else. You cannot rely on absence of alerts as evidence of health, so health has to be established some other way — which means testing.

What testing a detection actually means

The useful question is not “does the rule parse”. It is “given this input, does this rule fire, and given this other input, does it stay quiet”.

That implies two corpora. A set of positive samples — real or synthesised telemetry representing the behaviour you intend to catch — and a set of negative samples drawn from ordinary production activity that must not trigger. Both need to live alongside the rule, in the same repository, and both need to run automatically when anything changes.

Building the positive corpus is where most teams stall, because capturing attacker behaviour sounds like it requires a red team. Often it does not. Much of it can be produced by running the technique yourself in a lab, or by keeping the telemetry from previous incidents. An incident you have already suffered is the highest-value test fixture you will ever get, and most organisations discard it once the ticket closes.

Detections drift because environments move

Even a well-written rule decays. Not because anyone edited it, but because everything around it changed.

An agent upgrade renames a field. A cloud provider adjusts a log schema. A team migrates a service to a new platform and its telemetry starts arriving through a different pipeline with different normalisation. The rule is untouched and now wrong.

The mitigation is unglamorous: assert on your inputs, not only your outputs. Monitor whether each log source is still arriving, at roughly the expected volume, with the fields your rules reference actually populated. A pipeline health check that notices a source went quiet at 3am is worth more than several additional detections, because it catches the class of failure that hides all the others.

Version control and review are not bureaucracy

When a detection lives in a console, its history is whatever the audit log retained and its rationale is whatever the author remembered to type into a description box. When it lives in a repository, you get the change history, the reason for each change, and the ability to answer what did we look like in March — which is precisely the question that comes up during an investigation of something that started in March.

Review matters for a less obvious reason. The author of a detection is the person least able to judge its false-positive rate, because they wrote it against the behaviour they were imagining rather than against the behaviour your environment actually produces. A reviewer who knows the environment will spot that the “unusual” process being flagged is deployed by a build system on every host, nightly.

Coverage is a map, not a number

Frameworks that enumerate attacker techniques are useful for orientation, and they are routinely misused as scorecards. A count of techniques “covered” says nothing about whether the detections work, whether the required telemetry is being collected everywhere or only in the two subnets someone instrumented first, or whether the rule catches the technique in general or one specific tool that implements it.

A more honest artefact is a map with confidence attached: this technique, in this part of the estate, detected by this rule, last validated on this date, with these known gaps. It is less satisfying than a percentage and considerably more useful when deciding what to build next.

What to change first

If a programme does one thing, it should be to move detections into version control with a test corpus attached — even a small one, even manually constructed. Everything else follows from that: review has something to review, CI has something to run, and a change that silently breaks a rule starts failing loudly at the moment it is introduced rather than during the incident it was supposed to catch.

The second thing is to instrument the inputs. Rules that match nothing because their data stopped arriving are the most expensive kind of failure, because they cost you both the detection and the false confidence that you had one.