Lab results
Every round the lab has run, including the ones that failed.
Northwatch is trained and scored in a lab. Its first detector is a program that watches the sensor readings of a recorded test plant and raises an alarm when the plant behaves abnormally. A separate testing program does the measuring. This page shows everything it has recorded.
- Where these numbers come from
- One public recording of a test plant,
- What they do not prove
- That Northwatch is ready for a real site
- Lab record as of
- UTC, the world standard clock time
The current best detector
Where the detector stands after rounds.
A round is one change to the detector, measured once. Of the rounds so far, were kept and were thrown out because they did not make the detector better.
Attacks caught
of
The recording has labeled attacks. The detector raised an alarm during this many of them and missed .
False alarms per day
A false alarm is an alarm when nothing was wrong. Too many and people stop listening.
Seconds from attack to alarm
median
Median means the middle value: half of the caught attacks were flagged sooner, half later.
The lab numbers could not be loaded. Nothing is wrong with your device. Please reload the page, or try again in a moment.
Start, best so far, and target
| Measure | Starting detector | Best so far | Target |
|---|---|---|---|
| Attacks caught | of | of | at least of |
| False alarms per day | under | ||
| Median seconds from attack to alarm | none set | ||
| Score | none set |
The result must also hold on a hidden second exam.
The lab has not reached this target yet.
The lab has reached this target on its first exam. That is still a lab result on one recording.
Read this with the numbers
These are lab numbers. They do not prove that Northwatch is ready for a real site.
Northwatch is in development and has not been tested at a real site. Here is what the numbers on this page can and cannot tell you.
What they show
- On one public recording of a test plant, the current detector raised an alarm during of labeled attacks, with false alarms per day.
- The lab measures every change the same way and throws out the changes that do not help.
- The kept changes were also checked against a hidden second exam, to guard against a detector that only memorized the first one.
What they do not show
- That the detector would do as well at your site. Every site runs differently, and this is one recording of one test plant.
- Anything about network traffic. This detector reads sensor readings only. Network protection is the first job of the product, and it has no lab numbers yet.
- Anything about blocking or stopping an attack. The lab measures alarms. It has not yet measured how long containment takes.
- A firm ranking between close results. The first exam is small, so one attack more or less moves the numbers a lot.
Round by round
Every round, kept or thrown out.
Each round tries one idea. If the score does not improve, the change is reverted: it is thrown out and the detector goes back to the version before. We show the reverted rounds because the failures are part of the record.
Score by round
| Round | Verdict | Idea tried | Caught | False alarms per day | Median seconds to alarm | Score |
|---|
Every round the lab has run, oldest first. Score is the share of attacks caught minus 0.02 for every false alarm per day. Lab numbers on one public recording of a test plant (). They do not prove readiness for a real site.
The round by round record could not be loaded. Please reload the page.
The score
How the score is calculated.
Catching attacks is not enough. A detector that cries wolf all day gets switched off. So the score takes points away for every false alarm.
Score = share of attacks caught, minus 0.02 for every false alarm per day
The share of attacks caught is the number caught divided by the number of attacks. For the current best, round , that is divided by , minus 0.02 times , which gives .
A false alarm that drags on is counted again every 5 minutes. A detector cannot hide a long false alarm by calling it one event.
A perfect score is 1: every attack caught and no false alarms. A score can fall below zero when a detector raises a great many false alarms.
How the lab stays honest
The detector never grades its own work.
An automated loop improves the detector while nobody is watching. Four rules keep its results worth trusting.
-
One change at a time
Each round tries a single change to the detector, so the result says something clear about that one idea.
-
An independent tester decides
A separate testing program measures the change. It keeps the change only if the detector is really better. The loop cannot overrule it.
-
A hidden second exam
A second set of attacks is kept out of the loop's sight. Only the testing program runs it. It catches a detector that only memorized the first exam.
-
Every round is recorded
Kept or reverted, each round goes into the record. Every number on this site is read from that record. None is typed by hand.
Hidden second exam result for the current best detector, as written in the record: .
The training ladder
Five rungs to a product you can trust. We are on the first.
Industrial level means it works on real equipment at real sites, and outside experts have tested it. Everything on this page belongs to rung 1. We will not write as if a lab result proves more than its rung.
-
Public data
Train on free recordings of real test plants and networks with labeled attacks. It shows whether the method works at all.
We are here
-
Simulated site
A plant and network built in software. It allows unlimited practice, including attacks no recording contains.
-
Bench rig
Real controllers, the small computers that run a line, set up on a table. Needs hardware.
-
Real sites, listen only
Pilot systems learn from real traffic without sending any commands. Needs pilot sites.
-
Outside proof
Independent testers attack it, and we work toward IEC 62443, the industrial security standard. Needs outside experts.
Rungs 1 and 2 can be done in the lab. Rungs 3 to 5 need hardware, pilot sites and human experts. No lab result, however good, can stand in for them.
Pilot sites
Rung 4 needs real sites. Yours could be one of them.
Northwatch is looking for a small number of mid-sized factories and distribution centers. A pilot system would only listen and learn. It would send no commands to your equipment, and your data would stay in your building.