Lab results

Every round the lab has run, including the ones that failed.

Northwatch is trained and scored in a lab. Its first detector is a program that watches the sensor readings of a recorded test plant and raises an alarm when the plant behaves abnormally. A separate testing program does the measuring. This page shows everything it has recorded.

Where these numbers come from
One public recording of a test plant,
What they do not prove
That Northwatch is ready for a real site
Lab record as of
UTC, the world standard clock time

The current best detector

Where the detector stands after rounds.

A round is one change to the detector, measured once. Of the rounds so far, were kept and were thrown out because they did not make the detector better.

Attacks caught

of

The recording has labeled attacks. The detector raised an alarm during this many of them and missed .

False alarms per day

A false alarm is an alarm when nothing was wrong. Too many and people stop listening.

Seconds from attack to alarm

median

Median means the middle value: half of the caught attacks were flagged sooner, half later.

Score

One number that rewards catching attacks and punishes false alarms. How it is calculated.

Start, best so far, and target

Measure Starting detector Best so far Target
Attacks caught of of at least of
False alarms per day under
Median seconds from attack to alarm none set
Score none set
The target

The result must also hold on a hidden second exam.

Best so far is round . Lab numbers on one public recording of a test plant (, license , ). They do not prove readiness for a real site.

Read this with the numbers

These are lab numbers. They do not prove that Northwatch is ready for a real site.

Northwatch is in development and has not been tested at a real site. Here is what the numbers on this page can and cannot tell you.

What they show

  • On one public recording of a test plant, the current detector raised an alarm during of labeled attacks, with false alarms per day.
  • The lab measures every change the same way and throws out the changes that do not help.
  • The kept changes were also checked against a hidden second exam, to guard against a detector that only memorized the first one.

What they do not show

  • That the detector would do as well at your site. Every site runs differently, and this is one recording of one test plant.
  • Anything about network traffic. This detector reads sensor readings only. Network protection is the first job of the product, and it has no lab numbers yet.
  • Anything about blocking or stopping an attack. The lab measures alarms. It has not yet measured how long containment takes.
  • A firm ranking between close results. The first exam is small, so one attack more or less moves the numbers a lot.

Round by round

Every round, kept or thrown out.

Each round tries one idea. If the score does not improve, the change is reverted: it is thrown out and the detector goes back to the version before. We show the reverted rounds because the failures are part of the record.

Score by round

A higher score is better. Solid squares are the starting point and the rounds that were kept. Hollow squares are rounds that were reverted. The same numbers are in the table below.
Every round the lab has run, oldest first
Round Verdict Idea tried Caught False alarms per day Median seconds to alarm Score

Every round the lab has run, oldest first. Score is the share of attacks caught minus 0.02 for every false alarm per day. Lab numbers on one public recording of a test plant (). They do not prove readiness for a real site.

The score

How the score is calculated.

Catching attacks is not enough. A detector that cries wolf all day gets switched off. So the score takes points away for every false alarm.

Score = share of attacks caught, minus 0.02 for every false alarm per day

The share of attacks caught is the number caught divided by the number of attacks. For the current best, round , that is divided by , minus 0.02 times , which gives .

A false alarm that drags on is counted again every 5 minutes. A detector cannot hide a long false alarm by calling it one event.

A perfect score is 1: every attack caught and no false alarms. A score can fall below zero when a detector raises a great many false alarms.

How the lab stays honest

The detector never grades its own work.

An automated loop improves the detector while nobody is watching. Four rules keep its results worth trusting.

  1. One change at a time

    Each round tries a single change to the detector, so the result says something clear about that one idea.

  2. An independent tester decides

    A separate testing program measures the change. It keeps the change only if the detector is really better. The loop cannot overrule it.

  3. A hidden second exam

    A second set of attacks is kept out of the loop's sight. Only the testing program runs it. It catches a detector that only memorized the first exam.

  4. Every round is recorded

    Kept or reverted, each round goes into the record. Every number on this site is read from that record. None is typed by hand.

Hidden second exam result for the current best detector, as written in the record: .

The data

Where the recording comes from.

Dataset, a set of recorded data
What it is
License
Source
Lab record as of
UTC

The dataset was made and published by its authors, not by Northwatch. We use it under its license and thank them for sharing it.

The training ladder

Five rungs to a product you can trust. We are on the first.

Industrial level means it works on real equipment at real sites, and outside experts have tested it. Everything on this page belongs to rung 1. We will not write as if a lab result proves more than its rung.

  1. Public data

    Train on free recordings of real test plants and networks with labeled attacks. It shows whether the method works at all.

    We are here

  2. Simulated site

    A plant and network built in software. It allows unlimited practice, including attacks no recording contains.

  3. Bench rig

    Real controllers, the small computers that run a line, set up on a table. Needs hardware.

  4. Real sites, listen only

    Pilot systems learn from real traffic without sending any commands. Needs pilot sites.

  5. Outside proof

    Independent testers attack it, and we work toward IEC 62443, the industrial security standard. Needs outside experts.

What comes next

Rungs 1 and 2 can be done in the lab. Rungs 3 to 5 need hardware, pilot sites and human experts. No lab result, however good, can stand in for them.

Pilot sites

Rung 4 needs real sites. Yours could be one of them.

Northwatch is looking for a small number of mid-sized factories and distribution centers. A pilot system would only listen and learn. It would send no commands to your equipment, and your data would stay in your building.