// methodology
check our arithmetic.
we publish a benchmark and we sell what it finds. that is a conflict of interest, so the method is on this page in full: what a verdict means, which of our own rows we flag as not comparable, the layers we do not score at all, and the endpoints that return every figure below to a caller with no account and no key.
// vocabulary
what a verdict means
one cell is one named detector's call on one tool. three words cover every cell, and they are the words the console prints, imported from the same module so the two surfaces cannot drift apart.
the grid is 29 tools by 16 detectors, 213 cells, folded from the per-attempt run ledger.
two rules sit under those words. below a minimum number of definitive trials we publish no verdict for a cell, and a thin cell is never counted as a loss. and ranking runs on the wilson lower bound of a rate rather than on the rate itself, so a tool that went one for one cannot sit at the top of the board.
// our own caveats
the flags we put on our own rows
a configuration is a stack, and one attempt credits every component in it. where that makes a rate hard to attribute we mark the row rather than let it read as clean. these three fields ship on every tool in GET /v1/matrix, for anyone, which is the point of computing them.
of the 19 tools with a measured record:
- 15 carry confoundedWith. a stack-mate rode nearly every attempt this tool made, so the two rates cannot be told apart. read the row as the stack, not the tool.
- 4 carry mixImbalanced. the tool sat a materially different set of targets from the rows beside it, so a head-to-head against those rows is not a like-for-like comparison.
- 1 carry neverVaried. the tool was in effectively every attempt the corpus holds, so its rate is the corpus rate and there is nothing to contrast it against.
a benchmark that flags its own comparisons is one you can act on. the alternative is a leaderboard whose rows look interchangeable and are not.
// the boundary
what we measure, and what we do not
detection happens in layers, and we score a layer only where the fleet has a reading for it. the count beside each layer is the number of tools we have measured there. a layer at zero is one we do not score at all, and drawing that line is what makes the rest of the page mean something.
everything inside that boundary is a reading from the run ledger. nothing outside it is scored, and we do not fill the gap with a claim from a vendor datasheet.
// freshness
two dates, because two things age
the model half and the measured half age independently, so they carry separate dates. blending them into one would let the staler half borrow the fresher one's credit, which is the whole trick a single "last updated" plays.
the freshest fingerprint capture behind the model half.
the freshest definitive attempt behind the verdict grid.
// verify
check the arithmetic
every figure on this page came from a request you can make too, with no account, no key and no cooperation from us. they are the same endpoints the page renders from, in the order the sections use them.
the roster, the measured record, and the flags on each row
the verdict grid and when it was last measured
when the model half was last captured
the rest of the surface is documented in the api reference, request and response shapes included.