Responsibility
How we evaluate
A number counts when it has an evaluation behind it, and becomes a claim when someone outside the lab can reproduce it.
The rules
Pre-registered
Each learned component states its pass criteria before it is fitted. We publish when it fails.
Held out properly
Test data shares nothing with training data that would make the test easier, such as the same molecule, source clip or near-duplicate article.
Against a control
Results are reported against a control, such as a permutation or a simple baseline, not only against chance.
Labelled honestly
Results are marked internal until the evaluation has been run outside the lab or its data and code are public.
Reported by group
Where performance can differ by language or group, we report it separately.
Protocols by area
| Area | Protocol | Design | Status |
|---|---|---|---|
| Life sciences | Herb–drug interaction evaluation | Drugs held out entirely; known interactions tested out of sample; permutation control; pre-registered feature gates. | Protocol to be published Q4 2026 |
| Realtime video | Temporal consistency | Frame-to-frame feature distance, object persistence and paired human preference, each with what it misses; latency on stated hardware. | In preparation |
| Media intelligence | News attribution | Missed mentions, wrong tone and invented attribution, measured separately and by language. | In preparation |
| Software from intent | Intent to app | Plain-language requests paired with acceptance tests; data-handling and access-control checks. | In preparation |
Temporal consistency metrics
| Metric | Measures | Misses |
|---|---|---|
| Frame-to-frame feature distance | Drift between consecutive frames in a fixed feature space | Slow drift that stays small per step |
| Object persistence | Whether tracked objects survive across the clip | Objects that persist but deform |
| Paired human preference | Which of two clips people judge more stable | Anything people do not notice |
Negative results
We publish what did not work. In Herb–Drug Interactions, four candidate features failed the check that they were measuring pharmacology rather than how well studied a plant is. They were not shipped, and the reason is in the model card. A lab that only reports what worked is asking to be trusted rather than checked.