AI needs evidence, not only fluency

A language model will now answer almost any question in any field, in good prose, in a second. Ask which of a patient's medicines a herbal preparation might interfere with and you get a list, a mechanism for each, and a reassuring tone. Ask what the Hindi press said about a hospital last week and you get a summary. An app that tracks attendance at a school takes a sentence to describe and a minute to build, and it runs.

Until recently, prose like that was evidence of work. Someone had read the papers, or the articles, or written the code, and the fluency of what they produced was a rough signal of the effort behind it. That signal is gone. Fluent output costs nothing now, and what it rests on is invisible from the outside.

The cost of finding out whether it is right has not moved. A pharmacologist still has to check the mechanism against the literature, and a journalist still has to find the original article. Someone still has to test the app. The gap between those two costs, cheap fluent answers and expensive checking, is the problem Scaling Truth Research Foundation was set up to work on.

Where a confident answer costs the most

We work where a wrong answer from a model is more than an inconvenience. Medicine first, because it is where we have done the most work.

We have spent the past months building a system to predict when a medicinal plant changes how a prescribed drug behaves in the body. Early on we trained a classifier on the published record of herb–drug interactions, about two thousand labelled records. One version also learned from a hundred thousand drug–drug interactions, and it looked like a clear win: it beat the version without them in all five folds. Then a control showed that a single number reproduced most of the gain: how many compounds are known in the plant, which mostly reflects how well studied it is.

So we looked harder at the record itself. Atorvastatin appears in it with 330 herbs, and the outcome for every one is "no effect". One of the studies behind those records gave rats quercetin, a compound found in hundreds of plants, together with atorvastatin. Its text never names a herb. The database recorded that one result against every plant it lists as containing quercetin. A model trained on that record can score well while knowing almost nothing about the plant in front of it.

What eventually caught the harmful cases was a public list. In 2006 the United States Food and Drug Administration listed nineteen drugs with a narrow therapeutic index, the ones where a small change in exposure hurts the patient. Flagging a pair as harmful whenever the drug was on that list did better on harmful cases than any model we trained. The trained model, tested on drugs it had not seen in training, found two of the thirty-five harmful records in its test set. A nineteen-row table that fits on a postcard, and that a clinician can check, beat three rounds of machine learning. We had been treating severity as something to learn. It was a missing column.

Good scores told us nothing about whether a model was reasoning about the right thing. Only checks designed to catch that did: drugs held out entirely, a control for how well studied a plant is, a table a clinician can read. The model we now build, Herb–Drug Interactions (truth-hdi), starts from mechanism, and its model card lists what has been measured and what has not. A longer essay sets out what the labels contain and what each model learned from them.

The news is next. Organisations and researchers increasingly read the press through AI summaries, because nobody can read hundreds of outlets in a dozen languages by hand. A summary that sounds right can still attribute a quote to the wrong paper, or merge two stories about two different hospitals into one. Both read as good English. Neither can be caught without going back to the article, which is the work the summary was meant to save.

Then software. Tools that build an application from a plain description let a school or a clinic make what it needs without a programmer. The app runs and the forms submit. Whether every user can see every other user's records is not visible on the screen, and the person who asked for the app cannot read the code to find out.

What a checkable answer looks like

The output has to change shape. Three properties make an answer checkable, and we build for all of them.

It points to its sources. A prediction about a herb and a drug names the compound, the enzyme and the published study. A generated app comes with the tests that show it does what was asked, written in the same plain language as the request.

It says how sure it is, and stops when it cannot. A model that gives a verdict for every pair is guessing on most of them. Ours ranks and explains, abstains where the evidence is thin, and we would rather it abstained too often than invented a mechanism.

The evaluation behind it is public and can be run again. A number from an internal test is a claim until someone outside the lab has reproduced it. We write the pass criteria before a model is fitted, report against controls, and publish what failed. Four candidate features for truth-hdi failed the check that they were measuring pharmacology rather than how well studied a plant is. They were dropped, and the reason is in the model card.

A checkable model can still be wrong. The point is that a pharmacologist, a journalist or a school can find out when it is.

What to hold us to

Our charter and AI principles say what we will and will not do: no number without an evaluation behind it, internal results labelled as internal, failures published alongside successes, data we have the right to use, and a public log of what goes wrong. The release policy says what we publish, what we hold back, and why.

Those are only worth something if people use them. If a model of ours gives an answer it cannot support, or a claim on this site has no evaluation behind it, write to connect@scalingtruth.org and we will correct it in the incident and change log.