Think-Reason-Learn Tests Three New Models and Keeps Jev
Think-Reason-Learn will keep TypeSafe AI's Jev to answer the questions behind its predictions. On twelve datasets, OpenAI's GPT-6 Luna Decisions was about as accurate at almost five times the cost, while Cloudflare's Clef Flash and Microsoft-Decision-1 were cheaper but less accurate.
Get Think-Reason-Learn on GitHubThink-Reason-Learn, an open-source library from Vela Research and the University of Oxford, will keep TypeSafe AI's Jev as its answering model, the model that answers the questions behind each prediction. Jev is a decision model, which answers a yes-or-no question about a passage with a probability instead of writing text. OpenRouter, a service that gives one point of access to models from many AI companies, now offers several other decision models.2 We tested three of them, OpenAI's GPT-6 Luna Decisions, Cloudflare's Clef Flash and Microsoft-Decision-1 from Microsoft,3–5 and none of them offered Jev's accuracy at a similar cost.
Luna was about as accurate as Jev but would cost almost five times as much to run. Clef Flash was 28% cheaper than Jev and Microsoft-Decision-1 about 16% cheaper, but both were less accurate. In our previous post,1 switching from Google's Gemini 2.5 Flash to Jev made Think-Reason-Learn up to 17 times cheaper to run. Gemini 2.5 Flash appears here as the earlier baseline. The questions stay the same whichever model answers them, so Think-Reason-Learn can try each new answering model on a sample of cases for a few dollars and switch when one does better.
What does the answering model do in Think-Reason-Learn?
In Think-Reason-Learn,6 the answering model answers the questions the library uses to predict an outcome from text. On VCBench, the first benchmark for venture capital, for example, the library predicts from a founder's profile whether their company will reach a $500 million exit or funding round. In its Random Rule Forest method,7 a large language model writes yes-or-no questions about the text, such as whether the founder has worked as a professor or researcher in a related field. In its Policy Induction method,8 the large language model writes short policies of the form "founders like this tend to succeed". These questions and policies are written once for each dataset, but the answering model must answer all of them for every case, so it accounts for most of the running cost. A simple statistical model then turns the answers into a prediction.
How do the new models compare with Jev?
Luna was about as accurate as Jev on Random Rule Forest and slightly ahead on Policy Induction, although that small lead is not settled, for reasons set out below and in the FAQ. Clef Flash and Microsoft-Decision-1 were less accurate than Jev on both (Figure 1). As in our previous post, we tested on twelve datasets that pair a passage of text with a known outcome, about 53,000 cases in all, from medicine and law to finance and film.9 Every model answered exactly the same questions and policies, which Gemini 2.5 Flash wrote. Scores for Luna and Microsoft-Decision-1 are estimated from a sample of the cases, to keep costs down.
Four of the twelve datasets
| Dataset and text | Predict, with one of its questions | Cases | How often |
|---|---|---|---|
| Clinical trialsTrial registration | The drug passes the trial (Phases I, II and III)“Is the trial for a rare disease or a specific subtype of a common disease?” | 12,353 | 56%, 50%, 68% |
| Court casesFacts of a European Court of Human Rights case | The court finds at least one violation“Was there a delay in the proceedings that was attributable to the state?” | 2,194 | 84% |
| MoviesPlot summary | The film receives an award nomination“Does the plot involve a hero challenging a corrupt or oppressive system?” | 2,149 | 19% |
| arXiv papersTitle and abstract | Accepted at the venue it was submitted to“Does the paper propose a new algorithm or method?” | 5,000 | 36% |
Clinical trials count as three datasets, one for each phase. The last column shows how often the outcome happens.
Clef Flash and Microsoft-Decision-1 cost the least to run and Gemini 2.5 Flash the most (Figure 2), with the estimated costs explained in the FAQ.
Scores for the datasets shown
| Dataset | Jev | Gemini | Clef | Luna − Jev | Microsoft − Jev |
|---|---|---|---|---|---|
| Clinical I | 60.0 | 56.6 | 59.5 | −1.3 | −4.8 * |
| Clinical II | 66.0 | 61.1 | 66.9 * | −0.2 | −1.0 |
| Clinical III | 69.8 | 60.6 | 68.5 * | −3.8 * | −2.5 |
| Court cases | 71.8 | 59.6 | 69.2 * | +0.4 † | −8.4 * |
| Movies | 74.8 | 71.7 | 70.7 * | +4.1 * | +1.0 |
| arXiv papers | 76.1 | 67.5 | 75.2 | +0.3 | −1.8 |
| Mean, all twelve | 68.6 | 63.9 | 67.0 * | +0.4 | −1.9 * |
| Dataset | Jev | Gemini | Clef | Luna − Jev | Microsoft − Jev |
|---|---|---|---|---|---|
| Clinical I | 60.3 | 57.0 | 57.4 * | −4.3 | −0.2 |
| Clinical II | 67.5 | 67.3 | 65.6 * | +1.8 | −2.2 |
| Clinical III | 71.3 | 72.4 | 73.2 * | +0.8 | −6.7 * |
| Court cases | 70.0 | 63.0 | 69.0 | +10.4 * † | −0.9 |
| Movies | 82.3 | 76.9 | 76.8 * | +4.0 * | −8.0 * |
| arXiv papers | 81.4 | 78.7 | 76.7 * | +0.6 † | −1.4 |
| Mean, all twelve | 69.5 | 68.3 | 66.8 * | +2.0 * | −3.4 * |
AUC × 100 on cases held back from training, the share of success and failure pairs ranked correctly. Jev, Gemini 2.5 Flash and Clef Flash were scored on every held-back case, and Luna and Microsoft-Decision-1 against Jev on the sampled cases each answered. * marks a gap from Jev whose 95% paired bootstrap interval excludes zero, not corrected for the twelve comparisons. † Luna declined a third or more of these cases, so its gap covers only part of the dataset. Results for every dataset will appear in a forthcoming academic paper on Think-Reason-Learn.
Which cases did Luna decline?
OpenAI's service declined to answer 6% of the cases sent to Luna and gave no reason, while the other models answered every case. Most declined cases were court cases and arXiv papers, and most declined questions ask for a legal, medical or peer-review verdict about a real, identifiable case, such as whether a delay in court proceedings was the state's fault. Luna's declined cases were left out for every model in its comparison, which may bias its result either way (see the FAQ).
Which answering model should you use with Think-Reason-Learn?
Jev remains the default answering model in Think-Reason-Learn, including for its entries on VCBench.10 Luna could be worth testing where a possible small gain in accuracy on Policy Induction is worth the extra cost, as long as the cases do not involve the legal, medical or peer-review verdicts it tends to decline. Clef Flash could be worth testing for very large runs where a small loss in accuracy is acceptable, and as an open-source model it can also be run on your own hardware. Microsoft-Decision-1 answered every case and read long court judgments in full, so it could be worth testing on long texts.
OpenRouter also lists decision models from Perplexity, Liquid AI and Upstage. Researchers and VC teams can test these or other answering models with Think-Reason-Learn and email their results and costs to benchmark@vela.partners.
Frequently asked questions
Why were Luna and Microsoft-Decision-1 tested on a sample?
Testing on a sample kept costs down, since a full run on Luna would cost about $27. Luna and Microsoft-Decision-1 answered the same random sample of up to 500 training and 500 test cases from each dataset, about a fifth of all cases, for $5.33 and $1.02. Jev, Gemini 2.5 Flash and Clef Flash were scored on exactly the same cases. Repeating the same sampling on the other models' saved answers put the gap to Jev within about one point of the full run in nine tries out of ten, although the cases Luna declined add some uncertainty.
What is a decision model?
A model that answers a yes-or-no question about a passage with a probability instead of text. TypeSafe AI's Jev is one. OpenRouter serves others through its Decisions interface, which is how we ran GPT-6 Luna Decisions, Clef Flash and Microsoft-Decision-1. Clef Flash is the faster, smaller and cheaper of Cloudflare's two open-source Clef models.
Why compare with Gemini 2.5 Flash and not a frontier model?
Gemini 2.5 Flash answered the questions in Think-Reason-Learn's original analysis, so it is the model Jev replaced. A frontier model could answer them too, but our previous post estimated that one Random Rule Forest run over the twelve datasets would cost roughly $350 to $900 on Anthropic's Claude Opus 5.5 or OpenAI's GPT-6 Astra.
Is Luna's lead on Policy Induction reliable?
Luna's lead is small and not yet settled. Luna was more likely to decline some outcomes than others, such as accepted papers rather than rejected ones, so leaving those cases out may make its accuracy look better or worse than it really is. Counted in other ways, the declined cases shrink the lead or turn it into a small deficit, and the lead is no longer clear without the datasets where Luna declined the most cases and those where models may remember the outcomes, such as movies.
How were the costs worked out?
Jev costs $0.042 per million tokens of text and Clef Flash $0.021 through Prime Intellect, one of the companies that host it, and the costs shown for both are what the full runs cost. The estimates for Luna and Microsoft-Decision-1 apply their prices of $0.10 and $0.042 per million tokens to token counts projected from the sample. Each model counts the tokens in the same text differently, so the costs do not follow the prices exactly. Gemini 2.5 Flash's estimate follows the previous post's method. For both methods together it comes to about 13 times Jev's cost, against 17 times for Random Rule Forest alone. The previous post's Policy Induction cost also covered 4,500 VCBench founders, which this post leaves out.
Does it matter which company hosts Clef Flash?
The host matters only for very long texts. Prime Intellect's service stops reading after about 16,000 tokens without returning an error, while Cloudflare's own service reads up to about 66,000 at over four times the price. Every case in the twelve datasets was well under the shorter limit.
Is the comparison fair to each model?
All five models answered exactly the same questions and policies, which Gemini 2.5 Flash wrote, and the statistical model was refitted to each model's answers in the same way. If anything, questions written by Gemini should suit Gemini's answers. Vela Research has no investment in or partnership with TypeSafe AI.
References and notes
- Ben Griffin. Switching to Jev Made Think-Reason-Learn Up to 17× Cheaper. Vela Research, October 2026.
- OpenRouter, a service that gives one point of access to models from many AI companies. Its Decisions interface was in alpha in October 2026.
- OpenAI. GPT-6 Luna Decisions, released on OpenRouter on 6 October 2026 as GPT-6 Luna served through OpenAI's Decisions API.
- Cloudflare. Clef Flash, released on OpenRouter on 1 October 2026 as the fast member of Cloudflare's open-source Clef family of decision models.
- Microsoft. Microsoft-Decision-1, a decision model listed on OpenRouter in October 2026 and served through Microsoft Azure.
- Think-Reason-Learn: github.com/Vela-Research/think-reason-learn and thinkreasonlearn.com.
- Random Rule Forest (RRF): Interpretable and Manageable Ensembles of LLM-Generated Questions for Predicting Success from Unstructured Data. arXiv:2505.24622.
- Policy Induction: Predicting Startup Success via Explainable Memory-Augmented In-Context Learning. IEEE.
- Sources for the datasets shown: clinical-trial outcomes from Fu et al., HINT: Hierarchical Interaction Network for Clinical-Trial-Outcome Predictions (Patterns, 2022); court cases from Chalkidis, Androutsopoulos and Aletras, Neural Legal Judgment Prediction in English (ACL 2019); films from the Movie-O-Label dataset; and papers from Kang et al., A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications (NAACL 2018).
- Chen, Ternasky, Kwesi, Griffin, Yin, Salifu, Amoaba, Mu, Alican and Ihlamur. VCBench: Benchmarking LLMs in Venture Capital. In Intelligent Computing (Computing Conference 2026), Lecture Notes in Networks and Systems, vol. 1949, pp. 167–187. Springer, 2026. Preprint arXiv:2509.14448. The live leaderboard is at vcbench.com.