Model Benchmarks
Our models against the field, benchmark by benchmark: the competitors matrix first, then every system side by side on each benchmark, then where to get the models. Built to let AI developers save 30 to 90 percent per token or API call. Every number links to the evidence file and score script that re-derives it.
Competitors matrix
Every system that appears in our evidence, one row each, against every benchmark we publish. Our models are the first rows; the best number in each column is in accent. Where a model has several conditions on one benchmark the row shows its headline condition and says which. Blank means that system was not measured on that benchmark.
Our models hold the top number in 3 of 6 benchmark columns.
| System | BANKING77 accuracy, examples in the requestDecide Zero 1.1 · higher is better | BANKING77 accuracy, trainedTuatara Decide (trained) · higher is better | CNN/DailyMail ROUGE-1Tuatara Summarize 1.0 · higher is better | XSum ROUGE-1Tuatara Summarize 1.0 · higher is better | 20NG coherence (C_v)Tuatara Topics 1.0 · higher is better | 20NG NMITuatara Topics 1.0 · higher is better |
|---|---|---|---|---|---|---|
| Decide Zero 1.1 | 82.11%1 example per option | |||||
| Tuatara Decide (trained) | 94.09% | |||||
| Tuatara Summarize 1.0 | 39.46 | 17.42 | ||||
| Tuatara Topics 1.0 | 0.616 | 0.445 | ||||
| Jev (published) | 81.00%zero-shot | 92.40% | ||||
| Lead-k | 40.04 | 16.55 | ||||
| LexRank | 34.10 | 18.48 | ||||
| Luhn | 32.68 | 18.35 | ||||
| LSA | 30.49 | 16.02 | ||||
| TextRank | 29.77 | 18.29 | ||||
| FASTopic (measured) | 0.597 | 0.531 | ||||
| FASTopic (published) | 0.426 | 0.525 | ||||
| LDA (measured) | 0.597 | 0.442 | ||||
| LDA (published) | 0.371 | 0.451 |
Where to get them
Direct links to every directory, marketplace, catalog and router that lists these models today. The same models, billed by the venue you call through; or take a key from us and call the endpoint directly.
API marketplaces
- RapidAPI: Tuatara Summarize · Decide Zero · Tuatara Decide · tuatara topics
- api.market: Tuatara Summarize · tuatara topics
MCP catalogs
- Official MCP Registry: registry search: tuatara
- Glama: Tuatara Summarize
- mcpservers.org: Tuatara Summarize
Direct from Cymetica
Decide Zero 1.1live
decide-zero-1.1 · text · built 2026-09-29
Typed decisions with calibrated probabilities. Classifies text into one of your labels and says how sure it is. Listed on the "a few examples per option" condition - one example per option in the request scores 82.11% on BANKING77 (3 per option 85.13%, 5 per option 85.71%) with a calibration error of 0.017; with option names and descriptions only (zero-shot) it scores 77.18%, under the 81.0% bar, so send examples.
Side by side: BANKING77 accuracy, options and examples in the request, no training
ours beats 1 of 1 other system · higher is better
BANKING77 test, 3,080 messages, 77 intents
| Condition | Accuracy | Calibration error (ECE, 10 bins) |
|---|---|---|
| Published zero-shot bar (the leading hosted classifier, no examples) | 81.00% | 0.123 |
| Decide Zero, zero-shot (option names and descriptions only) | 77.18% | 0.023 |
| Decide Zero, one example per option (listing condition) | 82.11% | 0.017 |
| Decide Zero, three examples per option | 85.13% | 0.013 |
| Decide Zero, five examples per option | 85.71% | 0.015 |
Lower calibration error is better. Decide Zero needs no training: the options and examples travel in the request.
Tuatara Decide (trained)live
tuatara-decide · decisions · built 2026-11-02
A classifier trained on your own labelled examples in seconds, served on its own API and MCP server. The trained model beats the leading hosted classifier on the standard intent benchmark.
Side by side: BANKING77 accuracy, trained on the official train split
ours beats 1 of 1 other system · higher is better
BANKING77 test, 3,080 messages, 77 intents, scored row by row against the published answers
| System | Accuracy | Right where the comparator is wrong | Wrong where the comparator is right | Exact McNemar p |
|---|---|---|---|---|
| Jev jev-1.13.0 (published answers) | 92.40% | — | — | — |
| Tuatara Decide, trained (vector blend + embedding head) | 94.09% | 117 | 65 | 0.000 |
| Tuatara vector + TF-IDF blend alone | 91.79% | 121 | 140 | 0.270 |
Re-derive every number with the public score script: it downloads the official labels and both answer files.
Four more intent and sentiment sets, trained on each official train split
| Dataset | Tuatara Decide (vector blend) | Linear head alone | Tuatara vectors alone | Blend vs head, exact McNemar p |
|---|---|---|---|---|
| CLINC150 (in-scope), 4500 test items, 150 classes | 92.56% | 92.56% | 80.40% | 1.00 |
| HWU64, 1076 test items, 64 classes | 87.73% | 87.17% | 81.51% | 0.031 |
| AG News, 7600 test items, 4 classes | 91.13% | 90.99% | 87.24% | 0.161 |
| SST-2, 1821 test items, 2 classes | 84.40% | 82.59% | 80.23% | 0.006 |
The blend's gain over the linear head alone is what the Tuatara vectors add.
Tuatara Summarize 1.0live
tuatara-summarize-1.0 · text · built 2026-10-28
Extractive summarization of a document. Returns the document's own sentences, chosen by a vector-association model of the text, cut to a sentence count, a share of the document or a token budget, so an expensive model reads the extract instead of the whole document. Deterministic, nothing invented.
Side by side: CNN/DailyMail 3.0.0 test, 3 sentences: ROUGE-1, same harness
ours beats 4 of 5 other systems · higher is better
Side by side: XSum test, 1 sentence: ROUGE-1, same harness
ours beats 2 of 5 other systems · higher is better
CNN/DailyMail 3.0.0 test, 3 sentences, 1000 documents, same harness for every system
| System | ROUGE-1 | ROUGE-2 | ROUGE-Lsum | ms per document |
|---|---|---|---|---|
| Lead-k | 40.04 | 17.67 | 36.47 | 47.70 |
| TuataraSum tuned-best | 39.46 | 17.34 | 35.88 | 798.40 |
| TuataraSum tuned | 38.56 | 16.63 | 35.03 | 254.60 |
| TuataraSum faithful (2015 CGI) | 37.02 | 15.09 | 31.13 | 468.70 |
| TuataraSum GS1 | 36.01 | 15.04 | 32.64 | 201.80 |
| LexRank (sumy) | 34.10 | 12.09 | 30.44 | 378.70 |
| TuataraSum fusion | 33.74 | 13.07 | 30.36 | 858.80 |
| Luhn (sumy) | 32.68 | 12.35 | 29.24 | 321.30 |
| TuataraSum GS2 | 30.66 | 10.53 | 27.51 | 3072.50 |
| LSA (sumy) | 30.49 | 9.43 | 27.16 | 1195.90 |
| TextRank (sumy) | 29.77 | 10.01 | 26.50 | 461.00 |
XSum test, 1 sentence, 1000 documents, same harness for every system
| System | ROUGE-1 | ROUGE-2 | ROUGE-Lsum | ms per document |
|---|---|---|---|---|
| TuataraSum GS1 | 18.93 | 2.65 | 13.36 | 113.70 |
| TuataraSum fusion | 18.72 | 2.65 | 13.34 | 369.80 |
| TuataraSum faithful (2015 CGI) | 18.55 | 2.39 | 12.90 | 239.30 |
| LexRank (sumy) | 18.48 | 2.52 | 13.13 | 285.60 |
| TuataraSum tuned | 18.39 | 2.30 | 13.27 | 163.80 |
| Luhn (sumy) | 18.35 | 2.46 | 13.00 | 238.60 |
| TextRank (sumy) | 18.29 | 2.47 | 13.05 | 314.60 |
| TuataraSum GS2 | 17.91 | 2.34 | 12.87 | 1391.80 |
| TuataraSum tuned-best | 17.42 | 1.79 | 12.56 | 330.50 |
| Lead-k | 16.55 | 1.46 | 12.13 | 20.30 |
| LSA (sumy) | 16.02 | 1.67 | 11.31 | 612.80 |
Tuatara Topics 1.0
tuatara-topics-1.0 · text · built 2026-09-29 · evidence re-measured 2026-11-03
Topic discovery for a set of documents. Returns coherent, distinct topics and the documents that belong to each.
Side by side: 20 Newsgroups, K=50: topic coherence (C_v)
ours beats 4 of 4 other systems · higher is better
Side by side: 20 Newsgroups, K=50: NMI
ours beats 4 of 4 other systems · higher is better
20 Newsgroups, K=50 topics
| System | Coherence (C_v) | Topic diversity | Purity | NMI |
|---|---|---|---|---|
| FASTopic (measured, 5 runs) | 0.597 | 0.992 | 0.594 | 0.531 |
| FASTopic (published) | 0.426 | 0.983 | 0.577 | 0.525 |
| LDA (ours; published: LDA-Mallet) (measured, 5 runs) | 0.597 | 0.683 | 0.523 | 0.442 |
| LDA (ours; published: LDA-Mallet) (published) | 0.371 | 0.763 | 0.514 | 0.451 |
| Tuatara Topics 1.0 (served configuration) (measured, 5 runs) | 0.616 | 0.780 | 0.546 | 0.445 |
| Tuatara Topics (research configuration: runs=9, embed=0.6, iters=1000, 2026-09-29) (measured, 5 runs) | 0.643 | 1.000 | 0.646 | 0.579 |
Higher is better on every column.
FTA BTC-divergence short selectorinternal
internal/fta-divergence · text · built 2026-08-01
Internal strategy selector. Not offered externally.
Internal strategy component, not offered externally, no public benchmark.
Evolving daily
Every tunable setting of every model is a node in an optimization network. Each day the network scores one step up and one step down on each node against tuning data (never a published test split), applies the single best gain, measures yesterday's change on fresh documents and reverts a loss. Pricing nodes keep every rate 30 to 90 percent under the cheapest comparable hosted API. Ledger: /api/v1/models/optimizations.
| Day | Model | Setting | From | To | Tuning gain | Next-day check | Outcome |
|---|---|---|---|---|---|---|---|
| 2026-11-04 | tuatara-decide | embed_head.C | 10.0 | 7.5 | +0.0030 | pending | applied |
| 2026-11-04 | tuatara-topics-1.0 | unique | 0.2 | 0.4 | +0.0474 | pending | applied |
| 2026-11-04 | tuatara-summarize-1.0 | tuned.length_norm | 1.0 | 1.5 | +0.0016 | pending | applied |
| 2026-11-03 | tuatara-decide | embed_default_weight | 1.0 | 0.9 | +0.0030 | -0.0030 | reverted |
| 2026-11-03 | tuatara-topics-1.0 | unique | 0.1 | 0.2 | +0.0458 | +0.0504 | applied |
| 2026-11-03 | tuatara-summarize-1.0 | tuned-best.length_norm | 1.0 | 0.75 | +0.0024 | +0.0008 | applied |
| 2026-11-02 | tuatara-topics-1.0 | unique | 0.0 | 0.1 | +0.0784 | +0.0536 | applied |
Vector models (Tuatara Embed Stocks, Embed Crypto, Embed Words) are not benchmarked on this page for now: their embedding implementation is being reworked, and the comparison returns when it is.
How to read this page
- Matrix and charts: every system's number on a benchmark comes from the same harness and the same test split, run once after settings were frozen on the validation split; published numbers are marked as published and cite their source. Nothing is tuned on a test split.
- Best in cyan, ours marked: the top number in each column is in accent whether or not it is ours. A model that trails on a benchmark is shown trailing.
- Evolving daily: settings move one bounded step a day on tuning data and are checked the next day on fresh documents; a published number changes only when the research harness re-runs on the test split.
- Cost: the point of these models is the price. Rates are set daily below the cheapest comparable hosted API, so an AI developer saves 30 to 90 percent per token or API call at the quality shown here. Pricing and keys: /tuatara; model documents: /api/v1/models.