Four published benchmarks for financial reasoning, documented from their primary sources — and a plain statement that Finexia has not run any of them.
Primary sources, read and citedFinexia · published · updated · 8 sources
Every figure on this page comes from the people who built the benchmark. None of it comes from us.
Still missingFinexia has not run any of these benchmarks yet. There is no Finexia score on this page, and there will not be one until every item of the protocol below is satisfied and published alongside it.
A number is only readable if you know who produced it. The catalogue below tags every result with one of four origins, and never places two of them in the same column.
Origin
What it is
What it cannot tell you
Benchmark authors
A figure published in the paper or its repository.
How a different system, prompt or retrieval setup behaves.
Vendor announcement
A figure a model provider published about its own model.
Anything, unless the full protocol was published with it.
Third-party reproduction
A figure produced by someone other than the authors or the vendor.
How it compares to the paper, unless the protocol matches exactly.
Run by Finexia
A figure we produced ourselves, with the manifest attached.
Three of these four sets have been public since 2021, under permissive licences, on platforms designed to be crawled. Assuming a modern model has never seen them is not conservative — it is unrealistic. Any result we publish will state the model’s training cutoff next to the benchmark’s publication date, and a result where the second precedes the first will be labelled as such.
FinanceBench measured a strong system answering incorrectly or declining on 81% of a sampled subset 1. Those two outcomes are not equivalent. A desk that declines has cost you a minute; a desk that answers confidently from the wrong page has cost you a decision. Our own evaluation, when it happens, will report them as two separate numbers.
Islam, Kannappan, Kiela, Qian, Scherrer, Vidgen · 2023-11-20 · CC BY-NC-ND 4.0 — non-commercial, no derivative works.
Measure whether a system can answer plain questions about public companies from their own filings, and whether it knows when it cannot.
Data
Questions over the filings of publicly traded companies, each paired with an answer and the passage that supports it.
Size
10,231 questions with answers and supporting evidence.
Tasks
Open-book question answering over a filing · Retrieval of the supporting passage · Refusal when the document does not support an answer
Metrics
Correctness against the annotated answer · Manual review of a sampled subset
Protocol
The authors evaluated sixteen model configurations, and manually reviewed 2,400 answers across a 150-case sample. Systems were tested both with a retrieval component and with long-context prompting.
Why it matters here
This is the closest published proxy for what a fundamental mandate does all day: read a filing, answer a question about it, and say so when the filing does not answer. The refusal axis matters more to us than the accuracy axis, because a desk that guesses is worse than a desk that stops.
Contamination
The underlying filings are public and long predate the benchmark, so a model trained on the open web has almost certainly seen the documents. What it may not have seen is the annotated answer set. Any result reported without stating the model’s training cutoff is uninterpretable.
Reproducible?
Reproducible in principle: the set is published. The non-commercial licence has to be respected, and the retrieval component has to be described exactly, because most of the reported variance comes from retrieval rather than from the model.
Limits
The headline figure dates from 2023 and the models evaluated have since been superseded.
Questions are described by their authors as clear-cut, which is not what a research desk mostly receives.
The licence forbids derivative works, so the set cannot be adapted or redistributed in modified form.
Published results
Origin
System
Metric
Value
Published by the benchmark authors
GPT-4-Turbo with a retrieval system
Incorrect or declined, on the sampled subset
81%
Finexia has not run this benchmark yet.
FinQA
Chen, Chen, Smiley, Shah, Borova, Langdon, Moussa, Beane, Huang, Routledge, Wang · 2021-11-01 · MIT, for both the code and the dataset, per the official repository.
Measure multi-step numerical reasoning over financial reports, and make the reasoning itself checkable rather than only the answer.
Data
Question-answer pairs written by financial experts over corporate reports, each annotated with the reasoning program that produces the answer.
Size
8,281 examples: 6,251 train, 883 validation, 1,147 public test. Part of the test set is held private.
Tasks
Extract the operands from text and tables · Generate the reasoning program · Execute it to produce the answer
Metrics
Execution accuracy — the final number is right · Program accuracy — the reasoning is right
Protocol
Models are scored on the public test split against both metrics. The private split exists to limit overfitting to the leaderboard.
Why it matters here
Program accuracy is the metric that matches our contract. A workbook that reaches the right total with the wrong formula is a workbook that will be wrong next quarter. The core writes real formulas into the Output and Checks sheets for exactly this reason.
Contamination
The dataset has been public since 2021 and is widely mirrored. Any model trained after that date may have seen both the questions and the gold programs. This is the strongest contamination risk of the four.
Reproducible?
Reproducible: the split is published under MIT, and the repository ships the scoring code. Results on the private split cannot be reproduced independently.
Limits
The documents span 2003 to 2019, so accounting presentation conventions differ from current filings.
The reasoning programs are annotated in a fixed operator vocabulary, which is narrower than what an analyst actually does.
A model can reach the right answer through a program the annotation does not recognise, and be scored wrong.
Published results
Origin
System
Metric
Value
Published by the benchmark authors
FinQANet (RoBERTa-large)
Execution accuracy, public test
61.24
Published by the benchmark authors
FinQANet (RoBERTa-large)
Program accuracy, public test
58.86
Finexia has not run this benchmark yet.
TAT-QA
Zhu, Lei, Huang, Wang, Zhang, Lv, Feng, Chua · 2021-08-01 · Dataset under CC BY 4.0; evaluation code under MIT.
Measure question answering when the answer needs both a table and the prose around it.
Data
Hybrid contexts drawn from real financial reports: a table plus its surrounding paragraphs.
Size
16,552 questions over 2,757 hybrid contexts.
Tasks
Locate the relevant cells and the relevant spans · Apply an aggregation — addition, subtraction, multiplication, division, counting, comparison, sorting · Return the answer with its scale and unit
Metrics
Exact match · F1
Protocol
The official scripts published with the repository compute the metrics. The authors report both their own model and a human expert baseline on the same set.
Why it matters here
This is the shape of nearly every real question a desk receives: the number is in the table, the qualification is in the paragraph, and using one without the other produces a confident error. The human-versus-model gap here is the honest measure of how far the task is from solved.
Contamination
Public since 2021 under a permissive licence, therefore very likely present in modern training corpora. A result on TAT-QA reported without a training cutoff says little.
Reproducible?
The most reproducible of the four: permissive licence on the data, MIT on the scoring code, full split published.
Limits
Answers require a scale and a unit, and scoring is sensitive to how those are formatted.
The contexts are extracted, so a system is never asked to find the right page in a 200-page filing.
Published in 2021; presentation conventions have moved.
Separate the three abilities a quantitative business question needs: reading the document, knowing the concept, and solving with code.
Data
Eight quantitative reasoning tasks, including three code-generation tasks over newly collected and augmented question-answer data.
Size
Eight tasks. The paper does not state a single headline example count, and this page will not invent one.
Tasks
Extract intermediate values from financial text and tables · Apply the financial concept and formula the question implies · Synthesise a program that computes the answer
Metrics
Task-level accuracy, per sub-task
Protocol
Each of the eight tasks is scored separately, which is the point: an aggregate score hides which of the three abilities is failing.
Why it matters here
The finding matters more than the score. The authors report that the bottleneck sits in business and financial understanding rather than in arithmetic, which is an argument for written mandates and a tool contract rather than for a larger model.
Contamination
Partly built from newly collected data, which lowers the risk relative to the older three — but the augmentation sources are themselves public.
Reproducible?
Reproducible under the licence, per task. Cross-task aggregation should not be reported.
Limits
The tasks are heterogeneous, so a single average across them is not meaningful.
Program synthesis scoring rewards a specific solution shape.
The non-commercial share-alike licence constrains derivative use.
Published results
No headline figure is quoted here: the paper reports per-task results that an average would misrepresent.
Finexia has not run this benchmark yet.
#The protocol a Finexia score would have to satisfy
Name the dataset and its exact version or commit, and the licence under which it is used.
Name the subset evaluated. A result on a hand-picked slice is not a result on the benchmark.
Name the exact model identifier and the provider that served it. A family name is not a model.
Record the date of the run, because model endpoints change under a stable name.
Publish the prompt verbatim, the temperature, and whether any tool was available.
State whether a retrieval component was used, and if so, describe it — most reported variance comes from there.
Report the number of cases attempted and the number scored, separately.
Keep the raw outputs, and publish their SHA-256.
Describe the scoring method, and whether any human review was applied.
Name the Finexia commit that produced the run, and attach the run manifest.
Compare across models only when every line above is identical between them.
FinQA: A Dataset of Numerical Reasoning over Financial DataZhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, William Yang Wangpeer-reviewed paper · EMNLP 2021 · anthology 2021.emnlp-main.300 · published 2021-11-01 · read 2026-09-10 · CC BY 4.0https://aclanthology.org/2021.emnlp-main.300/
TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in FinanceFengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, Tat-Seng Chuapeer-reviewed paper · ACL-IJCNLP 2021 · doi 10.18653/v1/2021.acl-long.254 · published 2021-08-01 · read 2026-09-10 · CC BY 4.0https://aclanthology.org/2021.acl-long.254/
NExTplusplus/TAT-QA — official repositoryNExT++, National University of Singaporeofficial repository · read 2026-09-10 · dataset CC BY 4.0 · code MIThttps://github.com/NExTplusplus/TAT-QA
BizBench: A Quantitative Reasoning Benchmark for Business and FinanceRik Koncel-Kedziorski, Michael Krumdick, Viet Lai, Varshini Reddy, Charles Lovering, Chris Tannerpeer-reviewed paper · arXiv:2311.06602v2 · revised 2024-03-12 · published 2023-11-11 · read 2026-09-10 · CC BY-NC-SA 4.0https://arxiv.org/abs/2311.06602