Methods & benchmarks

Overview of benchmarks

Overview of the analyzed benchmarks.

42benchmarks
analyzed

Setting

1single table
N±bounded set
Nmulti-table provided
∞open corpus

Insight types

borderedrequires an artifact (a trained model, say), not just an insight

Find a benchmark Curation Corpus format Data setting Input protocol Insight type Validation function

42 benchmarks

	Corpus					Inputs				Validation
		Source			Supplement data				Insight types	Function	Output
AIT-QA	116	10-K financial reports of major airline companies	Mixed	Markup tables	hierarchical column and row headers,metadata	515	1	Static, single	Look	Exact match	Value
ARCADE	106	Kaggle, GitHub	Mixed	Standalone files	none released alongside the tables; the NL schema descriptions (column names and example cell values) used in prompts are derived from the tables themselves, and the notebook context is part of the input rather than supplementary corpus data	1,078	1	Static, multi-turn	LookAggChar	Similarity	ValueTableList
Archer	68	Spider (Yu et al., 2018), 20 of the 166 publicly released databases	Synthetic	Relational DBs	foreign-key relations and column type declarations from the database schema, plus a per-question commonsense knowledge statement for the questions that require commonsense reasoning	1,042	N±	Static, single	LookAggCF	Exact matchComponent	ValueTable
BEAVER	–	Oracle data warehouse, MySQL databases	Manual	Relational DBs	column names, column types, rows, column mapping annotation	203	N±	Static, single	LookAgg	Exact matchStructural match	ValueTable
BIRD	611	Kaggle (32%), CTU Prague Relational Learning Repository (48%), open tables assembled and schema-standardized by the authors (20%)	Mixed	Relational DBs	database description files (CSV) giving full schema names and value descriptions, one expert-annotated external knowledge evidence sentence per question, and the relational schemas with key relations between tables	10,962	N±	Static, single	LookAgg	Exact match	ValueTable
BIRD-INTERACT	–	LIVESQLBENCH	–	Relational DBs	hierarchical knowledge base, metadata files, database sandbox	900	N±	User simulation	LookAgg	Exact matchProgrammatic	ValueTable
BLADE	14	Scientific publications studied in meta-analysis papers, crowd-sourced analysis studies, and statistics textbooks	Manual	Standalone files	dataset description including details of each column, and the research question or hypothesis	188	1	Static, single	AssoSI	Exact matchStructural matchComponent	ValueText
BookSQL	7	existing real-world accounting databases, anonymized	Mixed	Relational DBs	foreign-key relations, primary keys	100,000	N	Static, single	LookAgg	Exact matchStructural matchProgram comparison	ValueTableList
CausalReasoning / Benchmark	132	peer-reviewed research papers, causal-inference textbooks	Manual	Standalone files	column descriptions, study context	173	1	Static, single	AggSIInt	SimilarityComponent	ValueText
ConDABench	1,855	TidyTuesday, Kaggle notebooks, ScienceDirect open-access articles	Mixed	Standalone files	supporting reference code (Python) grounding each query-answer pair, and the source article each problem is derived from	1,420	N	User simulation	LookAggCharAssoPred	LLM-as-judge	ValueTextTableChart
CoSQL	1,020	Databases inherited unchanged from Spider/SParC (originally sourced from Wikipedia, textbooks, and other web/DB sources)	Synthetic	Relational DBs	foreign-key relations, DB schemas	3,007	N±	Static, multi-turn	LookAgg	Exact matchStructural matchSimilarityHuman	TextTable
COTA	–	Kaggle	Mixed	Standalone files	column meanings, value illustrations, client personas, dialog histories	1,013	1	User simulation	LookAggCharAssoSIPred	Exact matchStructural matchSimilarityProgram comparison	ValueTextTableListChart
CRT-QA	423	Wikipedia	Manual	Web tables	table titles	1,000	1	Static, single	LookAggAsso	Exact matchHuman	Text
DA-Code	1,733	Kaggle, GitHub, Other Web Sources	Mixed	Mixed	documentation and specification files shipped alongside the data (e.g. data_standard.md, schema.yml), plus markdown, JSON and text resources; per-instance reference plans for the DA-Code-100 subset	500	N	Static, single	LookAggCharAssoGrpSIPred	Exact matchSimilarity	ValueTextTableChart
DA-Dataset	689	HangSeng Financial text-to-SQL dataset (DA-CCKS), BIRD (DA-BIRD)	Mixed	Relational DBs	database schema (table names, column names, data types), natural language table and column descriptions, reference SQL queries and reference reports; relational structure inherited from the source databases	735	N±	Static, single	AggCharAsso	Structural matchLLM-as-judge	Text
DABstep	3	Adyen	Synthetic	Data lake	column descriptions, documentation files	450	N±	Static, single	LookAggCF	Exact matchSimilarityHuman	ValueTextList
DACO	–	Spider, Kaggle	Mixed	Mixed	table titles, column names, example values	1,942	N	Static, single	LookAggCharAsso	SimilarityLLM-as-judgeHuman	Text
DAComp - DA	418	Web (base databases), enterprise SaaS schemas (analytical modeling layers via DE pipeline)	Synthetic	Relational DBs	foreign-key relations, analytical modeling layers, data contracts	100	N±	Static, single	LookAggCharAsso	LLM-as-judgeHuman	TextChart
DSBench	–	ModelOff, Kaggle	Mixed	Standalone files	images, task descriptions, training and testing files, sample submission files	540	N±	Static, single	LookAggPred	SimilarityProgrammaticLLM-as-judge	ValueTable
FeTaQA	10,330	Wikipedia (via ToTTo)	Manual	Web tables	page title, section title, table title, highlighted table cells (denotations)	10,330	1	Static, single	LookAgg	Structural matchSimilarityHuman	Text
GRI-QA	204	–	Mixed	Markup tables	GRI topic, row and column indices	4,089	N	Static, single	LookAgg	Exact match	ValueTextList
InfiAgent-DABench	67	GitHub	Mixed	Standalone files	–	257	1	Static, single	AggCharAssoSIPred	Exact match	Value
KaggleDBQA	17	Kaggle	Manual	Relational DBs	database documentation, column descriptions, table names, database overview, categorical value descriptions	272	N±	Static, single	LookAgg	Structural match	ValueTable
KramaBench	1,636	scientific papers/data repositories (archaeology, astronomy, biomedical) and government/civic open-data portals (FTC, Wikipedia, NOAA, NIFC, EPA, US Census, Zillow, Kaggle) grounding published domain-expert analyses	Manual	Data lake	unstructured textual data accompanying the tabular files (scientific papers, yearly financial reports, infographics) that the tasks are grounded in; per-domain source documentation is described narratively in Appendix A rather than shipped as separate documentation files	104	N±	Static, single	LookAggAssoPred	Exact matchSimilarityComponent	ValueTextList
LongTableBench	6,800	–	Synthetic	Mixed	foreign-key relations	5,950	N	Static, multi-turn	LookAgg	Similarity	ValueTextList
MMQA	3,312	Spider (cross-domain relational databases spanning 138 domains), not Wikipedia	Mixed	Relational DBs	foreign-key relations, primary keys, SQL query, gold answer, natural language question	3,312	∞	Static, single	LookAgg	Exact matchStructural matchProgram comparisonLLM-as-judge	ValueTextList
MT-RAIG	19,563	SPIDER (relational database tables) and Open-WikiTable (Wikipedia web tables)	Mixed	Mixed	none beyond the tables themselves; each table is serialized with its title and headers, while the foreign-key/topic groupings and the expanded natural-language facts were used only during benchmark construction and are not supplied to the evaluated system	18,532	∞	Static, single	LookAggCharAsso	Structural matchLLM-as-judge	Text
MULTIHIERTT	–	FinTabNet	Mixed	Markup tables	table hierarchies, cell formats, unstructured text, reasoning programs, supporting facts	10,440	N±	Static, single	LookAggAsso	Exact matchSimilarity	ValueText
MultiTableQA	59,307	Wikipedia (via HybridQA, SQA, TabFact and WikiTables)	Mixed	Web tables	table captions, column headers, and table metadata (contextual details, associated resources, and 3,000 example query/answer pairs injected into the metadata for demonstration)	23,785	∞	Static, single	LookAgg	Exact matchStructural matchSimilarityProgrammatic	ValueList
NQ-Tables	169,898	Wikipedia	Mixed	Web tables	Wikipedia page title per table, used as the table title by the retriever and reader; no other supplementary data is released	11,628	∞	Static, single	LookAgg	Exact matchSimilarity	Value
Open-WikiTable	24,680	Wikipedia, via the WikiSQL and WikiTableQuestions table corpora	Mixed	Web tables	Table descriptions (page title, section title, caption) plus a table ID, all flattened into the retrieval passage; each question is additionally annotated with a gold SQL query alongside the textual answer.	67,023	∞	Static, single	LookAgg	Exact matchStructural match	Value
OTT-QA	8,891	Wikipedia	Mixed	Web tables	text passages (5 million Wikipedia introduction passages), page title, page section title, section text; cell-wise hyperlinks in the training subset only	4,372	∞	Static, single	LookAgg	Exact matchStructural matchSimilarity	Value
RETQA	4,932	–	Mixed	Standalone files	table captions, intent labels, slot labels	20,762	∞	Static, single	LookAgg	Exact matchStructural matchSimilarityProgrammatic	TextTable
Spider	1,056	college database courses, SQL tutorial websites, online csv files, textbook examples, DatabaseAnswers data models, WikiSQL tables	Synthetic	Relational DBs	database schemas with foreign-key relations between tables	11,840	N±	Static, single	LookAgg	Exact matchStructural match	ValueTable
Spider 2.0	418	BigQuery public data, Snowflake Marketplace data and other database platforms (SQLite, DuckDB, PostgreSQL, ClickHouse); SQL queries from technical tutorials and community forums; data transformation projects from Fivetran and DBT	Mixed	Relational DBs	project codebases and DBT/Fivetran project files, SQL dialect and external function documentation, external reference documents, database schemas with column descriptions, sampled cell values, and (for Spider 2.0) predefined answer files and data model descriptions	632	N±	Static, single	LookAggAsso	Exact matchProgrammatic	TextTable
Tab-CQA	7,041	Chinese financial reports, listed companies on Shenzhen Stock Exchange and Shanghai Stock Exchange	Mixed	Standalone files	conversation flow tags, answer type annotations	109,089	1	Static, multi-turn	LookLookAgg	Similarity	ValueText
TableBench	886	–	Mixed	–	–	886	1	Static, single	LookAggAssoPred	SimilarityProgrammaticLLM-as-judgeHuman	TextChart
TableEval	617	–	Mixed	Markup tables	table captions, explanatory notes, adjacent text snippets	2,325	1	Static, multi-turn	LookAggCharAsso	LLM-as-judge	Text
Tabular Math Word Problems (TABMWP)	37,644	IXL	Mixed	Standalone files	table titles, images, semi-structured text, gold solutions, options, units	38,431	1	Static, single	LookAggChar	Exact match	ValueText
TACO	13,004	Beijing smart city data service, Beijing Municipal Open Data Platform, U.S. Government's Open Data Portal	Mixed	Relational DBs	foreign-key relations, table/column descriptions, schema metadata	13,000	∞	Static, single	LookAgg	Exact match	ValueTable
TOPBench	35	Kaggle	Mixed	Standalone files	column definitions, value distributions	779	1	Static, single	Pred	Exact matchSimilarityLLM-as-judge	TextTable
WikiTableQuestions	2,108	Wikipedia	Manual	Web tables	–	22,033	1	Static, single	LookAgg	Exact match	ValueList

Reading the table

Corpus: #Tab number of tables, Source source of the tables,Curation how the tables are filled with data, Format how the tables are stored, and Supplement data what is provided alongside the tables.
Inputs: #Inst. the number of instances, Setting the data setting under which the evaluation operates. Protocol how the input is supplied to the system.
Validation: Function the validation function the benchmark uses to check the validity, and Output the output modalities it expects/supports.
Insight types: The insight types that the inputs in the benchmark ask for. The items with border indicate that the benchmark asks for an artifact (e.g., a prediction model) instead of some specific analytical knowledge.
Look lookup
Agg aggregation
Char characterization
Asso association
Grp grouping
SI statistical inference
Pred prediction
Int interventional
CF counterfactual