Methods & benchmarks

Overview of systems

This page provides an overview of systems that work towards (parts of) the end-to-end scope of OpenTI.

58systems
surveyed

Capability

Interpretation
Data retrieval
Analytical composition and execution
Output synthesis
Process governance

Setting

1single table
N±bounded set
Nmulti-table provided
∞open corpus

Find a system Data setting Orchestration Input mode

58 systems

			Functional capabilities
			Interpretation	Data retrieval	Analysis	Output	Governance	Input modes	Evaluated on
ACR	1	Fixed	AugmentContextClarify	–	PythonPrep	TextTableChartContext	Error handling	Step-wiseSystem clarificationOutput-driven	COTA
AixelAsk	1	Plan-based	DecomposeAugmentScope	–	ModelPrep	TextIntegrate	Error handling	Single input	FeTaQA+TabFact+WikiSQL+WikiTQ+WikiTQ-4k
ALTER	1	Fixed	DecomposeAugmentScope	–	SQLModelPrep	TextIntegrate	Ensembling	Single input	TabFactWikiTQ
AOP	∞	Hybrid	DecomposeAugment	RepresentationGranularityAlignmentDynamic	SQLModelToolPrep	TextContextIntegrate	Error handlingValidationEnsemblingAdaptive steering	Single input	CRAGSports StackExchange
AutoTQA	N±	Hybrid	DecomposeAugmentContext	Representation	SQLPython	TextTableContextIntegrate	Error handlingValidationAdaptive steering	Single input	FeTaQAIndustrial DatasetTabFactWiKiTQ
Cao et al.	N±	Fixed	Decompose	Alignment	SQLPythonToolPrep	Text	Ensembling	Single input	AIT-QAHiTabSpiderWikiTQ
Chain-of-Table	1	Reactive	–	–	ModelToolPrep	Text	–	Single input	FeTaQATabFactWikiTQ
CHASE-SQL	N±	Fixed	DecomposeContext	Granularity	SQL	Table	Error handlingEnsembling	Single input	BIRDSpider
CQR-SQL	N±	Fixed	AugmentContext	–	SQL	–	–	Step-wise	CANARDCOSQLSPARC
CRUSH4SQL	∞	Fixed	Augment	RepresentationGranularityAlignment	SQL	–	–	Single input	BirdUnionSocialDBSpiderUnion
DA-Agent	N±	Reactive	–	–	SQLPythonPrep	TextTableChart	Error handlingValidationAdaptive steering	Single input	DA-CodeDA-Code-100
DAAgent	1	Reactive	Decompose	–	PythonPrep	Text	Error handling	Single input	DAEvalDAInstruct
DACO	N±	Reactive	Decompose	–	Python	TextContextIntegrate	Error handling	Single input	DACOFeTaQAInfiAgent-DA
DAgent	N±	Fixed	DecomposeContext	RepresentationGranularity	SQLModel	TextContextIntegrate	Validation	Single input	DA-BIRDDA-CCKS
DATAMIND	N±	Reactive	Scope	–	SQLPythonPrep	Text	Error handling	Single input	BIRDDABenchDATAMIND-12KOmniSQLQRDataTableBench
DataQue	N±	Fixed	AugmentContextClarify	RepresentationGranularity	SQLTool	TableChartContextModality select	Error handlingValidation	Output-drivenSystem clarificationSingle input	Own Dataset
Dater	1	Fixed	DecomposeAugmentScope	–	SQLModel	TextIntegrate	Ensembling	Single input	FetaQATabFactWikiTableQuestion
DBCopilot	∞	Fixed	–	RepresentationGranularityAlignmentGenerative	SQL	–	–	Single inputSystem clarification	BirdFibenSpiderSpiderrealSpidersyn
DIN-SQL	N±	Fixed	DecomposeAugment	–	SQL	Table	Error handling	Single input	BIRDSpider
DS-GURU	N±	Fixed	Augment	RepresentationAlignment	PythonPrep	Text	Error handlingValidation	Single input	KramaBench
FISQL	N±	Fixed	AugmentClarify	–	SQL	TextTableContext	–	Output-driven	Spider
GTR	∞	Fixed	–	RepresentationGranularity	ModelPrep	Text	–	Single input	MUTLITABLEQA
H-STAR	1	Fixed	DecomposeScope	–	SQLModelPrep	TextIntegrate	ValidationEnsembling	Single input	FeTaQATabFactWikiTQ
Inner Table Retriever (ITR)	1	Fixed	Scope	–	SQLModel	Text	ValidationEnsembling	Single input	WikiSQLWikiTableQuestions
JAR	∞	Fixed	DecomposeAugment	RepresentationGranularityAlignment	SQL	Table	–	Single input	BirdSpider
LI-RAGE	∞	Fixed	–	RepresentationGranularity	ModelPrep	Text	Ensembling	Single input	E2E-WTQNQ-TABLESWikiTableQuestions
LitE-SQL	N±	Fixed	–	RepresentationGranularity	SQL	Text	Error handling	Single input	BIRDSpider 1.0
MACT	1	Reactive	DecomposeAugment	–	PythonTool	Text	Error handlingEnsembling	Single input	CRTSCITABTATWTQ
MITQA	∞	Fixed	Scope	RepresentationGranularityDynamic	ModelPrep	Text	Ensembling	Single input	HybridQAMS MARCO Passage RetrievalOTT-QA
MultiTabQA	N	Fixed	–	–	Model	Table	–	Single input	AtisGeoQuerySpider
MURRE	∞	Fixed	Augment	RepresentationAlignmentDynamic	SQL	–	Ensembling	Single input	BirdUnionSpiderUnion
OpenTab	∞	Fixed	Scope	Granularity	SQLModelPrep	Text	Error handlingValidation	Single input	FEVEROUSOpen-WikiTableSpiderWikiTableQuestions
PI-SQL	N±	Fixed	DecomposeAugment	–	SQLPythonPrep	Text	Error handlingValidationEnsembling	Single input	ArcherBIRD
POTABLE	1	Fixed	–	–	PythonPrep	Text	Error handling	Single input	TabFactWikiTQ
QCMA-SQL	N±	Fixed	DecomposeAugmentContext	–	SQL	Text	Error handling	Single input	CSpiderSpiderSpider-DKSpider-RealisticSpider-SYN
ReAcTable	1	Reactive	Decompose	–	SQLPythonPrep	TextTable	Error handlingEnsembling	Single input	FeTaQATabFactWikiTQ
ReFSQL	N±	Fixed	Context	–	SQL	Table	–	Single input	SpiderSpider-DKSpider-RealisticSpider-SynWikiSQL
ROUTE	N±	Fixed	Augment	–	SQL	Text	Error handlingValidation	Single input	BIRDDr.SpiderSpiderSpider-DKSpider-RealisticSpider-SYN
RTS++	N±	Fixed	AugmentClarify	–	SQL	Text	ValidationEnsembling	System clarificationSingle input	BIRDSpider
S3HQA	1	Fixed	AugmentScope	–	ModelPrep	Text	–	Single input	HybridQA
SAFE-SQL	N±	Fixed	AugmentContext	–	SQL	Text	Validation	Single input	Bird dev datasetSpider dev dataset
SGAM	N±	Fixed	DecomposeAugment	RepresentationGranularityAlignment	SQLModelPrep	Text	–	Single input	BIRDCISS
STaR-SQL	N±	Fixed	–	–	SQL	TextTable	ValidationEnsembling	Single input	Spider
SYNTQA	1	Fixed	–	–	SQLModel	Text	Error handlingValidationEnsembling	Single input	ROBUT-WIKISQLSQUALLWIKISQLWTQ
TABDSR	1	Fixed	Decompose	–	PythonPrep	TextIntegrate	Error handlingValidation	Single input	CALTAB151TAT-QATableBench
Table-Critic	1	Fixed	–	–	ModelToolPrep	Text	ValidationAdaptive steering	Single input	TabFactWikiTableQuestions (WikiTQ)
TableGPT2	∞	Hybrid	ScopeContextClarify	Representation	SQLPythonToolPrep	TextTableChartContextModality select	Error handlingAdaptive steering	Step-wiseIterative elicitationSingle input	BIRD (dev)FEVEROUSFeTaQAHiTabHybridQARealTabBenchSpiderTURLTabFactTableBenchToTToWikiSQLWikiTQ
TABLELLM	1	Fixed	–	–	PythonModel	TextChart	–	Single input	TableBenchTableInstruct
TableRAG	1	Hybrid	AugmentScope	–	PythonPrep	Text	Error handlingEnsembling	Single input	ArcadeQABirdQASynthetic TabFactWikiTableQA
TabSQLify	1	Fixed	Scope	–	SQLModelPrep	Text	Error handlingValidation	Single input	FeTaQATabFactWikiSQLWikiTQ
TALON	1	Reactive	Scope	–	PythonToolPrep	Text	Error handlingAdaptive steering	Single input	BirdQAWTQWTQ-L
TAMO	N	Fixed	–	–	ModelPrep	Text	–	Single input	FeTaQAHiTabMultiTabQA-geoQueryStructQAWikiSQLWikiTableQuestions (WikiTQ)
TaPERA	1	Hybrid	Augment	–	Python	TextContextIntegrate	Error handlingAdaptive steering	Single inputStep-wise	FeTaQAQTSumm
TART	1	Fixed	AugmentScope	–	PythonPrep	TextContext	Error handling	Single input	FinQAHiTabHybridQAPubHealthTabSciTABTAT-QATOOLTABTabFactTabMWPWikiTableQuestion
TiInsight	∞	Fixed	DecomposeAugmentContext	RepresentationGranularityAlignment	SQL	TableChartModality select	Error handlingValidation	Single inputOutput-driven	BirdFinancialSpider
ToolWriter	1	Fixed	Scope	–	ModelTool	Text	Error handling	Single input	SQUALLWikiSQLWikiTableQuestionsWikiTableQuestions-Filter
UCS-SQL	N±	Fixed	AugmentContext	GranularityAlignment	SQL	Table	–	Single input	BIRDSpider
Weaver	1	Plan-based	DecomposeAugmentScope	–	SQLModelPrep	Text	Error handlingAdaptive steering	Single input	BIRDFinQAFinQAMMMMTabQAMMOTT-QAMMTabFactWikiTQ

Reading the table

Functional capabilities: Each tag captures one aspect of a capability the system implements; an empty cell means it does not address that capability at all.
Orchestration: Orchestration strategy for the functional capabilities: fixed at design time, plan-based at invocation time, reactive during execution, or hybrid.
Setting: What data the system is presented with. Differentiating when a single table is provided, multiple tables, a bounded set (like a database), or a full corpus.
Interaction modes: The interaction mode by which the system and user interact to communicate the insight need: a single upfront input, a step-wise sequence of inputs,system-initiated clarification, iterative elicitation refined over turns, or steering off an output-driven intermediate result. See Section 4.1.
Evaluated on: The datasets/benchmarks the system is evaluated against, as reported by the authors.