Open Tabular Insight Extraction: Where Do We Stand, and Where Should We Go?

Daniel Gomm, Maarten de Rijke, Madelon Hulsebos

8 Appendix

8.1 Overview of Notation

Table 4.

Overview of notation introduced in Sections 2, 3, and 5.

Symbol	Description
Insights and insight needs
$i$	An insight need, a person's requirement for analytical knowledge that, interpreted within their contextual understanding, constitutes an insight.
$i_{d}$	The data need, specifying what data entities, temporal scope, and domain boundaries are relevant to the insight need.
$i_{m}$	The methodological need, specifying what analytical operations and procedures should be applied.
$i_{o}$	The output need, specifying how results should be presented.
$\hat{i}$	The expression of an insight need $i$ communicated by a user, often only a partial approximation of $i$ .
$\hat{i}^{s}$	A system's representation of the insight need, constructed from $\hat{i}$ to approximate $i$ .
Tabular data and corpus
$t = (H, D, M)$	A table, consisting of a header $H$ , a data matrix $D$ , and a metadata set $M$ .
$T = {t_{1}, \dots, t_{n}}$	A corpus of $n$ tables over which insight needs are expressed.
$T$	The universe of all possible tables, with a corpus $T \subset T$ .
Realizations
$r = (O, E, β)$	A realization, a directed acyclic graph of fully parameterized operations $O$ connected by dataflow edges $E$ with a data binding $β$ .
$R$	The space of all possible realizations, with $r \in R$ .
$O = {o_{1}, \dots, o_{m}}$	The set of fully parameterized operations applied in a realization's derivation.
$o$	A single operation, a discrete functional unit ranging from granular primitives to high-level analytical functions.
$E \subset O \times O$	The set of directed edges connecting operations in a realization, defining the flow of data between them.
$β \subset T \times O$	The data binding connecting specific tables from the corpus to the source operations of a realization.
$out (r)$	The analytical knowledge produced as output by executing realization $r$ .
Utility and problem formulation
$u_{i}$	The realization utility function for insight need $i$ , $u_{i} : R \mapsto R$ , scoring how well a realization satisfies $i$ .
$R_{v a l i d} (i)$	The set of realizations valid for insight need $i$ , i.e., realizations whose data, operations, and output address $i$ .
$R_{f e a s ib l e} (T)$	The set of realizations feasible given corpus $T$ , i.e., where bound tables are in $T$ .
$R_{c an d} (i, T)$	The set of candidate realizations, $R_{v a l i d} (i) \cap R_{f e a s ib l e} (T)$ , that both address the insight need and are grounded in available data.
$r^{*}$	The candidate realization that maximizes utility; reused in the evaluation section to denote a benchmark's reference realization.
Evaluation
$B = (T, E)$	An evaluation benchmark, comprising a corpus of tables $T$ and a set of evaluation instances $E$ .
$E = {e_{1}, \dots, e_{n}}$	The set of $n$ evaluation instances in a benchmark.
$e = (\overset{ˉ}{i}, v)$	An evaluation instance, comprising an input $\overset{ˉ}{i}$ and a validation mechanism $v$ .
$\overset{ˉ}{i}$	The benchmark input, representing the expression of an insight need $\hat{i}$ under the benchmark's input protocol.
$v$	The validation mechanism scoring a realization, $v : R \mapsto [0, 1]$ , acting as an operational stand-in for $R_{v a l i d} (i)$ and $u_{i}$ .

Several existing works survey and systematize areas related to OpenTI, yet each addresses only a part of its problem space from within the conventions of particular research communities. A larger group of works exists that systematize systems that perform complex tasks over unseen information without targeting tabular data, spanning open-domain question answering , retrieval augmented generation , and LLM-agents . More closely related, other works target tabular data but organize contributions under the labels of the communities they originate from, such as text-to-SQL , table question answering , the application of LLMs to tabular tasks , and agents for data science and data analysis . Table 5 compares these works against the dimensions foregrounded in our framing of OpenTI, namely the coverage of the end-to-end scope, the treatment of an open setting in which the corpus is unknown when the need is expressed, an insight-centered rather than answer-centered framing, a formal definition of the problem, and the treatment of user-facing concerns.

Table 5.

Comparison of this paper with other works that survey related areas. $-$ indicates aspects that are not addressed, $\circ$ ones that are partially addressed or acknowledged for future work, and $\bullet$ signifies that the work foregrounds the aspect as a central part of its contribution. For the end-to-end scope coverage, we indicate the explicit treatment of parts of the end-to-end scope using the abbreviations UI for User Interaction, R for Table Retrieval, DI for Data Integration, DA for Data Analysis, and O for Output Presentation.

Work	Focus	Research communities	End-to-end scope coverage	Open setting	Insight-centered	Formal definition	User-facing concerns
	Text-to-SQL	DB, NLP	UIRDIDAO	−	−	○	−
	LLMs for tabular tasks	NLP, DB	UIRDIDAO	−	−	○	−
	LLMs for tabular tasks	NLP, DB	UIRDIDAO	−	○	○	○
	Table question answering	NLP, IR	UIRDIDAO	○	○	−	○
	LLM-agents for tabular tasks	NLP, DB	UIRDIDAO	○	○	○	○
	LLM-agents for tabular tasks	NLP, ML/AI	UIRDIDAO	−	−	−	○
	LLM-agents for data science	DB, NLP, ML/AI	UIRDIDAO	○	−	−	○
This paper	Open Tabular Insight Extraction	DB, HCI, IR, ML/AI, NLP	UIRDIDAO	●	●	●	●

Compared to this paper, these works cover only part of the path from an expressed need to a presented insight, mostly the analysis, while assuming that the relevant tables are supplied rather than discovered from an unknown corpus. The setting they address is thus a closed one, and where openness appears it does so as one task dimension among many rather than the premise of the problem . None of these works takes an insight-centered perspective, treating the tabular tasks as targeting surface-level answer correctness, not as the search for the insights that serve the user. Formalization, where offered, stays confined to isolated subtasks such as translating a question into SQL , and the user-facing side of the problem, from interaction to personalization and interpretability, is raised as an open challenge more than it is developed . Instead, our framing of OpenTI treats these dimensions as one problem, spanning the full path, taking the open setting and the insight need as its starting point, grounding both in a formal definition, and placing user-facing concerns as a central aspect.

8.3 Literature Review Protocol

We collect the works that provide the evidence base for this paper following a structured, multi-stage protocol organized into identification, screening, and selection stages, inspired by the PRISMA guidelines for reporting systematic reviews . We start with a broad keyword search over a curated set of venues (Section 8.3.1) and proceed through manual screening against inclusion criteria (Section 8.3.2), yielding a set of relevant works that informs this paper throughout. From this set we extract the systems and benchmarks that we analyze in Sections 3 and 5 (Section 8.3.3). The sets obtained through this protocol further provide the basis for the analysis of benchmark inputs in Appendix 8.4 and for the analysis of citation behavior in Appendix 8.5. Figure 11 summarizes the resulting funnel.

Figure 11.

Overview of the literature review funnel. A keyword search over 18 venues yields 399 distinct works after deduplication, which relevance screening reduces to a core set of 160 works. From this set, extended through backward and forward snowballing, we extract the works surveyed in Sections 3 and 5 and apply the overview-table criteria that select the 29 systems in Table 1 and the 23 benchmarks in Table 2.

8.3.1 Identification

We consider works published between 2021 and May 2026 at the venues most relevant to OpenTI across machine learning, information retrieval, human-computer interaction, database research, and natural language processing. We begin in 2021, the point from which work relevant to OpenTI accelerates markedly (Figure 1a). The venue set is not exhaustive but deliberately targets the leading venues of the communities we identify in Section 1 as contributing towards OpenTI, selecting the top venues of each to capture the most influential and representative work. To these we add the TRL workshops, which sit precisely at the intersection of tabular data and machine learning that OpenTI occupies and thereby concentrate directly relevant work that the general-purpose venues disperse. The search spans the following 18 venues.

Machine Learning. NeurIPS, ICML, ICLR, JMLR, TMLR
Information Retrieval. SIGIR, ECIR
Human-Computer Interaction. CHI, UIST
Database Research. VLDB, SIGMOD, EDBT
Natural Language Processing. ACL, EMNLP, NAACL, EACL
Tabular. TRL Workshop@NeurIPS/ACL, AI for Tabular Data Workshop@EurIPS

We extract the titles and abstracts of all papers published at these venues and organize them in a database. We then run seven keyword searches with diverse queries, mirroring the fragmentation of vocabulary across these communities. The queries target different labels under which communities frame facets of OpenTI, spanning data analysis agents, table understanding and reasoning, table retrieval, table question answering, and text-to-SQL. Table 6 reports the queries together with the number of matches each returns and the number of works from it judged relevant during the screening described below. Deduplicating the matches across queries yields a pool of 399 distinct works that enters screening.

Table 6.

Keyword queries run over the titles and abstracts, with the number of matches returned and, of these, the number of works judged relevant to the survey during screening (Section 8.3.2).

Query	Matches	Retained
"data analysis" AND ("agent" OR "automated" OR "LLM")	91	25
"data science" AND "table" AND ("agent" OR "automated" OR "LLM")	13	6
"open-domain" AND "table"	81	15
"table understanding" OR "table reasoning"	110	36
"table retrieval"	20	11
"table question answering" OR "TableQA" OR "(TQA)"	62	39
"text-to-sql"	206	54

8.3.2 Screening and Selection

We screen the pool of 399 works by assessing each title and abstract for relevance to the subject of the survey. We retain a work if it informs the treatment of OpenTI, which we operationalize through three criteria. A work qualifies if it (1) describes systems, components of systems, or assessments of systems that fall within the end-to-end scope of OpenTI, (2) proposes a benchmark or evaluation that targets directed insight needs over tabular data such as table question answering, text-to-SQL, or data analysis, or (3) concerns user-system interaction relevant to OpenTI. We further retain works that contribute neither a system nor a benchmark but that inform the conceptual treatment of the subject. Deduplicating the relevant works across queries yields a core set of 160 works, which provide the backbone of the literature review and inform its writing throughout.

From this core set we isolate the works that receive detailed treatment in the separate surveys of systems and benchmarks, extracting the works that propose systems spanning the functional capabilities of OpenTI agents and the works that propose benchmarks targeting directed insight needs over tabular data. We extend each of these two sets separately through targeted backward and forward snowballing, adding works with substantial contributions that fall outside the venues, date range, or queries defined above. This adds 8 works to the systems and 13 to the benchmarks, yielding 58 works covering methods and 42 works covering benchmarks that form the basis for the surveys in Sections 3 and 5 respectively.

8.3.3 Selection for the Overview Tables

From these works we select a subset for detailed presentation in the overview tables of Section 3 (Table 1) and Section 5 (Table 2), applying criteria aimed at presenting the most relevant works concerning the scope of OpenTI.

For methods, we analyze each system along the five functional capabilities established in Section 3 and score its coverage of each as fully, partially, or not covered. We include a system if it covers at least three capabilities fully or four at least partially, selecting systems covering a significant extent of the scope of OpenTI. We further require that a system produces some output beyond raw results or code, so that it exercises at least partial freedom in contextualizing and presenting the analytical knowledge it derives. This yields the 29 systems presented in Table 1.

For benchmarks, we first exclude non-English datasets, which we cannot fairly assess against English-language ones. We then score the remaining datasets on the degree to which they cover the data need $i_d$ , the methodological need $i_m$ , and the output need $i_o$ , assigning each component a score from 0 to 2 following the rubric in Table 7. We include datasets with a total score of at least 3, ensuring that the presented datasets cover substantial parts of the scope of insight needs, which yields 18 datasets. We supplement these with five datasets that fall below the threshold but are widely used and highly cited, whose inclusion readers would likely expect as points of reference, namely Spider , BIRD , OTT-QA , NQ-Tables , and FeTaQA . This results in the 23 benchmarks presented in Table 2.

Table 7.

Rubrics for scoring the degree to which benchmarks cover data need $i_d$ , methodological need $i_m$ , and output need $i_o$ . Insight types are counted following the taxonomy in Section 2.3. A dataset is presented in Table 2 if its scores sum to at least 3.

Need	Scored by	0	1	2
$i_{d}$	Data setting	Only the relevant tables provided per instance	A set of tables provided per instance	Open setting
$i_{m}$	Insight types covered	Shallow descriptive only (lookup, aggregation)	At least three insight types covered	Any inferential or causal type covered
$i_{o}$	Output modalities	Single bare modality (value or table)	Two modalities or text synthesis	Outputs include charts

8.3.4 Limitations

The results and discussion presented in this paper have to be read as claims about the surveyed works, which are limited in scope by the review methodology. The identification stage covers 18 peer-reviewed venues, so works published solely on preprint servers or at venues outside this set enter the review only through snowballing, which may underrepresent the most recent developments in this fast-moving space. Additionally, commercial and organizational systems are excluded, since their internals are not documented in a form that can be assessed against our framework, although several deployed products plausibly attempt parts of OpenTI. Our protocol departs from full PRISMA compliance. Notably, neither the search nor the inclusion criteria were registered ahead of time, so the criteria in Sections 8.3.2 and 8.3.3 were refined alongside our developing understanding of OpenTI rather than fixed in advance, reflecting that OpenTI is currently emerging from research.

8.4 Input Classification Methodology

The analysis of data privilege and insufficient specification reported in Section 5.2.3 (Figure 10) adopts the LLM-judge setup of Gomm et al. unchanged, applying their classifiers to the benchmarks in Table 2. We classify the 18 benchmarks whose inputs are self-contained natural language utterances for single-input interaction, excluding five benchmarks for which this does not hold. TACO and DA-Dataset are excluded because their inputs are not available to us, ARCADE and ConDABench because their step-wise inputs are incompatible with the LLM-judge setup, and similarly BLADE because its inputs follow a multiple-choice format over code rather than a natural language expression of an insight need. For each remaining benchmark we randomly sample 500 input definitions, or all inputs for benchmarks with fewer instances, and classify each sampled input individually. The full prompts and implementation are available in the repository accompanying Gomm et al. .

Data privilege is assessed along three dimensions: references to structural elements such as column headers, references to non-public values specific to the underlying data, and references to data containers such as files or tables. To stabilize the classification, we apply self-consistency, classifying each input five times and assigning the final label by majority vote requiring at least three identical votes. An input counts as data privileged if at least one dimension is flagged, where for value references both the explicit and the obscure label count as a flag. For this classification we employ gpt-5-mini-2025-08-07, following the original setup by Gomm et al. .

Sufficient specification is assessed by two classifiers covering the five specification dimensions of Gomm et al. , with data specification comprising entity, temporal, and domain specification and procedural specification comprising intent and methodological specification. The per-dimension labels are aggregated into boolean flags following their criteria, under which temporal specification also admits underspecification assuming recency or inapplicability, and domain specification admits underspecification assuming a universal domain. An input is fully specified if all five flags hold and insufficiently specified otherwise. We use gpt-5-2025-08-07 as an LLM-judge.

The validity of the LLM-judges is thoroughly established by Gomm et al. . To verify that the setup generalizes to our selection of benchmarks, we replicate their annotation study on a stratified sample of 168 annotations over the 18 classified benchmarks, following their validation protocols. For data privilege, a data-science expert independently annotates each sampled input along three dimensions of data privilege (schema, value, and container privilege), and we report the agreement between these annotations and the judge in Table 8. For sufficient specification, we follow the expert-corrected protocol of Gomm et al. , in which the expert corrects the labels emitted by the judge and we report the agreement between the initial and the corrected labels in Table 9. Gomm et al. adopt this correction-based protocol because resolving underlying ambiguities requires broader world knowledge than a single independent annotation pass affords. We report both raw agreement and Cohen’s $\kappa$ , which corrects for the agreement expected by chance.

Table 8.

Agreement between the data privilege judge and an independent expert annotation. We report raw agreement (%) and Cohen’s $\kappa$ .

Privilege dimension	Raw	$κ$
Schema privilege	0.881	0.606
Value privilege	0.911	0.585
Container privilege	0.946	0.760
Data privilege (agg.)	0.893	0.747

Table 9.

Agreement between the sufficient specification classifications and expert-corrected labels. We report raw agreement (%) and Cohen’s $\kappa$ .

Specification dimension	Raw	$κ$
Entities	0.924	0.838
Temporal	0.906	0.786
Domain	0.953	0.904
Intent	0.971	0.861
Methodology	0.929	0.850
Full specification (agg.)	0.953	0.869

In this annotation study, we find a substantial agreement of $\kappa = 0.747$ on the aggregated data privilege judgments and an almost perfect agreement of $\kappa = 0.869$ on the aggregated judgments of sufficient specification, following the scale of Landis and Koch . The raw agreement scores are in line with those reported by Gomm et al. , indicating that the judges generalize to the benchmarks we assess.

8.5 Citation Analysis

To support the claim that works relevant to OpenTI are organized by the task label under which they frame their contributions instead of the problem they address, and that works converging on similar system designs frequently remain unaware of one another, we analyze how surveyed works cite each other against what label they use to describe the task they target. In this citation analysis we construct a citation graph among the surveyed works, assign each work the task label it uses to frame itself, and measure whether citation is assortative on that label. We analyze the 181 works that we surface in our literature review, including the works identified through forward and backward snowballing.

8.5.1 Citation Graph Extraction

We extract the reference list of each work from its PDF with grobid , yielding 8,473 references across 180 of the 181 works. We match references against the surveyed works by comparing normalized titles. We apply a hierarchical matching procedure, first matching the full title, then matching only the title preceding any subtitle, and then fuzzy matching with a similarity threshold of $0.85$ . Since many titles begin with a short system name, we accept a match on the shortened title only when the first authors of both works match. In total, we identify 959 edges between the papers, of which $96\%$ come from an exact title match.

We additionally employ Semantic Scholar and OpenAlex to compare and complement our extraction, which yield 809 (Semantic Scholar) and 109 (OpenAlex) edges between surveyed papers respectively. We add the 40 edges not extracted through grobid, giving a final citation graph with 999 edges.

8.5.2 Assigning Task Labels

We assign each work one of three labels, namely text-to-SQL, table question answering, and data analysis agents. These are the three framings under which the surveyed works predominantly present themselves. Works framed around table retrieval are assigned the label of the downstream task they serve, since retrieval is a functional capability rather than a task in the sense of OpenTI (Section 3.3.2).

To label how each paper frames the task it addresses we use a multi-step process. First, we automatically assign a label to each paper by extracting how often the label, or a derivation of it, occurs in the title and the abstract of the papers, assigning the label that appears most often, not assigning a label if two labels appear the same number of times or if none appears. We cross-check the assignment by an independent classifier that labels the same titles and abstracts against a written codebook, using LLM-based classification with a majority vote over three samples, using Qwen3.5-9B as LLM. The two procedures agree on $86.2\%$ of the works, corresponding to a substantial agreement of $\kappa = 0.799$ . We manually review the 37 works on which they disagree, or for which neither procedure finds a task term, to correct any remaining inconsistencies. This yields 85 table question answering works, 54 text-to-SQL works, and 33 data analysis agent works, while 9 works are not assigned to any task label.

8.5.3 Analysis

We measure assortativity over feasible citations as pairs of works $(A, B)$ for which $A$ could have cited $B$ , meaning that $B$ is published before or in the same year as $A$ . Restricting to these feasible pairs removes the differing sizes of the label groups and the age distribution of the works as confounds, and yields 18,243 pairs. We compare the citation rate within labels against the rate across labels and test the ratio between the two by permuting the labels across works over $10{,}000$ draws.

Results

Works cite within their own label in $10.6\%$ of the feasible pairs and across labels in $1.8\%$ , a ratio of $6.02$ , significant at $p < 0.001$ . Looking into the surveyed systems specifically, an increase in the number of functional capabilities shared between two systems raises the citation rate within a label from $5.6\%$ to $18.5\%$ while the rate across labels stays at or below $3.0\%$ (Figure 6b), so systems converging on similar designs are no more likely to cite each other across a label boundary than systems sharing no capability at all.

Importantly, the absence of a citation is evidence of non-engagement rather than proof that works are unaware of each other. Nonetheless, these results show that literature engages less rigorously with prior works across task labels than within them. Additionally, around a quarter of the works we label under data analysis agents cite none of the surveyed works at all, so the corresponding row of Figure 6a reflects not just the isolation of that literature but also the reach of our review.