5 Evaluations in OpenTI

In this section we understand assessing the gap between a system’s theoretical capabilities and its practical utility as the primary objective of evaluation. This requires systematically quantifying a system’s strengths and weaknesses through a robust methodology that mirrors the complexities of real-world data interaction.

Formally, an evaluation benchmark $B = (T, \mathcal{E})$ consists of a corpus of tables $T$ and a set of $n$ evaluation instances $\mathcal{E} = \{e_1,\ldots, e_n\}$. Each evaluation instance $e = (\bar{i}, v)$ comprises an input $\bar{i}$, representing the expression of an insight need $\hat{i}$, and a validation mechanism $v: \mathcal{R} \to [0,1]$ that scores the quality of a system’s realization. In most current benchmarks, $\bar{i}$ is a single natural language utterance and $v$ reduces to comparing the output against a predetermined reference output. However, more sophisticated instantiations can capture interactive input protocols and nuanced validation criteria that assess not only the final output but also the analytical process by which it was derived.

This section first provides an overview of the surveyed benchmarks (Table 2), and then analyzes them along their components. We systematically analyze the benchmarks that we find in our literature review and select a subset of benchmarks relevant for OpenTI that we present in the overview in Table 2. The methodology for the survey and the selection criteria are detailed in Appendix 8.3. Section 5.1 examines the corpora underlying the benchmarks, Section 5.2 analyzes inputs, and Section 5.3 investigates validation mechanisms. Each section lays out key desiderata for the respective component before analyzing how benchmarks address them.

Table 2 gives an overview of the analyzed benchmarks, presenting the characteristics of their corpora, inputs, and validation mechanisms. The benchmarks span a wide range of scope and complexity, from single-table question answering over web tables to multi-step analytical workloads over relational and file-based corpora.

5.1 Corpora

The corpus $T$ in a benchmark $B$ defines the data environment on which systems are evaluated. In current benchmarks, corpora range from collections of thousands of web-extracted tables to smaller sets of analytical datasets sourced from repositories like Kaggle or GitHub. These corpora differ substantially in their size, the complexity of their tables, and the degree to which they reflect real-world data environments.

5.1.1 Desiderata

To simulate an evaluation environment representative of real-world data environments, $T$ should mirror the complexity and scale of real-world data sources. Thus, a corpus suitable for evaluating OpenTI systems should satisfy the following desiderata:

• Realism and Scale. Many academic datasets are built on tables extracted from web pages, which are typically small (Nan, L et al., 2022) (Chen, W et al., 2021). In contrast, representative organizational or scientific tables often have significantly more columns and rows (Li, K et al., 2017) (Hulsebos, M et al., 2023) and exhibit different distributions of semantic column types (Hulsebos, M et al., 2023). A benchmark should therefore feature a large-scale corpus (Lai, E et al., 2026) (Mitsopoulou, A & Koutrika, G, 2025) containing numerous, sizable tables from diverse domains to properly test a system’s ability to operate in a realistic open setting.
• Messiness. Real-world data is rarely pristine. A realistic benchmark should include tables characterized by a degree of messiness, such as missing values, inconsistent formatting, extraneous information, or varying schema quality (Lai, E et al., 2026) (Mitsopoulou, A & Koutrika, G, 2025) (Dutta, A et al., 2026). This ensures the evaluation tests not only a system’s analytical capabilities but also its essential data preparation and cleaning abilities (Chen, M et al., 2025).
• Domain Diversity. In the open setting systems must generalize across heterogeneous sources and topics. A corpus dominated by a narrow domain like sports statistics may overestimate performance by allowing systems to exploit specific world-knowledge or domain-specific patterns (Gan, Y et al., 2021). Corpora should span multiple domains to assess whether systems can transfer capabilities across different subject areas and data conventions.
• Structural Diversity. Tables have varied structures, including wide versus tall layouts (Döhmen, T et al., 2024), normalized versus denormalized schemas (Kohita, R, 2025), and different levels of metadata availability. A corpus with structural diversity enables testing different integration challenges, particularly for insight needs that span multiple tables (Kohita, R, 2025).

5.1.2 Analysis of Corpora

Current benchmark corpora exhibit a clear trade-off between the number of tables in the associated corpus $|T|$ and the structural complexity of the tables in these corpora. This becomes apparent when considering the #Tab and Source columns of Table 2 together with the distribution of row and column counts across individual tables, depicted in Figure 9. Corpora derived from Wikipedia web tables, such as NQ-Tables (Herzig, J et al., 2021), OTT-QA (Chen, W et al., 2021), FeTaQA (Nan, L et al., 2022), Open-WikiTable (Kweon, S et al., 2023), and MultiTableQA (Zou, J et al., 2026), offer large scale, often containing tens of thousands of tables. However, the individual tables are predominantly small and clean, with the overwhelming majority featuring fewer than ten columns and rarely exceeding a few hundred rows, intended for presentation rather than analysis. They thereby provide scale at the expense of structural realism and diversity. Additionally, it can be assumed that the overwhelming majority of large language models have been exposed to Wikipedia contents during training (Yu, J et al., 2024), potentially leading to data contamination and a better contextual understanding of Wikipedia contents (Ni, S et al., 2025), which can have downstream impacts on the measured performance and the domain generalization capabilities beyond Wikipedia tables.

Corpora sourced from relational databases and analytical repositories exhibit more realistic table characteristics at the cost of scale. Relational corpora such as Spider (Yu, T et al., 2018) and BIRD (Li, J et al., 2023) extend to far wider row distributions, with individual tables reaching hundreds of thousands of rows, while data-analysis corpora drawn from Kaggle, GitHub, and scientific sources, such as DA-Code (Huang, Y et al., 2024), InfiAgent-DABench (Hu, X et al., 2024), and KramaBench (Lai, E et al., 2026), exhibit the widest spread in both rows and columns, more closely reflecting the complexity of tables used in actual analytical workflows. However, these corpora are substantially smaller, containing hundreds rather than tens of thousands of tables, which limits their utility for evaluations in an open setting for table discovery and retrieval. KramaBench (Lai, E et al., 2026) is a notable exception in coupling raw, realistic data with a data-lake organization spanning multiple formats, though it too remains modest in its number of tables.

Beyond the tables themselves, benchmarks contain diverse additional data, reported under Supplement Data in Table 2. Some establish their corpus as relational data (Yu, T et al., 2018) (Li, J et al., 2023) (Lei, F et al., 2025) (Deng, C et al., 2026) (Xu, W et al., 2025) (Wu, J et al., 2025), combining tables with titles and relationships in the form of foreign-key relationships. Next to such relations, many benchmarks provide further contextual metadata associated with tables, like contextual descriptions (Nan, L et al., 2022) (Chen, W et al., 2021) or documentation files (Huang, Y et al., 2024) (Lai, E et al., 2026) (Egg, A et al., 2025). Such contextual information may be used to bridge the semantic gap between tabular data and user inputs to aid functional capabilities like interpretation, data retrieval (Gomm, D & Hulsebos, M, 2025), and analytical composition and execution (Egg, A et al., 2025).

The characteristics of the corpora largely follow from where they are sourced rather than from deliberate design. Realism and scale stand in direct tension, as most large-scale corpora are sourced from Wikipedia, yielding tables that do not resemble those in organizational corpora. Exceptions to this are KramaBench (Lai, E et al., 2026), which pairs raw scientific data with a data-lake organization spanning multiple formats, but at a modest number of tables, and TACO (Deng, C et al., 2026), which has a large corpus of 13,000 tables from municipal and federal open data portals but parses and cleans the raw tables into normalized relational databases. Messiness is thus rarely treated as a design property. Synthetically constructed corpora (Yu, T et al., 2018) (Egg, A et al., 2025) (Lei, F et al., 2025) are clean by construction, and corpora assembled from existing data are commonly filtered and standardized during curation, leaving only a few benchmarks that deliberately retain dirty data (Li, J et al., 2023) (Huang, Y et al., 2024) (Dutta, A et al., 2026) (Lai, E et al., 2026). The domains a corpus covers are likewise a byproduct of its source. Corpora sourced from Wikipedia or open data portals offer a broad range of domains, whereas the domain coverage is narrow in benchmarks targeting one specific analytical setting (Egg, A et al., 2025) (Xu, W et al., 2025) (Lei, F et al., 2025), so that the corpora spanning domains broadly are the same web table corpora with the least structurally realistic tables.

5.2 Inputs

The input $\bar{i}$ defines part of the environment for evaluated OpenTI systems, comprising the user inputs that are observable. Following the framework established in Section 2, $\hat{i}$ is the expressed form of the user’s latent insight need $i$ communicated in user-system interaction, comprising data, methodological, and output components. The input $\bar{i}$ defines the protocol under which $\hat{i}$ is emulated in the evaluation. Since potential interaction modes range from single inputs to mixed-lead collaborative interactions, requirements on what $\bar{i}$ needs to capture and how interaction should be emulated vary significantly between interaction modes. Benchmarks are usually proposed targeting specific interaction modes. In single-input settings, $\bar{i}$ can be a standalone natural language utterance; in interactive settings, it may comprise a sequence of exchanges or a protocol for responding to system-initiated clarification. Benchmarks vary considerably in the analytical complexity of insight needs, the degree to which inputs are suitable for evaluations under an open premise, and whether they support interaction beyond single utterances.

5.2.1 Desiderata

Inputs suitable for evaluating OpenTI should satisfy the following desiderata:

• Data-Independence. In the open setting, users formulate insight needs without knowledge of specific data structures or contents (Voorhees, E, 2001). Authentic inputs should thus be data-independent, formulated from the user’s conceptual perspective, without using privileged knowledge of the underlying corpus a user does not have. Data-privileged inputs, by contrast, reference structural elements such as column headers (e.g., “last_name”), specific values not in the public domain (e.g., “the airplane that took off at 2026-05-08T11:48:11+00:00”), or data containers themselves (e.g., “the user table”). Such references provide unrealistic signals linking queries to specific data structures, undermining the premise of an open corpus (Gomm, D et al., 2025) (Suhr, A et al., 2020) (Lee, C et al., 2021).
• Sufficient Specification. As argued in Section 2.1.2, insight needs may exhibit a degree of latency where users do not exhaustively articulate them but provide what they judge necessary and rely on systems to resolve the rest. What counts as sufficient specification therefore depends on the validation mechanism used to score a realization. When paired with a validation mechanism that admits only a single reference output or realization, inputs must fully determine the required data and methodology, since alternative interpretations would produce divergent but plausibly correct responses that the validation cannot accommodate. Validation mechanisms that accept multiple valid realizations allow inputs to leave ambiguities in corresponding aspects. Misalignment between the degree of ambiguity in inputs and the validation mechanism can conflate accuracy with interpretation capabilities, yielding unreliable signals (Bhaskar, A et al., 2023) (Saparina, I & Lapata, M, 2025) (Gomm, D et al., 2025).
• Analytical Complexity. Real-world analytical workloads often require multi-step reasoning, involving sequences of operations that build upon intermediate results (Gu, K et al., 2024) (Wu, X et al., 2025) (Lai, E et al., 2026). Inputs should reflect this complexity to test whether systems can plan and execute extended analytical procedures.
• Multi-Table Sources. Real-world questions often require integrating information from multiple tables. Therefore, a significant share of the evaluation instances should require retrieving and integrating information from multiple tables (Lai, E et al., 2026) (Wu, J et al., 2025) (Akhtar, M et al., 2025).

5.2.2 Input Protocols

Depending on how inputs are provided, evaluation protocols fall into two categories: static and user simulation. Static protocols fix all inputs upfront in the benchmark itself, while user simulation protocols define the mechanisms by which inputs that depend on the analytical state or system elicitation are determined.

In their simplest and most common form, static protocols define a single standalone utterance per evaluation instance to emulate single-input interactions, which allows tight control over the input and a consistent, replicable evaluation environment. Similarly, for step-wise interaction a multi-turn sequence of inputs can be fixed upfront (Yu, T et al., 2019) (Yin, P et al., 2023) (Liu, C et al., 2023) (Li, L et al., 2025). While such sequences emulate a multi-step process, the predetermined path does not adapt to the intermediate analytical state a system exposes at each step, trading the realism of a genuinely contingent interaction for replicability.

User simulation protocols instead generate inputs dynamically, emulating a user who provides additional inputs proactively based on observations of the analytical state and/or reactively in response to systems requesting inputs. User simulation has a long lineage in information access research (Balog, K & Zhai, C, 2024). It generally centers on a model of the user (Balog, K & Zhai, C, 2024), which can be based on rule-based and agenda-based simulators (Schatzmann, J et al., 2007) that generate inputs by reacting to system outputs according to an explicit representation of user goals, or an end-to-end neural architecture (Gür, I et al., 2018) (Lin, H et al., 2021) that learns the behavior. LLMs have been proposed as end-to-end user models (Sekulić, I et al., 2022) (Sekulić, I et al., 2024) (Shao, Y et al., 2026). Empirical assessment finds that such models correlate well with human judgments overall (Dou, Y et al., 2025), but cautions against possible goal misalignments, particularly in multi-turn contexts (Zhang, Z et al., 2020) (Mehri, S et al., 2026), and against unrealistic response patterns (Seshadri, P et al., 2026), against which scaffolding techniques have been proposed (Mehri, S et al., 2026).

5.2.3 Analysis of Inputs

Current benchmarks overwhelmingly adopt static input protocols for single-input interaction, whereas a smaller set fixes multi-turn inputs upfront to emulate step-wise interaction (Yu, T et al., 2019) (Yin, P et al., 2023) (Li, J et al., 2025) (Liu, C et al., 2023) (Li, L et al., 2025). User simulation remains rare, applied only in ConDABench (Dutta, A et al., 2026), BIRD-INTERACT (Huo, N et al., 2025), and COTA (Li, J et al., 2025). They all employ an LLM to simulate the user but differ in how they align it with the underlying insight need $i$. COTA (Li, J et al., 2025) grounds the user simulator in an extensive description of the insight need, decomposed into a set of sub-questions, as well as a sample of the table contents. BIRD-INTERACT (Huo, N et al., 2025) and ConDABench (Dutta, A et al., 2026) provide the schema of the data and directly expose the desired analytical operations as SQL and Python code to the LLM, while prompting it to use the code to inform interactions but not to share it, risking leakage of information the system did not ask for.

As shown in Table 2, the benchmarks focus mostly on descriptive insights, with little coverage of inferential and causal insights. Lookups and aggregations have the broadest coverage, and many benchmarks do not go beyond these two types. Some benchmarks remain restricted to simple $\min$ and $\max$ aggregations that do not require calculations (Chen, W et al., 2021) (Nan, L et al., 2022) (Herzig, J et al., 2021), allowing direct inference over the full table to derive answers from the raw data. In comparison, most benchmarks that expand to more complex aggregations, characterization, and association, which require more complex numerical operations, are rooted in data analysis workflows (Huang, Y et al., 2024) (Hu, X et al., 2024) (Lei, F et al., 2025) (Wu, X et al., 2025) (Gu, K et al., 2024) and are mostly absent from the question answering and text-to-SQL benchmarks. At the inferential tier, statistical inference is covered as an explicit analytical objective in BLADE (Gu, K et al., 2024), DA-Code (Huang, Y et al., 2024), InfiAgent-DABench (Hu, X et al., 2024), and CausalReasoningBenchmark (Sawarni, A et al., 2026). Prediction is covered more widely but to different extents. DA-Code (Huang, Y et al., 2024) and InfiAgent-DABench (Hu, X et al., 2024) contain inputs for constructing predictive models without seeking a specific insight, whereas TableBench (Wu, X et al., 2025), KramaBench (Lai, E et al., 2026), and ConDABench (Dutta, A et al., 2026) target simple forecasts through regression. TOPBench (Ji, A et al., 2026) is the first dataset that centers on predictive insights, verbalizing complex predictive needs that require building and applying models to extract the insights. Causal insight needs have been covered on a shallow level as counterfactual insights in Archer (Zheng, D et al., 2024) and DABstep (Egg, A et al., 2025), which explore counterfactual scenarios that are resolved under the application of provided static rules. Only CausalReasoningBenchmark (Sawarni, A et al., 2026) targets genuine interventional analysis, requiring formal observational identification strategies such as instrumental variables, regression discontinuity, and difference-in-differences alongside statistical estimation.

Beyond the covered insight needs, input definitions across benchmarks vary widely in the data requirements they impose. Many benchmarks, particularly those focused on question answering, simplify the problem setting by requiring evidence from a single table only, abstracting away the real-world complexities of data integration that define the full scope of OpenTI. The Tbl/Inst. column of Table 2 showcases this, with 9 of the 23 benchmarks assuming that either a single relevant table or a set of relevant tables is directly provided to a system, while only 6 target an open setting that necessitates retrieval. The remainder restrict selection to a bounded set, mostly a single database. Similarly, the complexity of the insight needs expressed in the input definitions is unevenly distributed across benchmarks. We report the average words per input definition as a proxy for the complexity of insight needs in the Words column of Table 2, observing a large range from an average of 8.9 words for inputs in NQ-Tables (Herzig, J et al., 2021) to 108.8 words in TOPBench (Ji, A et al., 2026), whose intent-rich queries embed the analytical scenario in a narrative description. However, while longer inputs generally allow for more detailed instructions, word count alone fails to capture the latency in expressions of insight needs, which allows insight needs to be compressed into shorter forms under greater latent assumptions and delegation. DABstep (Egg, A et al., 2025) and Spider 2.0 (Lei, F et al., 2025) are deliberately designed so that the expressed insight needs require multiple analytical steps that build on intermediate results, whereas most other benchmarks can be served by one-shot code generation.

Next to these structural properties, evaluating OpenTI systems in an open setting requires that realistic inputs are formulated independently of the exact data that serves as evidence while providing sufficient specification of the methodological and data need in the expression of the insight need. We assess the degree to which these desiderata are violated following the LLM-judge setup proposed by Gomm et al. (2025), with results shown in Figure 10, detailed in Appendix 8.4. Benchmarks vary significantly in the data privilege found in inputs. The analysis reveals high data privilege primarily in data-analysis benchmarks like InfiAgent-DABench (70.0%), DABstep (78.4%), and TOPBench (85.0%), whose inputs reference specific columns and files, limiting the degree to which evaluations in open settings capture realistic interactions. On the other end of the spectrum, NQ-Tables (0.0%) and FeTaQA (0.4%) have negligible levels of data privilege. In addition, all benchmarks show high levels of insufficient specification of inputs. All benchmarks but NQ-Tables (31.6%) and FeTaQA (47.4%) contain more than 50% of inputs that are insufficiently specified. While validation mechanisms based on similarity or LLM judgment may absorb some of this ambiguity, the dominance of exact-matching validation in surveyed benchmarks (Section 5.3.2) means that the bulk of insufficiently specified inputs are paired with validation that admits only one valid interpretation, conflating execution accuracy with the ability of systems to guess the intended interpretation.

Overall, inputs across the benchmarks remain limited for evaluating OpenTI systems in an open setting. Since most benchmarks are not designed for an open setting, many of their inputs exhibit high levels of data dependence and insufficient specification. The analytical complexity is bounded, with the most widespread focus on lookup and aggregation types, whereas inferential and causal insight types remain only sparsely covered. In contrast, many benchmarks include inputs that require combining evidence from multiple tables to address the insight need, though most do so in a data setting where a small set of tables is provided and the set of relevant tables does not have to be selected from a large corpus.

5.3 Validation Mechanisms

The validation mechanism $v: \mathcal{R} \to [0,1]$ determines how evaluation assesses the quality of a system’s realization against the user’s insight need. Ideally, validation would assess to which degree the complete realization $r = (O, E, \beta)$, consisting of the operations performed, their composition, and the data to which they are bound, constitutes a sound derivation of the analytical knowledge that addresses the user’s insight need. In practice, the validation target ranges from the terminal output $\text{out}(r)$ produced by the realization to components of the realization $r$ itself.

Connecting this to the problem formulation (Section 2.1.5), $v$ acts as an operational stand-in for the validity criterion captured by $R_{\text{valid}}(i)$ and the utility function $u_i$. Since the underlying insight need is fixed by the benchmark’s creators, the validation mechanism encodes their judgment of which realizations satisfy the need and how well. An ideal validation mechanism would take the form $v(r) = \mathbf{1}[r \in R_{\text{valid}}(i)] \cdot g(u_i(r))$, where $g$ maps utility monotonically into $(0, 1]$, with its exact form immaterial since the objective in Equation 2.1.5 depends only on the ordering that $u_i$ induces. Realizations that fail to address the insight need score zero, while valid realizations are scored by how well they serve it. In practice, validation in existing benchmarks primarily approximates the validity indicator, leaving the quality of realizations and outputs largely unjudged.

We separate validation functions that operate on the output level (Section 5.3.2) from those that operate on the realization level (Section 5.3.3), and consider combined functions (Section 5.3.4). Output-level validation compares the final output $\text{out}(r)$ against reference outputs or success criteria, without assessing the validity of the analytical process itself. In contrast, realization-level validation examines components of $r$ directly, assessing the analytical process itself rather than only its results. Table 3 provides an overview of the validation functions we discuss in this section. We focus on functions that score the end-to-end validity of an analysis or its output, setting aside metrics that isolate a single functional capability, such as the recall of the retrieval capability (Herzig, J et al., 2021) (Zou, J et al., 2026).

5.3.1 Desiderata

Validation mechanisms suitable for evaluating OpenTI should satisfy the following desiderata:

• Scalability and reproducibility. Human evaluation offers flexibility in assessing complex, open-ended outputs but is costly, slow, and difficult to reproduce consistently. Instead, a practical validation mechanism should be automated to ensure reproducibility, scalability, and objectivity, requiring minimal human intervention (Gu, K et al., 2024).
• Multi-modal coverage. Analytical results can be presented in different modalities like natural language, numerical results, tables, and visualizations. The validation mechanism should thus be able to handle the modalities relevant to the evaluated insight needs (Zhang, D et al., 2025) (Huang, Y et al., 2024).
• Robustness and faithfulness. Metrics should reliably quantify what they claim to measure, providing signal rather than noise. This is particularly challenging for open-ended analytical outputs, where traditional text similarity metrics such as BLEU, ROUGE, and BERTScore have been found to correlate poorly with actual response quality and to not reliably separate factually accurate responses from those that are fluent yet factually incorrect (Wolff, C & Hulsebos, M, 2025) (Wang, Y et al., 2024). Validation mechanisms must be calibrated to the specific characteristics of analytical outputs to yield meaningful performance estimates.
• Acceptance of valid variation. A single insight need may admit multiple valid realizations producing different but equally valid outputs (Zeng, Q et al., 2025), whether through alternative analytical methods, different but appropriate data selections, or varied presentation choices. Validation mechanisms should accommodate this variation rather than penalizing correct responses that diverge from a single gold standard (Gu, K et al., 2024) (Zeng, Q et al., 2025). This requires either curating sets of valid reference outputs or employing validation approaches capable of assessing correctness beyond exact matching.

5.3.2 Output-Level Validation

Output-level validation assesses the final output $\text{out}(r)$ produced by executing a realization $r$ against one or more reference outputs or a set of success criteria. This disregards the underlying analytical procedure captured in $r$, which simplifies automated evaluation but makes it impossible to distinguish correct reasoning from coincidentally correct answers (Liu, X et al., 2024).

Exact matching. Exact matching compares the output against a desired reference output under an equivalence function. Since validity is only confirmed if the system’s output exactly matches the reference output, the output must follow the exact form of the desired output. Different approaches have been explored to constrain the output space. Multiple-choice questions represent the most restrictive approach, limiting valid outputs to a predefined set of options (Lu, P et al., 2022) (Li, J et al., 2025). This makes complex analytical questions easily verifiable, but the few choices may allow systems to take shortcuts and guess instead of deriving the correct output (Balepur, N et al., 2025). Format-prompting similarly constrains outputs by requiring responses to follow specific templates, such as {@answer[value]}, which can be parsed and verified automatically (Hu, X et al., 2024). These constraints simplify validation at the cost of limiting the expressiveness of responses and potentially leaking information about expected answers. Another approach is to parse the outputs into a standardized format before matching. For instance, Huang et al. (2024) parse plotting scripts to extract numerical data and plotting parameters into numpy and JSON formats for comparison. Exact matching also often requires some cleaning of outputs to account for permutations in lists, different formatting (e.g., “16€” vs. “16.00”), and further insignificant variations (Egg, A et al., 2025). Exact matching is the most common validation function used across the analyzed benchmarks. Many benchmarks from tabular question answering are designed to allow only a single value that exactly matches the output (Pasupat, P & Liang, P, 2015) (Lu, P et al., 2022) (Zhang, Z et al., 2023) (Herzig, J et al., 2021) (Kweon, S et al., 2023) (Chen, W et al., 2021) (Zhao, Y et al., 2022) (Zou, J et al., 2026) (Contalbo, M et al., 2025) (Katsis, Y et al., 2022), whereas benchmarks rooted in text-to-SQL generation often employ execution accuracy (Yu, T et al., 2018) (Li, J et al., 2023) (Wang, Z et al., 2025) (Kumar, R et al., 2024) (Chen, P et al., 2025), defining an equivalence function that compares the execution results of SQL statements with expected results, sometimes in a column- or row-order invariant manner. Execution accuracy cannot differentiate between semantically distinct queries that incidentally yield the same result (Klopfenstein, R et al., 2026). Exact matching extends to insight needs that admit multiple valid outputs by crediting a match against any one of them, though this extension is not employed in any analyzed benchmark. To make exact matching more fine-grained, Hu et al. (2024) propose decomposing the output into multiple components, matching each against its own reference value and aggregating the results into a score.

Similarity-based matching. Similarity-based matching relaxes the exact-equivalence requirement by applying a similarity measure to compare a system’s output to a reference output. Different similarity measures have been proposed for individual output modalities, ranging from numeric tolerances that absorb floating-point differences (Egg, A et al., 2025), through fuzzy matching of tables that tolerates additional rows and columns (Yin, P et al., 2023) and image similarity for rendered charts (Li, J et al., 2025), to lexical and semantic overlap measures such as BLEU (Papineni, K et al., 2002), ROUGE (Lin, C, 2004), and BERTScore (Zhang, T et al., 2019) for free-form text (Nan, L et al., 2022) (Wu, X et al., 2025). However, these similarity metrics for free-form text have been found to correlate poorly with human judgments of free-form outputs (Wolff, C & Hulsebos, M, 2025) (Wang, Y et al., 2024). For predictive insight needs, similarity-based matching can be applied to the model used for making the prediction, for instance by calculating the relative performance gap to a well-performing baseline model (Huang, Y et al., 2024) (Jing, L et al., 2025). Similarity-based matching generalizes to sets of reference outputs analogously to exact matching, crediting the best-matching reference.

LLM-as-Judge. To handle genuinely open-ended outputs, recent benchmarks increasingly employ large language models as evaluators, prompted with the output, a reference output, and instructions that specify the evaluation criteria (Gu, K et al., 2024) (Chen, Z et al., 2025) (Xu, W et al., 2025) (Wolff, C & Hulsebos, M, 2025) (Lai, Y et al., 2023) (Lei, F et al., 2025) (Sahu, G et al., 2024) (Dutta, A et al., 2026) (Jing, L et al., 2025). The flexibility of these instructions enables specifying conditions that accept semantically equivalent outputs, multiple valid realizations, or fine-grained success criteria, offering flexibility beyond what surface metrics afford. Judges are structured in different ways to sharpen this signal, decomposing the evaluation criteria into fine-grained rubrics combined into a weighted score (Lei, F et al., 2025), augmenting the instructions with instance-specific context and criteria (Kim, H et al., 2025), or decomposing the output itself into atomic claims verified individually against the tables (Seo, K et al., 2025). TOPBench (Ji, A et al., 2026) further guards against hallucination-induced judgment errors through a multi-step approach that directly matches quotes from the result to expected outputs. A single judge can also span several output modalities at once to account for different ways of presenting the analytical knowledge (Dutta, A et al., 2026). Rather than scoring outputs in isolation, judges are also prompted to compare two outputs against each other, either by expressing a preference between two analyses (Wu, X et al., 2024) or by rating one relative to the other on aspects such as readability, analytical depth, and visualization (Lei, F et al., 2025). While offering greater flexibility, LLM judges may also introduce biases, cost, complexity, and reproducibility concerns (Egg, A et al., 2025). While Wolff and Hulsebos (2025) find a high accuracy of LLM-judges for simple, descriptive insight needs, Hu et al. (2024) only find a 67% consistency between human experts and LLM-judges for more complex insight needs beyond simple description, underscoring the need to validate the alignment of LLM judges with human judgment for the specific setting in which they are applied (Zhu, J et al., 2025) (Wang, Y et al., 2024).

Programmatic validation. Instead of relying on predefined desired outputs, programmatic validation introduces flexible, task-specific success criteria in the form of test cases capturing conditions the output must satisfy (Chen, Z et al., 2025). These test cases are usually expressed in code and can check aspects like verifying that a value falls within acceptable bounds (Chen, Z et al., 2025), or that a derived output exhibits specific features (Lei, F et al., 2025) (Wu, X et al., 2025), or that the output matches any of multiple patterns. Programmatic validation provides unambiguous and reproducible assessment but requires outputs amenable to automated testing and test cases specified per evaluation instance.

Human evaluation. Instead of automated scoring functions, human annotators are employed to rate outputs. Human evaluation can be applied to judge the outputs of individual evaluation instances against explicit criteria (Nan, L et al., 2022) (Yu, T et al., 2019) or for indicating a pairwise preference for one analysis over another (Wu, X et al., 2024) (Lei, F et al., 2025). Human evaluation makes the evaluation more flexible and can adapt to instance-specific particularities or faults outside anticipated criteria, but it introduces cost in terms of scalability and reproducibility, confining human evaluation to relatively small samples (Nan, L et al., 2022) (Yu, T et al., 2019) (Zhang, Z et al., 2023) (Wu, X et al., 2024) or to a fallback where automated matching is brittle, as demonstrated in CRT-QA (Zhang, Z et al., 2023), where it absorbs formatting errors that exact matching would otherwise penalize.

Output-level validation treats the realization as a black box, meaning systems can arrive at correct outputs through flawed analysis or bypass data evidence entirely (Liu, X et al., 2024), and partially correct analyses receive no credit. Realization-level validation addresses these gaps by inspecting components of $r$ directly.

5.3.3 Realization-Level Validation

Instead of narrowly focusing on the final output, realization-level validation assesses the analytical process captured in $r$ itself. It thereby allows distinguishing correct from incorrect analysis, regardless of the result, and allows more fine-grained scoring that can give partial credit to partially correct analysis. The central challenge in realization-level validation lies in surfacing components of $r$ in a form that allows automated checking, either by requiring systems to declare specific artifacts alongside their output or by extracting them from the executed realization.

Program comparison. Similar to similarity-based matching at the output-level, program comparison directly applies a similarity function to compare the operations and their composition captured in $r$ against those in a reference realization $r^*$, where both $r$ and $r^*$ are represented as code programs. The similarity measure can be algorithmic, such as CodeBLEU (Ren, S et al., 2020), or learned, such as CodeBERTScore (Zhou, S et al., 2023). Learned measures can be more permissive of syntactic variation by operating on learned representations of code (Zhou, S et al., 2023). Among surveyed works, algorithmic n-gram measures such as ROUGE and BLEU (Wu, J et al., 2025) (Kumar, R et al., 2024) are applied to compare generated with reference SQL, while COTA (Li, J et al., 2025) applies a learned embedding measure to general analysis code. Program comparison is constrained by its reliance on predefined reference realizations, which conflicts with valid variations in structuring and composing realizations where even semantically equivalent programs can diverge substantially in surface form (Pinna, G et al., 2025). It further requires that the realization is fully expressed as a code artifact, limiting the applicability to systems that emit such programs.

Structural matching. Rather than comparing realizations as textual programs, structural matching operates directly on the abstraction of realizations into operations and their composition, measuring the similarity with a reference realization at that level. In text-to-SQL, structural matching has a long lineage in the form of component matching, which decomposes both the predicted and reference SQL into clauses and compares them, either as an exact set match (Yu, T et al., 2018) (Lee, C et al., 2021) or with partial credit over parsed query trees (Kumar, R et al., 2024), though such comparisons are tied to the fixed clause structure of SQL. BLADE (Gu, K et al., 2024) extends structural matching beyond SQL to general data transformations represented in a graph, with nodes corresponding to discrete data transformations (e.g., filter, groupby, derive,…) and edges expressing data flows between them. It introduces two similarity measures. Value-based matching executes the transformations and compares intermediate column values at corresponding nodes, matching sequences of operations that produce equivalent data regardless of how they are expressed. Graph isomorphism matching compares the topology and node labels of the graph, accepting structurally equivalent transformations that may produce different values. These similarity measures capture complementary aspects, with value-based matching allowing syntactic diversity given identical results and graph matching tolerating differences in values under a consistent analytical structure. Generally, structural matching provides more flexibility than program comparison, but requires the realizations to be surfaced in a form that supports the abstraction. BLADE (Gu, K et al., 2024) addresses this by employing an LLM to convert transformation code into transform units, which introduces an additional source of error and reliability concerns.

Component validation. Component validation decomposes the realization into a set of components and validates each individually against a set of per-instance requirements, aggregating the results into a composite score. Aggregation is commonly an average over the requirements, which allows partial credit, though CausalReasoningBenchmark (Sawarni, A et al., 2026) instead credits a realization only if it satisfies every requirement. Unlike program comparison and structural matching, the realization is not compared to a reference realization, but checks are applied to determine whether specific analytical artifacts are present and correct. Component validation approaches vary in what components they extract from the realization and how they perform checks on top of them. Components may be defined as stages of a predefined analytical pipeline (Chen, Z et al., 2025), or task-specific key functionalities that a correct analysis must contain (Lai, E et al., 2026). This decomposition can also surface how errors propagate through dependent operations by aggregating individual judgments at different steps of the analysis (Lei, F et al., 2025). In contrast, Gu et al. (2024) require agents to declare specific analytical decisions (independent, dependent, and control variables) alongside their code, matching these declarations against a reference through LLM-judged semantic equivalence. Statistical models are similarly extracted from code into natural-language specifications and matched semantically. CausalReasoningBenchmark (Sawarni, A et al., 2026) similarly compares a set of conditions within the realizations with annotations on the desired realization. Through the decomposition of realizations into components, component validation yields fine-grained validation scores that not only indicate the end-to-end performance but also surface detailed information on error cases. Depending on the design of the decomposition and checks, component validation can support different valid realizations (Gu, K et al., 2024). Component validation inherits the reliability concerns of LLM-as-judge approaches. It further requires expensive curation of per-instance component sets, which must be validated to ensure completeness and correctness. The cost of curation scales with the complexity of the tasks and the number of components per task, which has so far limited the scale of benchmarks.

5.3.4 Combining Validation Signals

Validation mechanisms are not mutually exclusive. A benchmark may combine signals from different targets or different mechanisms to capture a broader or more robust validation signal than any single function (Pinna, G et al., 2025), or to support different output modalities with appropriate functions (Wu, X et al., 2025) (Egg, A et al., 2025). Functions can either be aggregated into a single, mixed-target composite score, or deployed separately to collect distinct, multi-criteria validation signals.

Mixed-target composites apply and aggregate the results of different validation scores into a single score, acting as an ensemble of validation functions. Output- and realization-level functions can be combined to capture complementary failure modes, where outputs may be correct despite a divergent realization or the realization may be sound despite differences in outputs (Pinna, G et al., 2025). A composite score simplifies comparisons to a single score at the cost of diagnostic granularity, as a mid-range score does not identify which component failed.

Multi-criteria reporting maintains separate scores from distinct validation functions rather than collapsing them into one and is commonly employed among the surveyed benchmarks (Lai, E et al., 2026) (Ji, A et al., 2026) (Gu, K et al., 2024). This enables a fine-grained attribution of performance while complicating cross-system comparisons, and can deliberately isolate distinct failure modes by scoring realization-level and output-level signals apart to separate errors in the analysis from errors in its execution (Sawarni, A et al., 2026).

Assessed against the desiderata, scalability and reproducibility are met through automatic evaluation protocols that do not rely on human judgment, although the shift towards LLM-based judging reintroduces variance over directly calculated metrics. Coverage of output modalities is narrow, as many benchmarks simplify evaluation by only assessing raw output values or tables, while most others are restricted to text outputs. Only DAComp (Lei, F et al., 2025) evaluates multi-modal outputs that combine text and charts, leaving outputs that combine different modalities mostly uncovered. Robustness and faithfulness are handled by confining outputs to specific formats and by replacing flawed similarity metrics with LLM-based assessments, the faithfulness and robustness of which varies by the specific setup and evaluation design. Overall, validation remains concentrated at the output level. Mechanisms that inspect the realization appear in a minority of benchmarks and are concentrated in text-to-SQL component matching, with genuine assessment of open-ended analytical processes confined to a few works such as BLADE (Gu, K et al., 2024), KramaBench (Lai, E et al., 2026), and CausalReasoningBenchmark (Sawarni, A et al., 2026). Support for multiple valid variations is similarly shallow, as most mechanisms that admit more than one valid realization do so only through output equivalence, where execution accuracy credits results that coincide with the reference, whereas crediting genuinely divergent yet valid realizations or outputs is restricted to component- and rubric-based approaches (Gu, K et al., 2024) (Lei, F et al., 2025) (Lai, E et al., 2026). Validation across benchmarks thus predominantly approximates the validity criterion $\mathbf{1}[r \in R_{\text{valid}}(i)]$ while the actual utility $u_i(r)$ remains largely unmeasured. Even where mechanisms produce intermediate scores, these mostly arise from aggregating validity checks over components of the output or realization, reflecting partial validity. The utility of different valid realizations remains largely unaddressed, only partially surfacing in rubric-based judges that score aspects of data selection, methodological rigor, and the effectiveness of the presentation explicitly (Lei, F et al., 2025) (Lai, E et al., 2026), leaving two systems producing valid realizations of substantially different utility largely indistinguishable under current benchmarks.