6 A Research Agenda for OpenTI

The preceding sections surface a landscape of related but fragmented work making progress towards OpenTI. Yet, none of the reviewed systems addresses OpenTI end-to-end, and significant gaps remain before such systems can deliver practical value. This section puts forward a research agenda towards closing these gaps and advancing OpenTI towards real-world applications.

Successfully implementing (data) systems and democratizing access to insights requires that systems are both reliable and trustworthy (Shneiderman, B, 2020), and that they provide accessible means for extracting insights (Jagadish, H et al., 2014). The gaps that separate current work from these ends come from simplifying assumptions in current methods, missing integration between capabilities, evaluation infrastructure that lags behind the systems we wish to build, and a limited perspective on the role of humans in the insight extraction process. We organize the research agenda following these requirements. To holistically address OpenTI end-to-end, systems must reliably retrieve relevant tables from the corpora available to them and address complex analytical needs that reach beyond descriptive insights into inferential and causal ones (Section 6.1). Further, we argue that OpenTI systems need to be trustworthy so that results can be relied upon and that new user interaction paradigms are required to align the analysis with the actual insight need a user has (Section 6.2). Lastly, we take a step back to rethink how systems are built and evaluated, arguing that realizing OpenTI requires new paradigms for both (Section 6.3).

6.1 Addressing Data and Methodological Needs

The insight needs a system can satisfy are bounded by the data it can locate, corresponding to the data need $i_d$, and by the analysis it can perform, corresponding to the methodological need $i_m$.

6.1.1 Retrieving Compatible and Relevant Data at Scale

Locating sufficient tables from a large corpus is a prerequisite for any downstream operation. As discussed in Section 3.3.2, existing retrieval methods have made progress on alignment between user inputs and individual tables, yet several challenges remain unaddressed.

What makes data relevant? Current retrieval methods predominantly operationalize relevance as semantic similarity between textual representations of data needs and tables, while accounting for structural compatibility (Section 3.3.2). However, relevance and the fit of tables to the specific task are multidimensional. In particular, when multiple sets of tables satisfy semantic and structural criteria, the realizations they enable may exhibit different utility $u_i$. Data quality literature has established such multidimensional relevance frameworks for tabular data (Wang, R & Strong, D, 1996) (Batini, C et al., 2009), highlighting aspects like completeness, consistency, and timeliness. Future work should thus define and operationalize quality-aware relevance criteria and integrate them with retrieval.

How to design indexes that support the discovery of relevant data? Retrieval methods predominantly build indexes as flat collections of independent table, column, or row representations that are scored against the input (Section 3.3.2). Such indexes treat each entry in isolation and remain agnostic to relationships between tables, data quality criteria, and the analytical use the retrieved data will serve. Instead, future work should explore indexes that are aware of the data they are serving, designed for surfacing relevant data rather than just similar data.

How can retrieval scale to heterogeneous real-world corpora? Real-world data environments such as organizational data lakes exhibit large numbers of tables, heterogeneous schemas, inconsistent naming conventions, varying quality, and potentially overlapping content (Nargesian, F et al., 2019). Maintaining retrieval performance requires methods that are robust to the messiness and ambiguity in such environments. Scaling also amplifies the challenge of distinguishing more from less relevant data. Progress on this is limited by evaluation, since current benchmarks have corpora that trade scale against structural realism (Section 5), leaving methods largely untested against the heterogeneity and ambiguity of the environments they are meant to operate in.

6.1.2 Addressing Inferential and Causal Insights

Insights span descriptive, inferential, and causal tiers (Section 2.3), yet reviewed systems and evaluations remain mainly concerned with descriptive insights. Inferential and causal insights require deriving knowledge that reaches beyond the observed data, justified against assumptions about the process that produced the data. Current models show a limited capacity to do so across inferential and causal insights. LLMs mainly rely on causal relationships learned from world knowledge (Kiciman, E et al., 2025) but show limitations in reliably inferring them from evidence (Jin, Z et al., 2024) (Liu, X et al., 2024). While recent reasoning models narrow this gap, they remain susceptible to bias in instructions (Fu, Z et al., 2026). They also show limitations in building predictive models, with reported tendencies to overfit validation signals (Chan, J et al., 2024) and difficulties in identifying methodological errors (Kumar, S et al., 2025). Evaluation is covered unevenly and mostly on a surface level. Within inferential and causal types, prediction is covered most widely, though mostly as a modeling task rather than focused on deriving an insight (Section 5), with TOPBench (Ji, A et al., 2026) the first to center on predictive insights and CausalReasoningBenchmark (Sawarni, A et al., 2026) the first to target interventional analysis, while statistical inference appears mostly as one objective among many (Gu, K et al., 2024) (Huang, Y et al., 2024) and counterfactuals remain limited to simplified rule-based instances (Egg, A et al., 2025) (Zheng, D et al., 2024).

How can systems establish and verify the data and analysis assumptions? Since the validity of inferential and causal analysis relies on assumptions about the process that generated the data, systems should make these explicit and verify them against the data where possible, and surface the rest to the user to judge. The assumptions differ by tier, as prediction rests on the sample being representative of the targets and free of leakage, statistical inference on the distributional and sampling conditions of the applied test, and causal estimation on identification conditions such as the absence of unobserved confounding. Benchmarks like BLADE (Gu, K et al., 2024) take a first step in this direction by requiring systems to declare analytical decisions such as control variables, yet none of the surveyed systems establishes or verifies such assumptions. Treating assumptions as first-class components of the analytical process could also extend interpretability from the provenance of results to their analytical validity (Section 6.2.2). Progress on this is also required to enable realization-level validation of inferential and causal results, which should establish whether a predictive model generalizes beyond its training sample rather than reproducing a stored value, and whether an inferential or causal estimate rests on justified assumptions and appropriate methodology.

How to select a methodology that matches the insight need and the data? Inferential and causal needs require choosing among methods whose validity depends on properties of the selected data. Predictive needs require selecting a model and features, while interventional needs demand formal identification strategies such as instrumental variables, regression discontinuity, and difference-in-differences, depending on the structural conditions (Sawarni, A et al., 2026) (Wang, X et al., 2025). In OpenTI, however, the appropriate methodology depends on retrieval and the data binding $\beta$ it surfaces, that is, whether the corpus supplies the features a model needs or a variable that serves as a valid instrument. Future work should reason jointly over the methodological need $i_m$ and the data need $i_d$, retrieving data that enables valid methods rather than fitting methods to whatever data was retrieved.

6.2 Making Systems Trustworthy and Cooperative

Beyond addressing the full spectrum of insight needs end-to-end, OpenTI systems should do so in a trustworthy manner and ensure that they address the actual need a user has. We thus argue that they need to produce robust realizations whose results users can rely upon, and that systems should be designed for cooperative interactions with users to align realizations with the users’ needs.

6.2.1 Ensuring Robustness of Results and Realizations

OpenTI systems are intended to provide factual information grounded in data to inform decisions. To make outputs trustworthy for real-world decision-making, systems need to either produce reliable results or transparently communicate their limitations. Automation bias creates the risk of unwarranted reliance on system outputs when the underlying realization is flawed (Parasuraman, R & Manzey, D, 2010), for instance, building on incomplete data, inappropriate methods, or compounding errors across operations.

How to detect and handle unanswerable insight needs? Not all insight needs are resolvable since corpora may lack sufficient data, inputs may be underspecified beyond what conventional or selective resolution can address, or the request may be logically ill-formed. Work on unanswerability is limited and largely confined to text-to-SQL settings (Li, J et al., 2025) (Ren, X et al., 2026) (Wang, B et al., 2023), where unanswerability is typically reduced to schema-level coverage gaps. The open setting introduces additional challenges of insufficient data quality, insufficient system capabilities to solve a task (e.g., a system that can only interact with data through SQL is limited to the operations supported by the specific SQL dialect), and data that supports a result but lacks statistical validity, for instance, when samples are too small or confounders undermine the intended interpretation. Future research should develop taxonomies of unanswerability in OpenTI and equip systems with the capacity to detect, categorize, and communicate such unanswerability accordingly.

How can the uncertainty of realizations and outputs be quantified? The reliability of produced realizations may vary substantially. A realization chains multiple operations, each introducing potential uncertainties and errors, for instance from imprecise retrieval and noisy data to inappropriate analytical choices. Errors may compound across the realization, such that individually reasonable steps may yield unreliable results (Zhao, Q et al., 2025). None of the surveyed systems provides systematic means to quantify this uncertainty, presenting outputs without confidence assessment or caveats about analytical limitations. Future work should propose methods that can quantify and propagate uncertainties, assess whether data and methodology support the conclusions, and surface these assessments to users (Kirchhof, M et al., 2025).

How to detect when realizations fail to serve the insight need? A realization can be methodologically sound yet fail to deliver the insight the user actually needs. Users may express their insight need $\hat{i}$ in a way that requests analytical knowledge that does not itself fully satisfy the underlying insight need $i$. For instance, a user interested in understanding the effect of one variable on another may ask for the correlation between them, yet actually require an analysis that establishes the direction and significance of the effect while accounting for confounders to satisfy their underlying insight need. This divergence may be driven by insufficient methodological knowledge, as users articulate high-level analytical goals without identifying the concrete analytical operations that would satisfy them (Grammel, L et al., 2010) (Amar, R et al., 2005), or by cognitive biases that influence how needs are framed, for instance, in users who seek analytical knowledge that confirms a prior belief (Lehner, P et al., 2008) (Cook, M & Smallman, H, 2008), where a system that faithfully answers may introduce or reinforce misinterpretations rather than contributing factual insights. Systems should thus recognize when an accurate response to $\hat{i}$ would not serve the underlying need, communicate this divergence, and propose better-suited analyses, requiring reasoning about the latent need beyond its surface expression.

6.2.2 Enabling Cooperative Human-System Interaction

Section 4 establishes insight extraction as a cooperative process in which users deliberately delegate unspecified aspects of their insight need to the system and both parties carry obligations of revealment. Research into analytical systems has traditionally placed the burden of alignment on the user, who is expected to form an accurate mental model of what the system can do and how it interprets inputs. Following the obligation for mutual revealment formulated by Radlinski and Craswell (2017) and the division of labor argued for by Gomm et al. (2025), we instead regard this alignment as a mutual responsibility in which the system maintains and refines a model of the user and the context their insight needs arise in, just as the user builds a model of the system’s data and capabilities. Progressing towards such mutual alignment requires work on interaction modes, interpretability mechanisms, and contextual knowledge.

What interaction modes best serve users? Complex analytical workloads may require iterative exchange, as argued in Section 4.1. Yet, system-initiated and collaborative interaction modes remain largely conceptual. Future research should investigate which interaction modes are most effective for different combinations of task complexity and user expertise, and develop systems that support richer interaction protocols. This requires extensive user studies and novel approaches to validate the accuracy of systems under these interaction protocols.

How to make realizations accessible and interpretable to users? Users verify system outputs by assessing both the operations performed and the data involved (Gu, K et al., 2024) (Xie, L et al., 2024). Making the realization $r$ explicit, interpretable, and actionable for intervention is essential to delegate control to users, fostering trust and enabling users to judge the reliability of realizations (Shneiderman, B, 2020). We argue for future work drawing on data provenance and lineage research (Buneman, P et al., 2001) (Cheney, J et al., 2009), surfacing data flows and operations at appropriate levels of abstraction. This requires balancing comprehensiveness with cognitive effort (Kazemitabaar, M et al., 2024) and developing representations that serve both verification and direct intervention.

How to adapt insight extraction to its context? Tabular data mostly exists in organizational settings where analytical tasks are deeply contextualized, with domain-specific vocabulary, conventions, implicit constraints, policies, and regulations (Cao, L & Zhang, C, 2007) (Egg, A et al., 2025) shaping how insight needs should be interpreted. Operating within this context reduces the degree to which insight needs must be explicitly specified, as shared understanding resolves ambiguities that would otherwise require specification. Instead of burdening users with accounting for the system’s missing contextual knowledge through redundant specification, which users embedded in a domain may not even recognize as necessary since much of this context is tacit (Polanyi, M, 1967), we argue that systems should acquire and maintain accurate models of organizational and domain context, facilitating efficient interactions by reducing the requirements on input specification. Beyond the organizational level, systems should also adapt to the specific user since different users bring distinct expertise, expectations, and analytical perspectives (Zhang, A et al., 2020), implying that similar insight needs may require different realizations depending on who expresses them. Similarly, users may regard their interaction histories as shared context, under which they express their insight needs (Yin, P et al., 2023). Such contextual knowledge could be informed by organizational documentation, domain ontologies, interaction histories, or user profiles.

How to determine appropriate output presentations? The output need $i_o$ is frequently left unspecified by users, yet the choice of output modality, like textual summaries, tables, charts, or a combination thereof, may substantially affect the utility of a realization. Surveyed systems either rely on explicit output specification in the input (Hu, X et al., 2024) (Huang, Y et al., 2024) or default to a fixed modality. Future work should explore inferring appropriate output presentations from the analytical context, the nature of the results, and user expectations, making the output need a first-class component of the realization process.

6.3 Rethinking How Systems Are Built and Evaluated

The preceding directions ask what systems should be able to do. We now shift the focus to how the paradigms that current systems inherit from the narrow task framings they are built and benchmarked for need to adapt to support this, spanning how analytical work is orchestrated, what models carry it out, and how systems are evaluated.

6.3.1 Dynamically Orchestrating Analytical Workloads

For simple insight needs, the analytical process can often be determined from the input alone. For instance, a lookup usually only requires locating the right table and extracting a value. However, as the complexity of insight needs increases, the realization that satisfies them becomes increasingly harder to specify upfront. Intermediate results may reveal that an initially chosen analytical method is unsuitable for the data, that required variables are distributed across tables not yet retrieved, or that the initial plan has a flaw. User inputs throughout processing also require mechanisms to adjust the analytical process. While reactive systems demonstrate initial capabilities in this direction (Zhang, Y et al., 2024) (Wang, Z et al., 2024), their adaptivity remains largely confined to selecting among predefined operations, with data bindings $\beta$ fixed upfront and limited capacity for backtracking and revising the analytical strategy. Advancing OpenTI towards complex workloads requires orchestration that can dynamically adapt all components of a realization $r=(O, E, \beta)$ in response to intermediate results and interactions.

How to acquire data during the analytical process? Most existing systems operating in an open setting separate retrieving data from the downstream usage of that data in a two-stage design (Kong, K et al., 2023) (Chen, W et al., 2021), requiring all relevant tables to be identified before any data operations take place. Yet complex insight needs may reveal data requirements only as the analysis unfolds, for instance, when intermediate results require linking tables or additional reference data not initially anticipated. While research into dynamic retrieval in retrieval augmented generation (Jiang, Z et al., 2023) (Asai, A et al., 2024) highlights the potential of retrieving data when it becomes relevant, initial steps by Wang and Li (2025) in applying this paradigm to OpenTI fall short of systematically representing and utilizing intermediate analytical context for retrieval, or of addressing data compatibility with intermediate results.

How can user input be integrated during processing? As discussed in Section 4.1, interaction modes beyond single-input remain underexplored. From an orchestration perspective, this requires systems to recognize the risk of producing misaligned realizations when proceeding with an interpretation that is uncertain due to ambiguity or data challenges, and to collect user input instead (Gomm, D et al., 2025) (Saparina, I & Lapata, M, 2025). This demands mechanisms to estimate confidence in interpretations and to incorporate intermediate user inputs into the ongoing workflow.

How should orchestration adapt to the task? The different types of insights (Section 2.3) have fundamentally different analytical requirements. A lookup may be best served by direct retrieval and a single SQL query, while a prediction may require integrating data from different tables, pre-processing it, building, validating, and applying a model. Current systems apply uniform orchestration strategies regardless of task characteristics. Future work should explore how to route insight needs to appropriate orchestration strategies based on the complexity and requirements of the analytical workload, potentially drawing on task classification or complexity estimation as an initial step.

6.3.2 Superseding and Complementing Language Models

Surveyed systems rely almost exclusively on language models for orchestration, reasoning, and code generation. Yet language models are architecturally mismatched with core aspects of insight extraction. Tables are two-dimensional structures that exhibit column-order invariance (Cong, T et al., 2023), contain data of various types, and stand in complex relationships to each other. Linearization into text does not capture these attributes, leading to limited performance in understanding tabular data, particularly as table size increases (Wolff, C & Hulsebos, M, 2025). Beyond tables, the unfolding analytical process, comprising partial results, transformed tables, and the evolving directed graph of operations, is inherently non-sequential, yet must be communicated to language models through linear context windows.

Which architectures fit tabular data and the analytical process? Initial efforts towards table-native models through modified training and architectures (Herzig, J et al., 2020) (Li, P et al., 2024) (Su, A et al., 2024) have shown limited improvements over general-purpose language models on downstream tasks. Large improvements have been made in recent in-context learning for regression and classification on tabular data, so-called Tabular Foundation Models (TFMs) (Hollmann, N et al., 2023) (Qu, J et al., 2025) (Hollmann, N et al., 2025). Yet, these advancements have only transferred to insight extraction insofar as TFMs have been proposed as tools for language model agents (Cong, T et al., 2025). Future research should thus explore architectures and methods that better fit the structure of the analytical process and the data it is applied to.

To what degree can single models cover the analytical process? The functional capabilities examined in Section 3.3 span locating, understanding, and transforming tabular data as well as orchestrating, reasoning within, and compiling outputs from the analytical process. It remains an open question to what degree these skills can be effectively covered by single models, and where specialization or composition of dedicated components yields more capable systems.

6.3.3 Designing Realistic and Faithful Evaluations

Evaluation methodology must faithfully assess the systems we aim to build. As shown in the analysis in Section 5, current benchmarks exhibit corpora that trade off scale against structural realism, input specifications suffer from ambiguities and data privileges, tasks remain concentrated on descriptive insights with limited coverage of inferential and causal types, and validation mechanisms predominantly rely on output-level exact matching. These limitations mean that benchmark performance may not reflect system capabilities under realistic conditions. Addressing these gaps requires not only more representative datasets but also methodological advances in how system behavior is assessed.

How to design robust realization-level verification? Benchmarking systems on their outputs alone risks conflating correct realizations with coincidentally correct answers, which has been shown to happen when systems bypass data evidence and instead guess a response (Liu, X et al., 2024). Moreover, for open-ended analytical tasks where multiple valid realizations exist (Gomm, D et al., 2025) (Saparina, I & Lapata, M, 2025), output-level validation against singular gold answers penalizes legitimate alternative interpretations. Realization-level verification, which explicitly examines operations $O$, their composition $E$, and data bindings $\beta$, offers more fine-grained diagnostic signal and enables partial credit for correct substeps (Chen, Z et al., 2025). However, this requires systems to expose realizations in standardized forms and demands validation methodology capable of assessing analytical soundness.

How to evaluate interactive systems and multimodal outputs? Current evaluations mostly assume a standalone input scored against a fixed reference output (Section 5), an assumption that breaks on both sides as OpenTI advances. On the input side, the cooperative interaction modes argued for in Section 6.2.2 replace the standalone utterance with a composition of inputs provided at different times, where the system’s state at each step shapes the subsequent user input. This interdependence between system and input complicates both the design of evaluation instances and the attribution of performance. Initial benchmarks emulate such interaction through LLM-based user simulation (Dutta, A et al., 2026) (Huo, N et al., 2025) (Li, J et al., 2025), yet user models remain prone to goal misalignment and unrealistic response patterns (Seshadri, P et al., 2026) (Mehri, S et al., 2026), calling for methods that faithfully emulate user behavior. On the output side, realizations increasingly produce results that combine text, tables, and visualizations, demanding validation mechanisms that assess these modalities jointly rather than in isolation (Dutta, A et al., 2026).