2 Open Tabular Insight Extraction
Since research towards OpenTI originates in different communities that address overlapping problems with divergent terminology and assumptions, this section develops a formal framework for Open Tabular Insight Extraction (OpenTI). We lay out this shared understanding by establishing the core concepts and formalizing OpenTI (Section 2.1). We then examine the conditions under which insight needs can be satisfied (Section 2.2). Finally, we cover the spectrum of insight needs that OpenTI addresses (Section 2.3).
Relationships and dependencies among core concepts of OpenTI. A user holds an insight need , consisting of a data need , methodological need , and output need , and expresses it as . A system maps to a realization , a graph of operations connected by dataflow edges with a binding that links operations to tables from the corpus . Executing yields analytical knowledge as output , which is the basis for an insight.
2.1 Conceptual Foundation
We develop the conceptual foundation of OpenTI from first principles. We first ground the general notion of an insight as the form of knowledge to be delivered (Section 2.1.1). We then operationalize this into insight needs, characterizing what users seek in tabular data and along which dimensions it varies (Section 2.1.2). Building on this, we formulate the problem of OpenTI (Section 2.1.3), show how the analytical knowledge that serves an insight need derives from data through realizations, executable compositions of operations bound to data (Section 2.1.4), and finally formulate OpenTI as an optimization problem (Section 2.1.5). Figure 2 provides an overview of the relationships between the concepts introduced throughout this section.
2.1.1 Insights
Insights have been defined and discussed in various ways across visualization research , cognitive science , and data analytics , with ongoing debate about their defining characteristics, including whether insights arise unexpectedly by definition or not . Adapting the recent synthesis of Battle and Ottley , which consolidates these perspectives, we understand an insight as a collection of knowledge that links analytical knowledge derived from data with a person’s pre-existing contextual understanding. Analytical knowledge is the direct output of computational operations on data, whereas contextual understanding denotes a person’s pre-existing knowledge, including domain expertise, organizational background, prior experiences, and previously acquired information . As such, insights exhibit the following characteristics:
- Grounded in data. An insight is grounded in analytical knowledge extracted from data through analytical operations, such as transformations, statistical modeling, or other computations .
- Contextual. An insight is not the analytical knowledge itself but its interpretation within a person’s contextual understanding . The role of contextual understanding ranges from minimal, when the analytical knowledge is largely self-explanatory (e.g., a retrieved fact), to substantial, where domain expertise fundamentally shapes interpretation (e.g., in understanding ROC curves or residual plots).
- Independent of presentation. Since insights emerge from interpretation, they are not bound to a specific form of presenting analytical knowledge . Instead, different forms of presenting analytical knowledge may invoke the same insights.
- Compositional. Insights contribute to a person’s contextual understanding, and subsequent insights can build on this enriched context . This hierarchical structure means that complex insights may build on the context of prior insights , that is, there exist insights that compose a set of sub-insights, which may themselves be composed of sub-insights.
2.1.2 Insight Needs
An insight need arises when a person1 requires analytical knowledge that is derived from data. We ground this concept in the established notion of information needs from information retrieval, where an information need represents the gap in a person’s knowledge that motivates interaction with an information system . An insight need specializes this to the analytical domain by targeting a knowledge gap concerning analytical conclusions residing in data, where satisfying this need requires computationally deriving knowledge rather than retrieving pre-existing information alone.
Definition 1 (Insight Need)
An insight need is a person’s requirement for analytical knowledge which, when interpreted within their contextual understanding, constitutes an insight.
Insight needs are analytical and insight-centered. They seek analytically derived knowledge (i.e., facts, patterns, relationships, predictions, or other analytical conclusions), rather than to produce an artifact. For instance, a need for a trained machine learning model is not an insight need, though artifacts may be instrumental in delivering insights, such as when a model is applied to make a desired prediction.
In addition to these characteristics, insight needs vary along the following dimensions:
- Directedness. Insight needs range from highly directed, targeting specific phenomena, entities, or relationships (e.g., “What is the correlation between drug dosage and patient recovery time?”), to loosely directed, where the analytical perspective remains open (e.g., “What patterns exist in hospital readmission data?”) . The degree of directedness determines whether it is possible to evaluate whether candidate analytical knowledge satisfies the insight need.
- Latency. The actual insight need exists in a person’s mind and may not be entirely specified in their expression of it . A person can express an approximation through utterances and contextual signals. This gap between latent need and expressed need mirrors the classical model of information need formation by Taylor , which traces how needs progress from an unexpressed visceral sense through conscious and formalized stages to a compromised expression adapted to system capabilities. For insight needs, the complexity of the need is compounded by the analytical requirements of selecting which operations should be performed on which data. Latency is rooted in the person’s contextual understanding, including tacit knowledge the person cannot fully articulate , assumptions the person considers obvious, and preferences the person has not consciously examined .
- Compositionality. Following the compositional nature of insights, insight needs can also be composed of atomic needs. They may decompose into sub-needs that jointly describe the insight need, which may themselves decompose recursively, allowing arbitrarily complex compositions of insight needs.
2.1.3 Open Tabular Insight Extraction
Insight needs may target any form of data and any analytical domain. Open Tabular Insight Extraction scopes this to satisfying insight needs by deriving analytical knowledge from a corpus of tables, where users may have an incomplete knowledge of its contents, schemas, and coverage.
Definition 2 (Open Tabular Insight Extraction (OpenTI))
Open Tabular Insight Extraction (OpenTI) is the task of satisfying a sufficiently directed insight need by deriving and presenting the analytical knowledge that most fully serves it, over a corpus of tabular data in an open setting where corpus contents are potentially diverse and not known when the need is expressed.
Recalling the directedness dimension introduced above, OpenTI addresses insight needs that are sufficiently directed so that it is determinable whether candidate analytical knowledge satisfies the need. Since the need itself is latent, no party can check this condition directly. We thus operationalize this sufficient specification on the observable expression of the insight need in Section 2.2.1.
Tabular Data.
The task operates on a collection of tabular data. A table consists of a header, an ordered set of column names that defines the table’s schema, a data matrix holding the core data values, where a row typically corresponds to an entity and a cell holds the value of the attribute named by its column, and metadata that may provide contextual information such as titles, captions, column descriptions, or surrounding text . A corpus comprises such tables and can correspond to any large-scale collection of tables like a data lake, warehouse, or repository. In practice, tabular data largely exists in relational databases , which add interconnections between tables through explicit schema relationships such as primary and foreign keys that define how entities relate across tables and how distributed information can be combined. In contrast, data lakes and collections of web tables aggregate tables from heterogeneous sources without such explicit relationships, exhibiting variation in schema quality, naming conventions, and completeness. The existence or absence of these relationships places different requirements on systems (Section 2.1.4).
Open Setting.
OpenTI operates under an open setting in which users express insight needs over a corpus whose contents, schemas, and coverage are unknown to them. This contrasts with closed settings , where users formulate requests with implicit knowledge of the available data. Users therefore cannot be expected to express insight needs in terms that align with the data’s vocabulary, structure, or scope, requiring systems to identify relevant data from . As under the open-world assumption , users do not know whether relevant data is captured in the corpus, meaning the system has to establish whether covers a given insight need. As in open-domain (tabular) question answering and retrieval , the relevant data must be retrieved from , though without being contingent on topical breadth, meaning that a large single-domain data lake is also considered open under OpenTI.
Decomposition of Insight Needs.
We decompose insight needs into data, methodological, and output factors following task typologies in visualization, which distinguish the dimensions of “what”, “how”, and “why” , and mirroring the implicit structure of database queries, which separately specify data selection, analytical operations, and output formatting:
- Data need (): The specification of what data entities, temporal scope, and domain boundaries are relevant for the analysis. This may be explicitly stated (e.g., “revenue of Fortune 500 companies in 2023”) or implicitly indicated through contextual cues and conventions (e.g., “GDP of Liechtenstein last year”, if the current year is known by the system).
- Methodological need (): The specification of what analytical operations and procedures should be applied to extract the desired analytical knowledge. This ranges from explicit methodological directives (e.g., “calculate the Pearson correlation”) to high-level analytical goals that delegate methodological choices to the system (e.g., “analyze the relationship between…”).
- Output need (): The specification of how results should be presented. This may remain completely unspecified by a user that expects the system to select an appropriate presentation.
2.1.4 Realizations
We formalize the relationship between data and the derived analytical knowledge as realizations:
Definition 3 (Realization)
A realization is a complete, executable specification of computational logic with a binding of specific tables to that logic, which, when executed, yields analytical knowledge as output .
We adopt the term from systems theory, where a realization denotes a concrete implementation of an input-output behavior . In OpenTI, a realization captures a derivation path from data in the corpus to analytical knowledge end-to-end. Whether the analytical knowledge a realization derives satisfies a given need is a property of the realization relative to that need.
We represent realizations as a Directed Acyclic Graph (DAG) of operations connected by edges denoting data flow, similar to representations of computations established in scientific workflow specifications and database query execution plans . We adapt this representation to insight extraction, where computational operations are anchored to data discovered from a corpus rather than pre-specified inputs. Unlike a database query, a realization is often a multistep workflow involving data discovery, integration, transformation, analysis, and presentation. Formally, a realization is a tuple describing a converging directed acyclic graph of operations:
- is the set of fully parameterized operations applied in the derivation.
- is a set of directed edges connecting the output of operations to the inputs of subsequent operations, defining the data flow.
- is the data binding that connects specific tables to the source nodes of the graph.
An operation represents a discrete functional unit, which can range from granular primitives (e.g., filter, join, …) to high-level analytical functions (e.g., fit a regression model, solve an optimization problem, …). Each operation in is fully parameterized, meaning all relevant choices, such as aggregation functions, model specification, or visualization parameters, are determined. The edges capture the compositional structure of the derivation, specifying how intermediate results flow between operations. The data binding anchors the computation to concrete tables, specifying which tables serve as inputs to which operations. The DAG converges into a single sink operation, which produces the final output resulting from the realization. Figure 3 exemplifies an abstracted realization that derives analytical knowledge to address an insight need.
Importantly, a realization is a declarative account of how analytical knowledge is derived from data. It does not capture the process by which a system arrived at this specification, which may include exploratory search, retrieval failures, user interactions, and iterative refinement. This parallels the established differentiation between prospective provenance (the specification of a computational task) and retrospective provenance (the record of what was actually executed) in scientific workflow research . Section 3 examines the process by which systems derive realizations.
Example of an abstracted realization . Nodes are fully parameterized operations , edges denote dataflow, and the binding anchors source operations to corpus tables. The graph converges to a single sink operation producing the output.
Multiple Realizations and Utility.
A single insight need may admit many realizations. When users express a methodological goal like “analyze the relationship between X and Y,” different operational compositions (e.g., Pearson correlation, Spearman correlation, regression analysis) may each constitute a valid way to address this intent. Similarly, different combinations of tables from could support distinct realizations. This one-to-many relationship reflects a division of labor in cooperative interaction between the user and the system , where users grant systems agency over the aspects of their needs they leave open, requiring the system to select among the resulting alternatives. These alternatives, however, are rarely equal in value. A realization drawing on more recent or more complete data, employing more robust analytical methods, or producing clearer output may substantially better serve the user’s goals. We model this qualitative dimension through a utility function .
Definition 4 (Realization Utility Function)
For a given insight need , the realization utility function maps possible realizations to a real-valued score, where a higher value indicates a realization that more effectively, accurately, or suitably satisfies the user’s insight need.
The utility function aggregates the quality with which a realization addresses data, methodological, and output needs, capturing not just the value of the final outputs but the full realization performed . For the data component, utility reflects the relevance of the data , reflecting factors such as recency, completeness, and reliability . For the methodological structure, utility may capture the appropriateness and rigor of analytical procedures given the analytical intent. For the output presentation, utility may reflect clarity, informativeness, as well as the effectiveness and appropriateness of visualizations . While users do not explicitly specify what contributes to their utility functions , these functions embody the implicit quality criteria that distinguish superior realizations from adequate ones. Utility only captures how well a realization satisfies an insight need, regardless of how it was derived. Computational effort, latency, and the interaction burden placed on the user are properties of the process by which a system constructs a realization, which influence the usefulness of a system but do not influence the utility of a realization.
2.1.5 Problem Formulation
A realization is valid for an insight need if the tables bound through provide the information the data need mandates, its operations and their composition correctly address the methodological need , and its output delivers the kind of result the output need calls for. Validity is a property of the realization relative to the insight need, judged on the content of the bound tables and the analytical logic applied to them, independent of whether the bound tables actually exist in the corpus. We denote the set of valid realizations for an insight need as . Validity and utility are complementary. Validity is a binary criterion that separates realizations addressing the insight need from those that do not, whereas the utility function grades how well realizations serve it. A realization can thus be valid yet exhibit low utility, for instance when its output delivers the required analytical knowledge but presents it poorly.
A realization is feasible given a corpus if the tables it requires are present in the corpus, that is, all tables bound via exist in , . Feasibility is a property of the data binding relative to the corpus, independent of whether the realization addresses any particular insight need or whether a given system possesses the capabilities to combine the bound tables. We denote the set of feasible realizations as .
The set of candidate realizations captures realizations that both address the insight need and are grounded in available data. The realization that optimally satisfies the insight need is then the candidate realization that maximizes utility:
where represents a distinct failure state indicating that the insight need cannot be resolved given the corpus. This objective is defined relative to the insight need and thus characterizes an idealized target. A system never has direct access to , and consequently not to or , observing only the expressed need . To construct an approximation of , a system must construct its own representation of the need, denoted as , from the expression and available contextual information, bringing into closer correspondence with (see Section 4). The realization space is combinatorially vast, as the number of possible operation compositions, parameterizations, and data bindings grows with the scale of the corpus and the expressiveness of the available operations. Neither an analytical solution nor enumeration is practical. Section 3 thus explores how systems can navigate this space effectively to construct utility-maximizing realizations.
2.2 Sufficiency Criteria
Not all insight needs can be satisfied in OpenTI. As implied in the formalization, a system can only produce an output that satisfies the insight need if the candidate set is non-empty. This requires that the expression of the insight need is sufficiently specified for a system to delineate (Section 2.2.1), and that the corpus contains sufficient data so that at least one valid realization is feasible, i.e., is non-empty (Section 2.2.2).
2.2.1 Sufficient Specification
Sufficient specification describes the expressed need relative to the underlying need . needs to allow delineating , separating realizations that satisfy the insight need from those that do not. This requires that the data, methodological, and output components , , are specified or can be inferred accordingly. For instance, “What is the average temperature in Amsterdam in January?” is sufficiently specified since the implicit methodological details can be resolved through conventions and reasonable defaults. In contrast, “How does construction waste compare?” is insufficient since no convention fixes what to compare against, by what metric, or in what region, leaving valid and invalid realizations indistinguishable. We discuss how systems close such gaps autonomously and in collaboration with the user in Section 4.
2.2.2 Data Sufficiency
Even a fully specified insight need is impossible to satisfy if the corpus lacks the required data, which is particularly relevant when users are unaware of the contents and structure of . Satisfiability requires that the corpus supports at least one valid realization. We formalize this as sufficient sets of tables:
Definition 5 (Sufficient)
A set of tables is sufficient for an insight need if there exists at least one valid realization where the set of tables bound via satisfies .
When no sufficient set exists within the corpus, that is sufficient for , the candidate set is empty and the insight need cannot be resolved, an instance of insufficient data coverage for in . Within a sufficient set, tables align both with the insight need and with each other. Following Kothyari et al. and Chen et al. , we distinguish two forms of alignment:
- Semantic alignment (need-table): The tables in collectively cover all semantic concepts specified in the data need .
- Structural alignment (table-table): Tables holding complementary information are combinable through the operations available to the system (e.g., joinable via shared keys, unionable,…).
The two forms are independent. A set may cover all relevant semantic concepts yet be structurally incompatible, and tables may be perfectly joinable yet lack the required semantic coverage, both rendering the set insufficient. The insight need defines how semantic components must relate, while a system’s operational capabilities determine how information can be combined across tables.
Example of semantic and structural alignment. A candidate set is semantically aligned when its tables jointly cover the required concepts and structurally aligned when they are combinable, here via a join on CountryCode. Both must hold for sufficiency.
Figure 4 exemplifies these conditions. Where the required information does not reside in a single table, it must be integrated across multiple tables. As shown, a sufficient set must cover population counts, GDP values, country identifiers, and the year 2021 (semantic alignment), and these must be integrable through the means a system possesses, here through shared keys like CountryCode (structural alignment). The conditions for structural alignment depend on the capabilities of the system processing the data. For a system capable of aligning French and English contents, for example, the middle column would be structurally aligned. How systems surface such sets and navigate the search over candidate sets that both conditions induce is examined under the data retrieval capability in Section 3.3.2.
2.3 Taxonomy of Insight Types
Deriving the analytical knowledge that satisfies an insight need requires substantially different assumptions and capabilities depending on which kind of knowledge is sought. We organize this space by insight type, the kind of analytical knowledge a realization derives, rather than by insight need. An insight need is latent and only partially captured by its expression (Section 2.1.2), whereas the insight type is observable on the realization and most directly determines the capabilities a system requires. The operations and their composition differ categorically across types, such as between reading a stored value and estimating a causal effect, while data and output complexity vary within a type without imposing qualitatively different requirements. Figure 5 organizes insight types into three tiers ordered by what must be assumed, beyond locating the relevant data, for the derived knowledge to be justified, following the established distinction between descriptive, inferential, and causal analysis applied to OpenTI.
Taxonomy of insight types in OpenTI, organized by the assumptions required for the derived analytical knowledge. Each tier encompasses representative insight types with examples.
Descriptive insight types concern the observed data itself and require no assumptions about anything beyond it. They form the most varied and, in current research, the most common tier (see Figure 1b), ranging from exact computed facts to summary characterizations of the data. Retrieving a cell value or selecting entities that meet a criterion (Lookup), summarizing values over records such as counts, sums, extrema, and rankings (Aggregation), describing the distribution of a variable (Characterization), measuring how variables co-vary (Association), and surfacing latent groups among records (Grouping) all describe what is present in the data and commit to nothing further. Many such needs, like computing a key performance indicator or ranking entities by an aggregate, have a single correct answer fixed by the data, yet deriving them can demand considerable retrieval, integration, and computation across a corpus. Others, like characterizing a distribution or quantifying an association, produce summary statistics, but these still describe the actual records rather than generalizing beyond them. Inferential insight types concern the data-generating process behind the observed data, making claims that are meant to hold for instances the corpus does not contain and therefore relying on assumptions that link the data to that process. Determining whether an observed pattern reflects a stable feature of the process rather than noise (Statistical Inference) and estimating values for instances the corpus does not contain, including future states (Prediction), both reach beyond the data at hand, differing in whether the goal is to characterize the process or to forecast values . Causal insight types make the strongest claims, concerning not what the process looks like but how it would respond to intervention. Estimating the effect of an action or policy at the population level (Interventional) and determining the outcome of a specific case under a condition contrary to fact (Counterfactual) cannot be settled from observed patterns alone and require assumptions about the causes behind the data, such as a structural causal model .
These tiers stand in an asymmetric relationship to each other. Knowledge from a higher tier entails regularities on a lower tier, but not vice versa. A model of the process entails the patterns in any data it generates, and a causal account of the process entails both those patterns and the effects of intervening on it. On the other hand, no observed association, however strong, implies how the process would respond to intervention, and no body of observations uniquely determines the process that produced them. For example, a structural causal model fixes the distribution of the data it generates and the effect of intervening on any variable, whereas that same distribution is consistent with many distinct causal models that disagree about those effects. The familiar maxim that correlation does not imply causation is an example of this relationship.
The tiers in the taxonomy shown in Figure 5 span the full spectrum of possible insight types based on analytical knowledge extracted from tabular data. An analytical claim either concerns the observed data or generalizes beyond it to the process that produced it. A claim about that process either concerns the process as observed or under intervention . Any analytical outcome thus falls into exactly one tier, and no further level of assumptions exists between them. The insight types listed within each tier are representative but not necessarily exhaustive, capturing the categories most relevant in OpenTI. We ground the descriptive types in established accounts of low-level analytical primitives , the inferential split in the distinction between explanatory and predictive modeling , and the causal tier in the causal hierarchy .
The insight types in the taxonomy are atomic. Since insights are inherently compositional (Section 2.1.1), an insight may compose an arbitrary number of atomic insights of the same or different types, and many common analytical tasks represent such compositions. For instance, comparing two means composes two aggregations when it asks which is larger, and becomes a statistical inference when it asks whether the difference reflects the underlying process rather than noise. Similarly, analyzing a trend decomposes into a descriptive slope over observed values or an inferential forecast. At the far end of this spectrum, insight needs may demand elaborate compositions that layer substantial methodology onto atomic types, as in scientific workflows that apply optimization, computational modeling, or simulation, all of which inherit the tier of the knowledge they operate over rather than forming a tier of their own. The latent nature of insight needs means that the type implied by an expression need not match the type that satisfies the underlying need. The same comparison, expressed as “Does X or Y have the larger mean Z?”, resolves on its surface to a composition of aggregations, yet the user may actually seek to know whether the two means differ meaningfully, which is a question of statistical inference. We develop this methodological latency in Sections 4 and 6.
Share of chronologically feasible citations between works by their task label.
Citation rate among reviewed systems by the number of capabilities two systems have in common.
Citation rates between reviewed works against how they label the task they target. Appendix 8.5 details the analysis.
This taxonomy situates OpenTI within a landscape of related but independently studied tasks. A substantial body of work covers deriving knowledge from tables, yet this work originates from different research communities that frame the challenge in diverging terminology and without an overarching direction. Tabular question answering , table-based reasoning , text-to-SQL , data science and analysis agents , and various further characterizations all address the problem space of OpenTI, facing similar challenges and often arriving at substantially similar system designs. Looking at how works across these task terminologies cite each other shows this fragmentation. Figure 6a shows that across the reviewed works, works cite of works that are chronologically possible within the same label against just of the works of a different label.
These labels primarily reflect conventions, such as the assumed input format or data setting, rather than the covered analytical scope. A tabular question answering system may address only Lookup, or it may extend across the descriptive tier and into inferential or causal types, which demand categorically different capabilities, and the label alone does not reveal which. Conversely, systems operating under different labels may address the same insight types with similar capabilities. As shown in Figure 6b, the more capabilities two systems share, the more likely they are to cite one another when they share the same label, but they are barely more likely to do so if they do not. The taxonomy of insight types makes the analytical scope explicit. Together with the preceding formalization, which centers the insight need as the object a system seeks to satisfy, it provides a unified vocabulary for describing what a given work actually concerns. The subsequent sections employ this framework to examine how systems construct realizations (Section 3), the possibilities of user-system interaction (Section 4), how evaluation methodology can assess system capabilities (Section 5), and what research is missing on the path toward capable OpenTI systems (Section 6).
Footnotes
-
We refer to a person when characterizing insight needs conceptually and to a user when referring to the entity interacting with a system. We note that this entity need not be human and may instead be an automated agent acting on an insight need. ↩