Integrating diverse datasets within data lakes presents a significant challenge, especially when semantically related tables lack explicit attributes for direct joining. To address this, a research team has developed LOKI (Latent-space Optimization for Knowledge Integration), a horizontal bidirectional cross-attention architecture. This system aims to discover and integrate relationships between disjoint tables by leveraging unstructured text as a mediating factor.
The core problem, termed "Discovery-Driven Integration," involves identifying relevant data sources and their implicit relational structures before integration. LOKI formalizes this as "Text-Mediated Join Path Discovery," focusing on connecting individual rows from different tables through specific sentences in text documents.
LOKI operates by mapping linearized table rows and unstructured text into a shared latent space. It employs a dual-path bidirectional cross-attention mechanism: a forward path where row queries attend to sentence representations, and a reverse path where sentence queries attend to row representations. Both paths incorporate an inner sparse-attention gate and an outer query-dependent vector gate before residual summation. This process allows LOKI to learn contextualized representations of table rows and sentences.
A global table-text contrastive objective guides LOKI, enabling fine-grained row-sentence associations to emerge without explicit local supervision. Unlike existing multi-modal discovery methods that often retrieve coarse-grained column-text associations or assume pre-supplied row-text links, LOKI transforms these implicit associations into explicit, interpretable join paths. It then organizes these paths into relation-consistent groups and materializes them as typed integrated tables, complete with sentence-level provenance.
The researchers evaluated LOKI on real-world benchmarks, demonstrating its ability to outperform existing multi-modal data discovery approaches. It materialized typed integrated tables with a 0.982 macro typed-pair precision. The system also showed efficiency benefits, being up to 40 times less expensive in LLM API costs compared to direct prompting methods.
LOKI's approach is particularly relevant for complex domains like healthcare and enterprise data lakes, where relational databases often lack explicit foreign keys and shared primary schemas. Traditional schema matching and entity resolution techniques frequently fail in these scenarios because tables may not share common attribute names or overlapping key distributions. LOKI addresses this by using external text as a semantic bridge to discover row-level relationships.
The architecture's ability to discover and materialize fine-grained relation tables corresponding to distinct cross-table relationship types was tested on 382 MIMIC-IV admissions. It achieved a pair precision of 97.95 (with GPT-OSS 20B) and 98.00 (with Qwen-3.6), and a physical table micro precision of 84.0 and 84.8, respectively. The cluster structural purity was measured at 99.5% or higher, with an Adjusted Rand Index (ARI) between 0.806 and 0.858. The inference efficiency showed a 67.4% to 68.8% reduction in token consumption, resulting in a total cost of $0.70 for GPT-OSS 20B versus $30.60 for Qwen-3.7-Max.