Detecting reconciliation discrepancies in tabular data using transformers.

Introduction: Large organisations maintain heterogeneous datasets in data lakes, where schema variability, inconsistent formats, and semantic ambiguity complicate reconciliation. Achieving a unified view of entities requires methods that capture both structural equality and semantic relationships. Methods: We propose a transformer-based methodology that adapts pre-trained language models (PLMs) to tabular data by generating metadata-enriched embeddings (table title, column name, type, and statistics). These embeddings are compared using mutual top- Results: Experiments on large-scale tabular data (185,909 tables) demonstrate that semantic embeddings efficiently uncover relationships and discrepancies, achieving precision of 0.958 at Discussion: Unlike prior schema-aligned or query-driven approaches such as DeepJoin, WarpGate, and Lotus, this study introduces a unified reconciliation pipeline that integrates equality-based and semantic matching for large-scale tabular data. The key contribution is a scalable and generalisable reconciliation methodology that operationalises transformer architectures for automated discrepancy detection in heterogeneous environments, establishing both methodological novelty and practical effectiveness.