An image and legend text alignment model based on position guided enhanced attention mechanism.

The problem of fine-grained matching between images and legend texts is crucial for achieving a refined understanding of document image structure. This task aims to accurately identify the relationships between image regions and their corresponding legend texts in document images. Although progress has been made in document structure understanding in recent years, current mainstream methods primarily focus on overall layout analysis and still lack effective models for fine-grained semantic matching between image regions and legend texts. To address this challenge, this paper proposes a fusion model based on a position-guided enhanced attention mechanism. In this model, features from images and legend texts are processed separately through heterogeneous structures, effectively capturing the distinctive characteristics of both modalities. To further improve the model's ability to perceive local structure, we introduce region coordinates as auxiliary features, providing additional detailed information. Building upon this, a cross-attention mechanism is incorporated to deeply fuse the features of the image and text modalities. Through deep interactive learning, the method achieves accurate fine-grained semantic alignment and matching judgments between images and text. In the Ancient Book Digitization Dataset and the DocBank Dataset, our model achieved matching accuracy rates of 90.34% and 68.5%, respectively, outperforming existing mainstream methods. This demonstrates the superior performance of our approach in fine-grained structural understanding tasks.