Skip to main navigation Skip to search Skip to main content

Cross-modal 2D-3D feature matching: simultaneous local feature description and detection across images and point clouds

  • Wei Ma*
  • , Yucheng Huang
  • , Shengjun Tang
  • , Xianwei Zheng
  • , Zhen Dong
  • , Liang Ge
  • , Jianping Pan
  • , Qingquan Li
  • , Bing Wang
  • *Corresponding author for this work
  • Chongqing Jiaotong Univ
  • Minist Nat Resources
  • Shenzhen Univ
  • Tianjin Inst Surveying & Mapping Co Ltd
  • Hong Kong Polytech Univ

Research output: Contribution to journal › Article › peer-review

7 Scopus citations

Abstract

Establishing correspondences between 2D images and 3D models is essential for precise 3D modeling and accurate positioning. However, widely adopted techniques for aligning 2D images with 3D features heavily depend on dense 3D reconstructions, which not only incur significant computational demands but also tend to exhibit reduced accuracy in texture-poor environments. In this study, we propose a novel method that combines local feature description and detection to enable direct and automatic alignment of 2D images with 3D models. Our approach utilizes a twin convolutional network architecture to process images and 3D data, generating respective feature maps. To address the non-uniform distribution of pixel and spatial point densities, we introduce an ultrawide perception mechanism to expand the receptive field of image convolution kernels. Next, we apply a nonlocal maximum suppression criterion to concurrently evaluate the salience of pixels and 3D points. Additionally, we design an adaptive weight optimization loss function that dynamically guides learning objectives toward sample similarity. We rigorously validate our approach on multiple datasets, and our findings demonstrate successful co-extraction of cross-modal feature points. Through comprehensive 2D-3D feature matching experiments, we benchmark our method against several state-of-the-art techniques from recent literature. The results show that our method outperforms nearly all evaluated metrics, underscoring its effectiveness.
Original languageEnglish
Pages (from-to)155-169
Number of pages15
JournalISPRS Journal of Photogrammetry and Remote Sensing
Volume229
Early online dateAug 2025
DOIs
StatePublished - Nov 2025

Funding

This work was jointly supported by the Young Scientists Fund of the National Natural Science Foundation of China (42301520) , the Early Career Scheme of Hong Kong Research Grants Council (25206524) , the Science and Technology Research Project of Chongqing Education Commission (KJQN202200744) , the Open Research Fund of Shenzhen Key Laboratory of Spatial Smart Sensing and Services (GEMLab-2024001) .

Keywords

  • 2D-3D Local Feature Matching
  • Cross-Modal Feature Description and Detection
  • Twin Convolutional Networks
  • Non-Local Maximum Suppression

Indexed by

  • SCIE
  • EI
  • Scopus

Fingerprint

Dive into the research topics of 'Cross-modal 2D-3D feature matching: simultaneous local feature description and detection across images and point clouds'. Together they form a unique fingerprint.

Cite this