Skip to main navigation Skip to search Skip to main content

MaeFuse: Transferring Omni Features With Pretrained Masked Autoencoders for Infrared and Visible Image Fusion via Guided Training

  • Jiayang Li
  • , Junjun Jiang*
  • , Pengwei Liang
  • , Jiayi Ma
  • , Liqiang Nie
  • *Corresponding author for this work
  • Harbin Inst Technol
  • Harbin Inst Technol Shenzhen

Research output: Contribution to journal › Article › peer-review

82 Scopus citations

Abstract

In this paper, we introduce MaeFuse, a novel autoencoder model designed for Infrared and Visible Image Fusion (IVIF). The existing approaches for image fusion often rely on training combined with downstream tasks to obtain high-level visual information, which is effective in emphasizing target objects and delivering impressive results in visual quality and task-specific applications. Instead of being driven by downstream tasks, our model called MaeFuse utilizes a pretrained encoder from Masked Autoencoders (MAE), which facilities the omni features extraction for low-level reconstruction and high-level vision tasks, to obtain perception friendly features with a low cost. In order to eliminate the domain gap of different modal features and the block effect caused by the MAE encoder, we further develop a guided training strategy. This strategy is meticulously crafted to ensure that the fusion layer seamlessly adjusts to the feature space of the encoder, gradually enhancing the fusion performance. The proposed method can facilitate the comprehensive integration of feature vectors from both infrared and visible modalities, thus preserving the rich details inherent in each modal. MaeFuse not only introduces a novel perspective in the realm of fusion techniques but also stands out with impressive performance across various public datasets. The code is available at https://github.com/Henry-Lee-real/MaeFuse.
Original languageEnglish
Pages (from-to)1340-1353
Number of pages14
JournalIEEE Transactions on Image Processing
Volume34
DOIs
StatePublished - Jan 2025

Funding

This work was supported by the National Natural Science Foundation of China under Grant U23B2009 and Grant 62471158.

Keywords

  • Image fusion
  • vision transformer
  • masked autoencoder
  • guided training

Indexed by

  • SCIE
  • EI
  • Scopus

Fingerprint

Dive into the research topics of 'MaeFuse: Transferring Omni Features With Pretrained Masked Autoencoders for Infrared and Visible Image Fusion via Guided Training'. Together they form a unique fingerprint.

Cite this