Skip to main navigation Skip to search Skip to main content

Vanilla Feature Distillation for Improving the Accuracy-Robustness Trade-Off in Adversarial Training

  • Guodong Cao
  • , Zhibo Wang*
  • , Xiaowei Dong
  • , Zhifei Zhang
  • , Hengchang Guo
  • , Zhan Qin
  • , Kui Ren
  • *Corresponding author for this work
  • Adobe Systems Inc.
  • Zhejiang University

Research output: Contribution to journal › Article › peer-review

6 Scopus citations

Abstract

Adversarial training has been widely explored for mitigating attacks against deep models. However, a critical limitation of existing works is that robustness enhancement is at the cost of noticeable accuracy degradation. To achieve a better trade-off between robustness and accuracy, we propose the Vanilla Feature Distillation Adversarial Training (VFDAT), which conducts knowledge distillation from a pre-trained model (optimized towards high accuracy) to guide adversarial training model towards generating high-quality and well-separable features by constraining the obtained features of natural and adversarial examples. More specifically, both adversarial examples and their natural counterparts are forced to be aligned in feature space by distilling predictive representations from a pre-trained natural model. In this way, the adversarial training model can be updated towards maximally preserving the accuracy as gaining robustness. A key advantage of our method is that it can be universally adapted to and boost existing works. Exhaustive experiments on various datasets, classification models, and adversarial training algorithms demonstrate the effectiveness of our proposed method.
Original languageEnglish
Pages (from-to)664-676
Number of pages13
JournalIEEE Transactions on Dependable and Secure Computing
Volume22
Issue number1
Early online dateJan 2024
DOIs
StatePublished - Jan 2025

Funding

This work was supported in part by National Key R&D Program of China under Grant 2021ZD0112803, in part by National Natural Science Foundation of China under Grant 62122066 and Grant U20A20182 and in part by Key R&D Program of Zhejiang under Grant 2024C01164 and Grant 2022C01018.

Keywords

  • Adversarial training
  • well-separable features
  • feature alignment
  • knowledge distillation

Indexed by

  • EI
  • Scopus
  • SCIE

Fingerprint

Dive into the research topics of 'Vanilla Feature Distillation for Improving the Accuracy-Robustness Trade-Off in Adversarial Training'. Together they form a unique fingerprint.

Cite this