Abstract
Adversarial training has been widely explored for mitigating attacks against deep models. However, a critical limitation of existing works is that robustness enhancement is at the cost of noticeable accuracy degradation. To achieve a better trade-off between robustness and accuracy, we propose the Vanilla Feature Distillation Adversarial Training (VFDAT), which conducts knowledge distillation from a pre-trained model (optimized towards high accuracy) to guide adversarial training model towards generating high-quality and well-separable features by constraining the obtained features of natural and adversarial examples. More specifically, both adversarial examples and their natural counterparts are forced to be aligned in feature space by distilling predictive representations from a pre-trained natural model. In this way, the adversarial training model can be updated towards maximally preserving the accuracy as gaining robustness. A key advantage of our method is that it can be universally adapted to and boost existing works. Exhaustive experiments on various datasets, classification models, and adversarial training algorithms demonstrate the effectiveness of our proposed method.
| Original language | English |
|---|---|
| Pages (from-to) | 664-676 |
| Number of pages | 13 |
| Journal | IEEE Transactions on Dependable and Secure Computing |
| Volume | 22 |
| Issue number | 1 |
| Early online date | Jan 2024 |
| DOIs | |
| State | Published - Jan 2025 |
Funding
This work was supported in part by National Key R&D Program of China under Grant 2021ZD0112803, in part by National Natural Science Foundation of China under Grant 62122066 and Grant U20A20182 and in part by Key R&D Program of Zhejiang under Grant 2024C01164 and Grant 2022C01018.
Keywords
- Adversarial training
- well-separable features
- feature alignment
- knowledge distillation
Indexed by
- EI
- Scopus
- SCIE
Fingerprint
Dive into the research topics of 'Vanilla Feature Distillation for Improving the Accuracy-Robustness Trade-Off in Adversarial Training'. Together they form a unique fingerprint.Cite this
- APA
- Author
- BIBTEX
- Harvard
- Standard
- RIS
- Vancouver