Momentum Contrast Self-Supervised Based Training for Adversarial Robustness

Document Type : Research Article

Authors

Department of Computer Engineering, Shahid Bahonar University of Kerman, Kerman, Iran.

Abstract

By the rapid progress of deep learning and its use in a variety of applications, however, deep networks have shown that they are vulnerable to adversarial examples. Recently developed researches show that using self-supervised learning (SSL) in various ways results in increasing
network robustness. This paper examines the effect of a particular type of Contrastive SelfSupervised learning (CSSL) called Momentum Contrast (MoCo) on increasing network robustness to adversarial examples. For this purpose, MoCo is employed as a pre-text task and a deep network is pre-trained for this task. Then fine-tuning will cause to increase the robustness of the network against adversarial attacks examples. A new attack method is introduced based on MoCo and one of the Projected Gradient Descent (PGD) or Fast Gradient Sign (FGSM) methods that do not require any labeled data. Using this corrupted data and adversarial training method, a deep network is pre-trained and the representation provided by it is used to fine-tune downstream tasks that results in increasing network robustness. For an instance, the setup including Resnet50 structure, PGD attack, and MoCo-v1 shows 2.79%, 2%, and 1.35% of improvements comparing to the Jigsaw, Rotation, Selfie, respectively. More details of experiments and the improvements raised by MoCo are given in the results part and show the superiority of MoCo based models on CIFAR-10 and CIFAR-10-C datasets. Also, the obtained results for validating the robustness of proposed models against various noises with different corruption strengths, confirm the resistance of the proposed methods.

Keywords

Main Subjects

References

[1] T. Chen, S. Liu, S. Chang, Y. Cheng, L. Amini, and Z. Wang. Adversarial robustness: From self-supervised pre-training to fine-tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 699--708, 2020. [ bib | DOI ]
[2] Y. Bengio, P.Lamblin, D. Popovici, and H.Larochelle. Greedy Layer-Wise Training of Deep Networks. In Advances in Neural Information Processing Systems, pages 153--160, 2007. [ bib | DOI ]
[3] G. Hinton, S. Osindero, and Y. Teh. A Fast Learning Algorithm for Deep Belief Nets. Neural computation, 18(7):1527--1554, 2006. [ bib | DOI ]
[4] R. Raina, A. Battle, H. Lee, B. Packer, and A. Ng. Self-taught learning: transfer learning from unlabeled data. In Proceedings of the 24th international conference on Machine learning, pages 759--766. ACM, 2007. [ bib | DOI ]
[5] P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P. Manzagol. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of machine learning research, 11:3371–3408, 2010. [ bib | DOI ]
[6] D. Hendrycksa, M. Mazeika, S. Kadavatha, and D. Song. Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, page 15663–15674. ACM, 2019. [ bib | DOI ]
[7] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum Contrast for Unsupervised Visual Representation Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729--9738. IEEE, 2020. [ bib | DOI ]
[8] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A Simple Framework for Contrastive Learning of Visual Representations. In International conference on machine learning, pages 1597--1607. PMLR, 2020. [ bib | DOI ]
[9] X. Chen, H. Fan, R. B. Girshick, and K. He. Improved Baselines with Momentum Contrastive Learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). ArXiv, 2020. [ bib | DOI ]
[10] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards Deep Learning Models Resistant to Adversarial Attacks. In International Conference on Learning Representations (ICLR). ArXiv, 2018. [ bib | DOI ]
[11] M. Kim, J. Tack, and S. Hwang. Adversarial Self-Supervised Contrastive Learning. In Neural Information Processing Systems (NIPS), 2020. [ bib | DOI ]
[12] Y. Carmon, A.Raghunathan, L.Schmidt, P.Liang, and J. Duchi. Unlabeled Data Improves Adversarial Robustness. In Neural Information Processing Systems (NIPS), 2019. [ bib | DOI ]
[13] Y. Carmon, A.Raghunathan, L.Schmidt, P.Liang, and J. Duchi. Using Pre-Training Can Improve Model Robustness and Uncertainty. In 36th International Conference on Machine Learning, pages 2712--2721, 2019. [ bib | DOI ]
[14] A. Criminisi, P. Perez, and K. Toyama. Region filling and object removal by exemplar-based image inpainting. IEEE Transactions on image processing, 13(9):1200 -- 1212, 2004. [ bib | DOI ]
[15] R. Zhang, P. Isola, and A A. Efros. Colorful image colorization. In Colorful image colorizationIn European conference on computer vision (ECCV), pages 649--666. Springer, 2016. [ bib | DOI ]
[16] S. Gidaris, P. Singh, and N. Komodakis. Unsupervised Representation Learning by Predicting Image Rotations. In International Conference on Learning Representations (ICLR), 2018. [ bib | DOI ]
[17] A. Dosovitskiy, P. Fischer, J. Springenberg, M. Riedmiller, and T. Brox. Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks. IEEE transactions on pattern analysis and machine intelligence, 38(9):1734 -- 1747, 2016. [ bib | DOI ]
[18] F. Carlucci, A. D'Innocente, S. Bucci, B. Caputo, and T. Tommasi. Domain Generalization by Solving Jigsaw Puzzles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2229--2238. IEEE, 2019. [ bib | DOI ]
[19] M. Noroozi and P. Favaro. Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles. In European conference on computer vision, pages 69--84. Springer, 2016. [ bib | DOI ]
[20] T.Trinh, M.Luong, and Q. Le. Selfie: Self-supervised Pretraining for Image Embedding. In European conference on computer vision. Arxiv, 2019. [ bib | DOI ]
[21] H. Fang, S.Wang, M. Zhou, J.Ding, and P.Xie. CERT: Contrastive Self-supervised Learning for Language Understanding. In International Conference on Learning Representations (ICLR), 2020. [ bib | DOI ]
[22] Z. Wu, Y.Xiong, S.Yu, and Da. Lin. Unsupervised Feature Learning via Non-parametric Instance Discrimination. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3733--3742. IEEE, 2018. [ bib | DOI ]
[23] X. Yuan, P. He, Q. Zhu, and X. Li. Adversarial Examples: Attacks and Defenses for Deep Learning. IEEE transactions on neural networks and learning systems, 30(9):2805 -- 2824, 2019. [ bib | DOI ]
[24] N. Papernot, P. Mcdaniel, and I. Goodfellow. Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples. In International Conference on Learning Representations (ICLR). ArXiv, 2016. [ bib | DOI ]
[25] I. Goodfellow, J. Shlens, and C. Szegedy. Explaining and Harnessing Adversarial Examples. In International Conference on Learning Representations (ICLR). ArXiv, 2014. [ bib | DOI ]
[26] S. Qiu, Q. Liu, S. Zhou, and C. Wu. Review of Artificial Intelligence Adversarial Attack and Defense Technologies. Applied Sciences, 9(5), 2019. [ bib | DOI ]
[27] H. Xu, Y. Ma, H. Liu, D. Deb, H. Liu, J. Tang, and A. Jain. Adversarial Attacks and Defenses in Images, Graphs and Text: A Review. International Journal of Automation and Computing, 17(2):151–178, 2020. [ bib | DOI ]
[28] H. Zhang, Y. Yu, J. Jiao, E. Xing, L. Ghaoui, and M. Jordan. Theoretically Principled Trade-off between Robustness and Accuracy. In International Conference on Machine Learning, pages 7472--7482. PMLR, 2019. [ bib | DOI ]
[29] D.Hendrycks and T. Dietterich. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. In Proceedings of the International Conference on Learning Representations (ICLR). ArXiv, 2019. [ bib | DOI ]
Journal of Computing and Security
Volume 8, Issue 1 - Serial Number 1
January 2021
Pages 33-43

Files

History

  • Receive Date: 05 January 2021
  • Revise Date: 02 June 2021
  • Accept Date: 29 June 2021

Share

How to cite

Statistics