Visual transformer with depthwise separable convolution projections for video-based human action recognition

¹ Department of Computer Science, Brunel University of London, Kingston Lane, Uxbridge, United Kingdom
² Henan Key Laboratory on Public Opinion Intelligent Analysis, Zhongyuan University of Technology, Zhengzhou, China

^* e-mail: This email address is being protected from spambots. You need JavaScript enabled to view it.
^** e-mail: This email address is being protected from spambots. You need JavaScript enabled to view it.
^** e-mail: This email address is being protected from spambots. You need JavaScript enabled to view it.

Abstract

Human action recognition is a task that utilizes algorithms to recognize human actions from videos. Transformer-based algorithms have attracted growing attention in recent years. However, transformer networks often suffer from slow convergence and require large amounts of training data, due to their inability to prioritize information from neighboring pixels. To address these issues, we propose a novel network architecture that combines a depthwise separable convolution layer with transformer modules. The proposed network has been evaluated on the medium-sized benchmark dataset UCF101 and the results have demonstrated that the proposed model converges quickly during training and achieves competitive performance compared with SOTA pure transformer network, while reducing approximately 7.4 million parameters.