Research on subliminal visual messages based on EEG signal and convolutional neural network

The visual information that can't be detected by consciousness but can affect individual's behavior and attitude under specific conditions is called subliminal visual messages. In order to better apply subliminal visual messages to commercial advertising, education and other fields, this paper studied the process of subliminal visual messages in the brain. First, this paper designed a experiment to allow the subjects to see a series of pictures stimulation of different durations and collect the EEG signals, then analyzed the impact of stimulation time on classification accuracy. The experimental results showed that when the stimulus time is short, the classification accuracy increases with the increase of time, resulting in subliminal visual effects. However, with the increase of stimulus time, the classification accuracy began to decline. We speculated that the visual information changed from subthreshold to suprathreshold. The subliminal visual effects were disturbed until disappeared.


Introduction
The visual information that can't be detected by consciousness but can affect individual's behavior and attitude under specific conditions is called subliminal visual messages [1]. The stimulation of the information to human is lower than the corresponding sensory threshold, and human consciousness can not detect the information, but it can stimulate people's unconscious response, affect the attitude tendency of the target object in a hidden way, and then affect their behavior.
With the development of unconscious research in cognitive psychology, the influence of subthreshold stimuli on cognitive, emotional, emotional, attention and will has gradually become another hot spot in cognitive psychology.
Previous studies on subliminal visual information mainly focused on visual subliminal persuasion and subliminal priming effect.
visual subliminal persuasion refers to the process in which subjects are stimulated by subliminal visual information and then try to influence their views, attitudes or behaviors [2]. The research of visual subliminal persuasion is mainly focused on the influence of subliminal visual information on behavior. It mainly involves three levels: the change of subliminal visual information on individual subjective experience [3], the change of subliminal visual information on individual behaviors [4], and the change of subliminal visual information on individual choices [5].
Subliminal priming effect refers to the phenomenon that an individual, without being aware of the presented stimulus, affects the subsequent processing of the relevant stimulus or the related behavioral intention or behavior [6]. The typical research mode of subliminal priming effect is to present subliminal priming stimulus first, and then present target stimulus above threshold. Subjects are required to respond to target stimulus to investigate the influence of subliminal priming stimulus on target stimulus processing.
Both of these two directions were to explore the changes of subjects after the occurrence of subthreshold visual effects, and had not studied the process of the occurrence of the subliminal visual effects. In the past studies, the main basis for judging whether subjects were affected by subliminal information is based on the changes of behavior and choice after subjects receive subliminal stimulation. The quality of the results depended on the subjective initiative of the subjects. Without an objective standard to restrict and judge them, it will lead to wrong answers and affect the results.
Therefore, an objective method to avoid the subjective influence of the subjects is urgently needed to study the process of subliminal visual information. With the development of research technology, the emergence of EEG and fMRI provided a new means for the study of subliminal visual information. Previous studies on EEG, ECG, and skin electricity showed that subthreshold stimulation could lead to significant changes in physiological indexes of subjects [7]. Brooks et al conducted a meta-analysis of studies on subliminal information using fMRI. The results showed that subthreshold stimulation can induce strong activation in amygdala, hippocampus, anterior cingulate gyrus, insular cortex and primary visual cortex [8].
There were differences in the areas of cerebral cortex activated by visual stimuli at different times, resulting in different characteristics of evoked EEG signals. The EEG signals caused by different visual stimuli are sent into the depth neural network to get the classification accuracy. The difference of classification accuracy may reflect the change process of visual information stimulus from subthreshold to suprathreshold. Based on this principle, a method based on EEG and convolution neural network is proposed to study the subliminal priming effect.

Data
In this paper, the subjects were in a quiet state and focused on the screen when collecting EEG signals from picture stimuli.

Experimental design
Ten kinds of pictures will appear on the screen, including airliner, banana, bike, bolete, broom, butterfly, camera, canoe, capuchin and cat. After each type of picture was played, the subjects took a ten seconds rest. The specific experimental process is as follows: (1)Three seconds before the experiment, the screen was in a black state, and a "ready" character appeared in the center of the screen to prompt the subjects that the screen was about to appear a picture.
(2) After 4-13 seconds of the experiment, the screen remained black, and the name of the type of picture to be presented appeared in the center of the screen.
(3) In the 14th second of the experiment, pictures appeared on the screen, and each picture was presented for t ms, and then the screen showed a gray background of (500-t) ms, 50 photos were presented at a time, and the sharing time was twenty five seconds. Each subject conducted seven experiments, and the size of t was 40, 60, 80, 100, 120, 200, 500.
(4) Repeat steps 2 and 3 for 10 times, and a total of 10 types of pictures are displayed on the screen.
(5) The screen becomes black, and the character "program end" appears in the center of the screen, lasting for two seconds, and the experiment ends.
(6) When the "ready" character appears on the computer screen, the EEG acquisition device starts to collect signals, and when the "program end" character in the center of the computer screen disappears, the EEG acquisition device finishes collecting signals. They should be centred above the tables and flush left beneath the figures.

EEG signal acquisition system
In this experiment, EEG signals were collected by wet electrode EEG acquisition system. The number of electrodes is 64. The sampling frequency is 1000 Hz. TP10 electrode was used as reference electrode.

EEG signal preprocess
EEG signal strength is weak, and it is easy to be interfered by various noises in the process of EEG signal acquisition, such as power frequency interference, eye electrical interference, ECG interference, electromyography interference and so on. In order to eliminate these interferences and avoid noise affecting the accuracy of signal classification, In this paper, we performed 0.5-100 Hz band-pass filter and 50 Hz band-stop filter. In addition, subjects should keep quiet and relaxed when collecting EEG signals.
Because the total time length of each picture and the gray background after the picture is 500ms, and the sampling frequency is 1000 Hz. So we cut the EEG signal into 63 * 500.
In this experiment, we choose EEG signals collected by 28 electrodes located in parietal lobe and 28 electrodes in occipital lobe. Because of the equipment defects, the phenomenon of frame dropping will occur in the process of stimulation. Only the first 480 collected signals are selected. Finally, the size of each EEG signals sample is 23 * 480. The experiment included 14 sample sets , and the size of each sample set is 450.

Methodology
Convolution neural network is a kind of neural network which is used to deal with data with similar grid structure. it is widely used in computer vision, signal classification and other fields.
Inspired by the classic CNN network structures such as LeNet [9] and AlexNet [10], a CNN network structure consists of a series of convolution layers, full connection layers and pooling layers is put forward in this paper.
In order to avoid over fitting, dropout function is added to this network [11]. The specific parameters of each layer of network are shown in Table 1, conv means convolution layer, mp means pooling layer, fc means full connection layer, relu means activation function, bn means Batch Normalization layer.
The training parameters of the network are as follows: learning rate is 0.002, batch size is 32, dropout is 0.5, The number of iterations is 500. This experiment hardware configuration is Intel i7-8700 CPU, 64g internal memory, gtx1080ti GPU. The operating system is Ubuntu 16.04, and the deep learning framework is Pytorch. All the codes are written in Python 3.7. Table 1. Main parameters of each layer of network.

Results and discussion
In order to verify the robustness of this CNN model in EEG feature learning, we use k-fold cross validation to verify. The occipital lobe signals collected when the visual stimulus duration was 500 ms was selected as the data set. 10% of the data set is used as the test set, and the remaining 90% is used for cross validation. The cross validation data set is randomly divided into 10 subsets, one of which is used as the verification machine and the remaining subset as the training set. The cross validation is completed ten times. The results are shown in table 2. k is the number of times, maxta is the maximum accuracy of the test set, tra is the accuracy of the corresponding training set. It can be seen from table 2 that the neural network model constructed in this paper has no over fitting to specific data, and it is stable and robust. Next, the EEG signals collected at 40 ms, 60 ms, 80 ms, 100 ms, 120 ms, 200 ms and 500 ms were selected as the data set, and 10% of the data set was selected as the test set and the remaining 90% as the training set.
The results are shown in Fig. 1. Acc1 is the classification accuracy of parietal lobe EEG signals, and acc2 is the classification accuracy of occipital lobe EEG signals. As can be seen from figure 1, the accuracy rate reaches the maximum value when the picture stimulation time is 80 ms and reaches the minimum value at 200 ms. After 80 ms, due to the increase of time, the activated brain areas are not only in the primary visual cortex, but also in the parietal and temporal cortex.
The EEG activities generated by parietal cortex and temporal cortex interfere with the EEG activities of primary visual cortex triggered by subthreshold visual information, resulting in the reduction of classification accuracy. After 200 ms, the visual information can be perceived by the subjects, the subliminal visual effects disappears, and the visual information enters a more advanced attention processing stage in the brain. The eventrelated potentials produced by parietal cortex and temporal cortex have more influence on EEG signals. With the further increase of visual information stimulation time, the eventrelated potentials produced by parietal cortex and temporal cortex gradually increased, and the accuracy rate increased.

Conclusion
The results of Experiment 1 showed that the neural network model is reliable and robust, and the results were credible. According to result 2, when the visual information stimulation time is short, the accuracy rate increased with the increase of stimulation time, which proves the existence of subthreshold visual effect; When the time of visual information stimulation is in a certain range, the subliminal visual effect and normal visual effect existed simultaneously. When the visual information stimulation time is greater than the upper limit of this interval, the subthreshold visual effect disappeared, which proves that there is a transformation process of visual information from subthreshold to suprathreshold with the increase of time.