MalDetect: A Structure of Encrypted Malware Traffic Detection

Recently, TLS protocol has been widely used to secure the application data carried in network traffic. It becomes more difficult for attackers to decipher messages through capturing the traffic generated from communications of hosts. On the other hand, malwares adopt TLS protocol when accessing to internet, which makes most malware traffic detection methods, such as DPI (Deep Packet Inspection), ineffective. Some literatures use statistical method with extracting the observable data fields exposed in TLS connections to train machine learning classifiers so as to infer whether a traffic flow is malware or not. However, most of them adopt the features based on the complete flow, such as flow duration, but seldom consider that the detection result should be given out as soon as possible. In this paper, we propose MalDetect, a structure of encrypted malware traffic detection. MalDetect only extracts features from approximately 8 packets (the number varies in different flows) at the beginning of traffic flows, which makes it capable of detecting malware traffic before the malware behaviors take practical impacts. In addition, observing that it is inefficient and time-consuming to re-train the offline classifier when new flow samples arrive, we deploy Online Random Forest in MalDetect. This enables the classifier to update its parameters in online mode and gets rid of the re-training process. MalDetect is coded in C++ language and open in Github. Furthermore, MalDetect is thoroughly evaluated from three aspects: effectiveness, timeliness and performance.