A comprehensive survey on deep learning based malware detection techniques

Recent theoretical and practical studies have revealed that malware is one of the most harmful threats to the digital world. Malware mitigation techniques have evolved over the years to ensure security. Earlier, several classical methods were used for detecting malware embedded with various features like the signature, heuristic, and others. Traditional malware detection techniques were unable to defeat new generations of malware and their sophisticated obfuscation tactics. Deep Learning is increasingly used in malware detection as DL-based systems outperform conventional malware detection approaches at finding new malware variants. Furthermore, DL-based techniques provide rapid malware prediction with excellent detection rates and analysis of different malware types. Investigating recently proposed Deep Learning-based malware detection systems and their evolution is hence of interest to this work. It offers a thorough analysis of the recently developed DL-based malware detection techniques. Furthermore, current trending malwares are studied and detection techniques of Mobile malware (both Android and iOS), Windows malware, IoT malware, Advanced Persistent Threats (APTs), and Ransomware are precisely reviewed.     

MACSPMD:基于恶意API调用序列模式挖掘的恶意代码检测

基于动态分析的恶意代码检测方法由于能有效对抗恶意代码的多态和代码混淆技术,而且可以检测新的未知恶意代码等,因此得到了研究者的青睐。在这种情况下,恶意代码的编写者通过在恶意代码中嵌入大量反检测功能来逃避现有恶意代码动态检测方法的检测。针对该问题,提出了基于恶意API调用序列模式挖掘的恶意代码检测方法MACSPMD。首先,使用真机模拟恶意代码的实际运行环境来获取文件的动态API调用序列;其次,引入面向目标关联挖掘的概念,以挖掘出能够代表潜在恶意行为模式的恶意API调用序列模式;最后,将挖掘到的恶意API调用序列模式作为异常行为特征进行恶意代码的检测。基于真实数据集的实验结果表明,MACSPMD对未知和逃避型恶意代码进行检测的准确率分别达到了94.55%和97.73%,比其他基于API调用数据的恶意代码检测方法 的准确率分别提高了2.47%和2.66%,且挖掘过程消耗的时间更少。因此,MACSPMD能有效检测包括逃避型在内的已知和未知恶意代码。     

基于感知哈希算法和特征融合的恶意代码检测方法

在当前的恶意代码家族检测中,通过恶意代码灰度图像提取的局部特征或全局特征无法全面描述恶意代码,针对这个问题并为提高检测效率,提出了一种基于感知哈希算法和特征融合的恶意代码检测方法。首先,通过感知哈希算法对恶意代码灰度图样本进行检测,快速划分出具体恶意代码家族和不确定恶意代码家族的样本,实验测试表明约有67%的恶意代码能够通过感知哈希算法检测出来。然后,对于不确定恶意代码家族样本再进一步提取局部特征局部二值模式（LBP）与全局特征Gist,并利用二者融合后的特征通过机器学习算法对恶意代码样本进行分类检测。最后,对于25类恶意代码家族检测的实验结果表明,相较于仅用单一特征,使用LBP与Gist的融合特征时的检测准确率更高,并且所提方法与仅采用机器学习的检测算法相比分类检测效率更高,检测速度提高了93.5%。     

Software transformations to improve malware detection

Malware is code designed for a malicious purpose, such as obtaining root privilege on a host. A malware detector identifies malware and thus prevents it from adversely affecting a host. In order to evade detection, malware writers use various obfuscation techniques to transform their malware. There is strong evidence that commercial malware detectors are susceptible to these evasion tactics. In this paper, we describe the design and implementation of a malware transformer that reverses the obfuscations performed by a malware writer. Our experimental evaluation demonstrates that this malware transformer can drastically improve the detection rates of commercial malware detectors.     

Malware Pattern Scanning Schemes Secure Against Black-box Analysis

As a general rule, copycats produce most of malware variants from an original malware strain. For this purpose, they widely perform black-box analyses of commercial scanners aiming at extracting malware detection patterns. In this paper, we first study the malware detection pattern extraction problem from a complexity point of view and provide the results of a wide-scale study of commercial scanners’ black-box analysis. These results clearly show that most of the tested commercial products fail to thwart black-box analysis. Such weaknesses therefore urge copycats to produce even more malware variants. Then, we present a new model of malware detection pattern based on Boolean functions and identify some properties that a reliable detection pattern should have. Lastly, we describe a combinatorial, probabilistic malware pattern scanning scheme that, on the one hand, highly limits black-box analysis and on the other hand can only be bypassed in the case where there is collusion between a number of copycats. This scheme can incidentally provide some useful technical information to malware crime investigators, thus allowing a faster identification of copycats.     

Windows平台恶意软件智能检测综述

近年来,恶意软件给信息技术的发展带来了很多负面的影响.为了解决这一问题,如何有效检测恶意软件则一直备受关注.随着人工智能的迅速发展,机器学习与深度学习技术逐渐被引入到恶意软件的检测中,这类技术称之为恶意软件智能检测技术.相比于传统的检测方法,由于人工智能技术的应用,智能检测技术不需要人工制定检测规则.此外,具有更强的泛化能力,能够更好地检测先前未见过的恶意软件.恶意软件智能检测已经成为当前检测领域的研究热点.主要介绍了当前的恶意软件智能检测相关工作,包含了智能检测所需的主要环节.从智能检测中常用的特征、如何进行特征处理、智能检测中常用的分类器、当前恶意软件智能检测所面临的主要问题4个方面对智能检测相关工作进行了系统地阐述与分类.最后,总结了先前智能检测相关工作,阐明了未来潜在的研究方向,旨在能够助力恶意软件智能检测的发展.     

基于语义的恶意代码行为特征提取及检测方法

提出一种基于语义的恶意代码行为特征提取及检测方法,通过结合指令层的污点传播分析与行为层的语义分析,提取恶意代码的关键行为及行为间的依赖关系;然后,利用抗混淆引擎识别语义无关及语义等价行为,获取具有一定抗干扰能力的恶意代码行为特征.在此基础上,实现特征提取及检测原型系统.通过对多个恶意代码样本的分析和检测,完成了对该系统的实验验证.实验结果表明,基于上述方法提取的特征具有抗干扰能力强等特点,基于此特征的检测对恶意代码具有较好的识别能力.     

面向安卓恶意软件检测的对抗攻击技术综述

随着对Android恶意软件检测精度和性能要求的提高,越来越多的Android恶意软件检测引擎使用人工智能算法。与此同时,攻击者开始尝试对Android恶意软件进行一定的修改,使得Android恶意软件可以在保留本身的功能的前提下绕过这些基于人工智能算法的检测。上述过程即是Android恶意软件检测领域的对抗攻击。本文梳理了目前存在的基于人工智能算法的Android恶意软件检测模型,概述了针对Android恶意软件检测模型的对抗攻击方法,并从特征和算法两方面总结了相应的增强模型安全性的防护手段,最后提出了Android恶意软件检测模型和对抗攻击的发展趋势,并分析了对抗攻击对Android恶意软件检测的影响。     

基于行为的分布式恶意代码检测技术

针对已有恶意代码检测技术存在不足,研究恶意代码网络传播行为,提取相应行为特征,在此基础上提出基于行为的分布式恶意代码检测技术,并进行NS-2仿真实验。实验结果表明该方法具有较低的误报率和漏报率,可有效检测恶意代码。     

基于图模式与内存足迹的Android恶意应用与行为检测

现有的各个Android应用商店大多检查已知的静态恶意应用,难以检测新颖、动态加载的恶意应用与行为,对此提出一种基于图结构与内存足迹分析的恶意应用检测系统。首先,采集应用的内存信息,分析应用的足迹与序列号,检测动态打包的恶意代码与新颖的恶意应用;然后,提取应用所请求的共生权限,将权限建模为图结构,并使用图的度量指标分析图的分类模式与中心权限,根据中心权限值选择可表示各类的最优图指标;最终,计算应用的隐私分数与风险阈值,基于该阈值检测各种恶意软件或恶意行为。仿真实验的结果表明,本算法对不同类型的恶意应用均具有较好的效果,对于未知的恶意应用也具有较好的检测率。     

Joint network-host based malware detection using information-theoretic tools

In this paper, we propose two joint network-host based anomaly detection techniques that detect self-propagating malware in real-time by observing deviations from a behavioral model derived from a benign data profile. The proposed malware detection techniques employ perturbations in the distribution of keystrokes that are used to initiate network sessions. We show that the keystrokes’ entropy increases and the session-keystroke mutual information decreases when an endpoint is compromised by a self-propagating malware. These two types of perturbations are used for real-time malware detection. The proposed malware detection techniques are further compared with three prominent anomaly detectors, namely the maximum entropy detector, the rate limiting detector and the credit-based threshold random walk detector. We show that the proposed detectors provide considerably higher accuracy with almost 100% detection rates and very low false alarm rates.     

基于模型解释的PE文件对抗性恶意代码检测

深度学习已经逐渐应用于恶意代码检测并取得了不错的效果.然而,最近的研究表明:深度学习模型自身存在不安全因素,容易遭受对抗样本攻击.在不改变恶意代码原有功能的前提下,攻击者通过对恶意代码做少量修改,可以误导恶意代码检测器做出错误的决策,造成恶意代码的漏报.为防御对抗样本攻击,已有的研究工作中最常用的方法是对抗训练.然而对抗训练方法需要生成大量对抗样本加入训练集中重新训练模型,效率较低,并且防御效果受限于训练中所使用的对抗样本生成方法.为此,提出一种PE文件格式恶意代码对抗样本检测方法,针对在程序功能无关区域添加修改的一类对抗样本攻击,利用模型解释技术提取端到端恶意代码检测模型的决策依据作为特征,进而通过异常检测方法准确识别对抗样本.该方法作为恶意代码检测模型的附加模块,不需要对原有模型做修改,相较于对抗训练等其他防御方法效率更高,且具有更强的泛化能力,能够防御多种对抗样本攻击.在真实的恶意代码数据集上进行了实验,实验结果表明,该方法能够有效防御针对端到端PE文件恶意代码检测模型的对抗样本攻击.     

Intelligent OS X malware threat detection with code inspection

With the increasing market share of Mac OS X operating system, there is a corresponding increase in the number of malicious programs (malware) designed to exploit vulnerabilities on Mac OS X platforms. However, existing manual and heuristic OS X malware detection techniques are not capable of coping with such a high rate of malware. While machine learning techniques offer promising results in automated detection of Windows and Android malware, there have been limited efforts in extending them to OS X malware detection. In this paper, we propose a supervised machine learning model. The model applies kernel base Support Vector Machine and a novel weighting measure based on application library calls to detect OS X malware. For training and evaluating the model, a dataset with a combination of 152 malware and 450 benign were created. Using common supervised Machine Learning algorithm on the dataset, we obtain over 91% detection accuracy with 3.9% false alarm rate. We also utilize Synthetic Minority Over-sampling Technique (SMOTE) to create three synthetic datasets with different distributions based on the refined version of collected dataset to investigate impact of different sample sizes on accuracy of malware detection. Using SMOTE datasets we could achieve over 96% detection accuracy and false alarm of less than 4%. All malware classification experiments are tested using cross validation technique. Our results reflect that increasing sample size in synthetic datasets has direct positive effect on detection accuracy while increases false alarm rate in compare to the original dataset.     

基于非用户操作序列的恶意软件检测方法

针对Android恶意软件持续大幅增加的现状以及恶意软件检测能力不足这一问题,提出了一种基于非用户操作序列的静态检测方法。首先,通过对恶意软件进行逆向工程分析,提取出恶意软件的应用程序编程接口（API）调用信息;然后,采用广度优先遍历算法构建恶意软件的函数调用流程图;进而,从函数流程图中提取出其中的非用户操作序列形成恶意行为库;最后,采用编辑距离算法计算待检测样本与恶意行为库中的非用户操作序列的相似度进行恶意软件识别。在对360个恶意样本和300的正常样本进行的检测中,所提方法可达到90.8%的召回率和90.3%的正确率。与Android恶意软件检测系统Androguard相比,所提方法在恶意样本检测中召回率提高了30个百分点;与FlowDroid方法相比,所提方法在正常样本检测中准确率提高了11个百分点,在恶意样本检测中召回率提高了4.4个百分点。实验结果表明,所提方法提高了恶意软件检测的召回率,有效提升恶意软件的检测效果。     

DroidChain: A novel Android malware detection method based on behavior chains

The drastic increase of Android malware has led to strong interest in automating malware analysis. In this paper, to fight against malware variants and zero-day malware, we proposed DroidChain: a method combining static analysis and a behavior chain model. We transform the malware detection problem into more accessible matrix form. Using this method, we propose four kinds of malware models, including privacy leakage, SMS financial charges, malware installation, and privilege escalation. To reduce time complexity, we propose the WxShall-extend algorithm. We had moved the prototype to GitHub and evaluate using 1260 malware samples. Experimental malware detection results demonstrate accuracy, precision, and recall of 73%–93%, 71%–99%, and 42%–92%, respectively. Calculation time accounts for 6.58% of the well-known Warshall algorithm’s expense. Results demonstrate that our method, which can detect four kinds of malware simultaneously, is better than Androguard and Kirin.     

Detecting machine-morphed malware variants via engine attribution

One method malware authors use to defeat detection of their programs is to use morphing engines to rapidly generate a large number of variants. Inspired by previous works in author attribution of natural language text, we investigate a problem of attributing a malware to a morphing engine. Specifically, we present the malware engine attribution problem and formally define its three variations: MVRP, DENSITY and GEN, that reflect the challenges malware analysts face nowadays. We design and implement heuristics to address these problems and show their effectiveness on a set of well-known malware morphing engines and a real-world malware collection reaching detection accuracies of 96 % and higher. Our experiments confirm the applicability of the proposed approach in practice and indicate that engine attribution may offer a viable enhancement of current defenses against malware.     

一种基于主动学习的恶意代码检测方法

现有恶意代码的检测往往依赖于对足够数量样本的分析.然而新型恶意代码大量涌现,其出现之初,样本数量有限,现有方法无法迅速检测出新型恶意代码及其变种.本文在数据流依赖网络中分析进程访问行为异常度与相似度,引入了恶意代码检测估计风险,并提出一种通过最小化估计风险实现主动学习的恶意代码检测方法.该方法只需要很少比例的训练样本就可实现准确的恶意代码检测,较现有方法更适用于新型恶意代码检测.通过我们对真实的8,340个正常进程以及7,257个恶意代码进程的实验分析,相比于传统基于统计分类器的检测方法,本文方法明显地提升了恶意代码检测效果.即便在训练样本仅为总体样本数量1%的情况下,本文方法可以也可达到5.55%的错误率水平,比传统方法降低了36.5%.     

融合多特征的Android恶意软件检测方法

针对当前基于机器学习的Android恶意软件检测方法特征构建维度单一,难以全方位表征Android恶意软件行为特点的问题,文章提出一种融合软件行为特征、Android Manifest.xml文件结构特征和Android恶意软件分析经验特征的恶意软件检测方法。该方法提取Android应用的Dalvik操作码N-gram语义信息、系统敏感API、系统Intent、系统Category、敏感权限和相关经验特征,多方位表征Android恶意软件的行为并构建特征向量,采用基于XGBoost的集成学习算法构建分类模型,实现对恶意软件的准确分类。在公开数据集DREBIN和AMD上进行实验,实验结果表明,该方法能够达到高于97%的检测准确率,有效提升了Android恶意软件的检测效果。     

Automatic malware classification and new malware detection using machine learning

The explosive growth of malware variants poses a major threat to information security. Traditional anti-virus systems based on signatures fail to classify unknown malware into their corresponding families and to detect new kinds of malware programs. Therefore, we propose a machine learning based malware analysis system, which is composed of three modules: data processing, decision making, and new malware detection. The data processing module deals with gray-scale images, Opcode n-gram, and import functions, which are employed to extract the features of the malware. The decision-making module uses the features to classify the malware and to identify suspicious malware. Finally, the detection module uses the shared nearest neighbor (SNN) clustering algorithm to discover new malware families. Our approach is evaluated on more than 20 000 malware instances, which were collected by Kingsoft, ESET NOD32, and Anubis. The results show that our system can effectively classify the unknown malware with a best accuracy of 98.9%, and successfully detects 86.7% of the new malware.