Malicious sequential pattern mining for automatic malware detection

Due to its damage to Internet security, malware (e.g., virus, worm, trojan) and its detection has caught the attention of both anti-malware industry and researchers for decades. To protect legitimate users from the attacks, the most significant line of defense against malware is anti-malware software products, which mainly use signature-based method for detection. However, this method fails to recognize new, unseen malicious executables. To solve this problem, in this paper, based on the instruction sequences extracted from the file sample set, we propose an effective sequence mining algorithm to discover malicious sequential patterns, and then All-Nearest-Neighbor (ANN) classifier is constructed for malware detection based on the discovered patterns. The developed data mining framework composed of the proposed sequential pattern mining method and ANN classifier can well characterize the malicious patterns from the collected file sample set to effectively detect newly unseen malware samples. A comprehensive experimental study on a real data collection is performed to evaluate our detection framework. Promising experimental results show that our framework outperforms other alternate data mining based detection methods in identifying new malicious executables.     

A novel malware for subversion of self‐protection in anti‐virus

Major anti‐virus solutions have introduced a feature known as ‘self‐protection’ so that malware (and even users) cannot modify or disable the core functionality of their products. In this paper, we have investigated 12 anti‐virus products from four vendors (AVG, Avira, McAfee and Symantec) and have discovered that they have certain security weaknesses that can be exploited by malware. We have then designed a novel malware, which makes use of the weaknesses in anti‐virus software and embeds itself to become a part of the vulnerable anti‐virus solution. It subverts the self‐protection features of several anti‐virus software solutions. This malware integrated anti‐virus enjoys several advantages such as longevity (anti‐virus is active while the system is running), improved stealthy behaviour, highest privilege and capability to bypass security measures. Then we propose an effective defence against such malware. We have also implemented the defensive measure and evaluated its effectiveness. Finally, we show how the proposed defence can be applied to the current versions of vulnerable anti‐virus solutions without requiring signficant modifications. Copyright © 2015 John Wiley & Sons, Ltd.     

Android malware detection through hybrid features fusion and ensemble classifiers: The AndroPyTool framework and the OmniDroid dataset

Cybersecurity has become a major concern for society, mainly motivated by the increasing number of cyber attacks and the wide range of targeted objectives. Due to the popularity of smartphones and tablets, Android devices are considered an entry point in many attack vectors. Malware applications are among the most used tactics and tools to perpetrate a cyber attack, so it is critical to study new ways of detecting them. In these detection mechanisms, machine learning has been used to build classifiers that are effective in discerning if an application is malware or benignware. However, training such classifiers require big amounts of labelled data which, in this context, consist of categorised malware and benignware Android applications represented by a set of features able to describe their behaviour. For that purpose, in this paper we present OmniDroid, a large and comprehensive dataset of features extracted from 22,000 real malware and goodware samples, aiming to help anti-malware tools creators and researchers when improving, or developing, new mechanisms and tools for Android malware detection. Furthermore, the characteristics of the dataset make it suitable to be used as a benchmark dataset to test classification and clustering algorithms or new representation techniques, among others. The dataset has been released under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License and was built using AndroPyTool, our automated framework for dynamic and static analysis of Android applications. Finally, we test a set of ensemble classifiers over this dataset and propose a malware detection approach based on the fusion of static and dynamic features through the combination of ensemble classifiers. The experimental results show the feasibility and potential usability (for the machine learning, soft computing and cyber security communities) of our automated framework and the publicly available dataset.     

Holography: a behavior‐based profiler for malware analysis

Behavior‐based detection and signature‐based detection are two popular approaches to malware (malicious software) analysis. The security industry, such as the sector selling antivirus tools, has been using signature and heuristic‐based technologies for years. However, this approach has been proven to be inefficient in identifying unknown malware strains. On the other hand, the behavior‐based malware detection approach has a greater potential in identifying previously unknown instances of malicious software. The accuracy of this approach relies on techniques to profile and recognize accurate behavior models. Unfortunately, with the increasing complexity of malicious software and limitations of existing automatic tools, the current behavior‐based approach cannot discover many newer forms of malware either. In this paper, we implement ‘holography platform’, a behavior‐based profiler on top of a virtual machine emulator that intercepts the system processes and analyzes the CPU instructions, CPU registers, and memory. The captured information is stored in a relational database, and data mining techniques are used to extract information. We demonstrate the breadth of the ‘holography platform’ by conducting two experiments: a packed binary behavior analysis and a malvertising (malicious advertising) incident tracing. Both tasks are known to be very difficult to do efficiently using existing methods and tools. We demonstrate how the precise behavior information can be easily obtained using the ‘holography platform’ tool. With these two experiments, we show that the ‘holography platform’ can provide security researchers and automatic malware detection systems with an efficient malicious software behavior analysis solution. Copyright © 2011 John Wiley & Sons, Ltd.     

Structural analysis of binary executable headers for malware detection optimization

In the context of the OpenDAVFI project (a fork of the French initiative DAVFI for giving birth to a new generation, open antivirus engine which has been funded by the French Government), different AV filters have been developped and chained to detect both known and unknown malware very accurately while requiring a very limited number of updates. While most AV software use different static and dynamic detection techniques which are mostly based on the general concept of (static or heuristic) signature, we have observed that many malware do not comply to the Microsoft specifications with respect to the MZ-PE format. In this technical correspondence, we present structural analysis tests which have been implemented in the DAVFI/OpenDAVFi project. These tests accurately detect malware and therefore greatly reduce the number of malware that have to be analyzed by subsequent modules in our detection chain.     

Fingerprinting Android malware families

The domination of the Android operating system in the market share of smart terminals has engendered increasing threats of malicious applications (apps). Research on Android malware detection has received considerable attention in academia and the industry. In particular, studies on malware families have been beneficial to malware detection and behavior analysis. However, identifying the characteristics of malware families and the features that can describe a particular family have been less frequently discussed in existing work. In this paper, we are motivated to explore the key features that can classify and describe the behaviors of Android malware families to enable fingerprinting the malware families with these features.We present a framework for signature-based key feature construction. In addition, we propose a frequency-based feature elimination algorithm to select the key features. Finally, we construct the fingerprints of ten malware families, including twenty key features in three categories. Results of extensive experiments using Support Vector Machine demonstrate that the malware family classification achieves an accuracy of 92% to 99%. The typical behaviors of malware families are analyzed based on the selected key features. The results demonstrate the feasibility and effectiveness of the presented algorithm and fingerprinting method.     

基于环境敏感分析的恶意代码脱壳方法

加壳技术是软件的常用保护手段,但也常被恶意代码用于躲避杀毒软件的检测.通用脱壳工具根据加壳恶意代码运行时的行为特征或统计特征进行脱壳,需要建立监控环境,因此易受环境敏感技术的干扰.文中提出了一种基于环境敏感分析的恶意代码脱壳方法,利用动静结合的分析技术检测并清除恶意代码的环境敏感性.首先,利用中间语言对恶意代码的执行轨迹进行形式化表示;然后,分析执行轨迹中环境敏感数据的来源和传播过程,提取脱壳行为的环境约束;最后,求解环境约束条件,根据求解结果对恶意代码进行二进制代码插装,清除其环境敏感性.基于此方法,作者实现了一个通用的恶意代码脱壳工具:MalUnpack,并对321个最新的恶意代码样本进行了对比实验.实验结果表明MalUnpack能有效对抗恶意代码的环境敏感技术,其脱壳率达到了89.1%,显著高于现有基于动态监控的通用脱壳工具的35.5%和基于特征的定向脱壳工具的28.0%.     

MACSPMD:基于恶意API调用序列模式挖掘的恶意代码检测

基于动态分析的恶意代码检测方法由于能有效对抗恶意代码的多态和代码混淆技术,而且可以检测新的未知恶意代码等,因此得到了研究者的青睐。在这种情况下,恶意代码的编写者通过在恶意代码中嵌入大量反检测功能来逃避现有恶意代码动态检测方法的检测。针对该问题,提出了基于恶意API调用序列模式挖掘的恶意代码检测方法MACSPMD。首先,使用真机模拟恶意代码的实际运行环境来获取文件的动态API调用序列;其次,引入面向目标关联挖掘的概念,以挖掘出能够代表潜在恶意行为模式的恶意API调用序列模式;最后,将挖掘到的恶意API调用序列模式作为异常行为特征进行恶意代码的检测。基于真实数据集的实验结果表明,MACSPMD对未知和逃避型恶意代码进行检测的准确率分别达到了94.55%和97.73%,比其他基于API调用数据的恶意代码检测方法 的准确率分别提高了2.47%和2.66%,且挖掘过程消耗的时间更少。因此,MACSPMD能有效检测包括逃避型在内的已知和未知恶意代码。     

Classification of malware based on integrated static and dynamic features

Collection of dynamic information requires that malware be executed in a controlled environment; the malware unpacks itself as a preliminary to the execution process. On the other hand, while execution of malware is not needed in order to collect static information, the file must first be unpacked manually. None-the-less, if a file has been executed, it is possible to use both static and dynamic information in designing a single classification method.In this paper, we present the first classification method integrating static and dynamic features into a single test. Our approach improves on previous results based on individual features and reduces by half the time needed to test such features separately.Robustness to changes in malware development is tested by comparing results on two sets of malware, the first collected between 2003 and 2007, and the second collected between 2009 and 2010. When classifying the older set as compared to the entire data set, our integrated test demonstrates significantly more robustness than previous methods by losing just 2.7% in accuracy as opposed to a drop of 7%. We conclude that to achieve acceptable accuracy in classifying the latest malware, some older malware should be included in the set of data.     

Enhanced capsule network-based executable files malware detection and classification—deep learning approach

Malware has considerably increased recently, posing a serious security danger to both people and enterprises. In order to distinguish and stop the negative effects of malware, a variety of machine and deep learning approaches have been used to detect it. However, while extracting malware features, the feature-to-feature spatial hierarchy is not taken into account by the existing techniques and as a result, information is lost during the pooling operation. Hence, a modified capsule deep neural network was developed in which discriminative features are extracted from three channel image derived from malware binary with considering feature-to-feature spatial hierarchy. Also, conventional capsule deep neural network is modified by adding a global average pooling layer before fully connected layer thereby classified the dataset as malicious or benign without any loss of information. Moreover, these malwares were not accurately classified based on their families using existing variants of convolutional neural network (CNN) since malware family variants can modify due to minute changes in malware binaries. Hence, a hybrid deep convolutional neural network (DCNN) and long-short-term memory (LSTM) has been utilized that determine minute changes in malware binaries using LSTM without vanishing gradient issue and effectively perform malware family classification using DCNN. As a result, the proposed approach successfully identifies malware in executable files and categorizes malware into families with 98.5% accuracy.     

Hybrids of support vector machine wrapper and filter based framework for malware detection

Malware replicates itself and produces offspring with the same characteristics but different signatures by using code obfuscation techniques. Current generation Anti-Virus (AV) engines employ a signature-template type detection approach where malware can easily evade existing signatures in the database. This reduces the capability of current AV engines in detecting malware. In this paper we propose a hybrid framework for malware detection by using the hybrids of Support Vector Machines Wrapper, Maximum-Relevance–Minimum-Redundancy Filter heuristics where Application Program Interface (API) call statistics are used as a malware features. The novelty of our hybrid framework is that it injects the filter’s ranking score in the wrapper selection process and combines the properties of both wrapper and filters and API call statistics which can detect malware based on the nature of infectious actions instead of signature. To the best of our knowledge, this kind of hybrid approach has not been explored yet in the literature in the context of feature selection and malware detection. Knowledge about the intrinsic characteristics of malicious activities is determined by the API call statistics which is injected as a filter score into the wrapper’s backward elimination process in order to find the most significant APIs. While using the most significant APIs in the wrapper classification on both obfuscated and benign types malware datasets, the results show that the proposed hybrid framework clearly surpasses the existing models including the independent filters and wrappers using only a very compact set of significant APIs. The performances of the proposed and existing models have further been compared using binary logistic regression. Various goodness of fit comparison criteria such as Chi Square, Akaike’s Information Criterion (AIC) and Receiver Operating Characteristic Curve ROC are deployed to identify the best performing models. Experimental outcomes based on the above criteria also show that the proposed hybrid framework outperforms other existing models of signature types including independent wrapper and filter approaches to identify malware.     

Hierarchical associative classifier (HAC) for malware detection from the large and imbalanced gray list

Nowadays, numerous attacks made by the malware (e.g., viruses, backdoors, spyware, trojans and worms) have presented a major security threat to computer users. Currently, the most significant line of defense against malware is anti-virus products which focus on authenticating valid software from a whitelist, blocking invalid software from a blacklist, and running any unknown software (i.e., the gray list) in a controlled manner. The gray list, containing unknown software programs which could be either normal or malicious, is usually authenticated or rejected manually by virus analysts. Unfortunately, along with the development of the malware writing techniques, the number of file samples in the gray list that need to be analyzed by virus analysts on a daily basis is constantly increasing. The gray list is not only large in size, but also has an imbalanced class distribution where malware is the minority class. In this paper, we describe our research effort on building automatic, effective, and interpretable classifiers resting on the analysis of Application Programming Interfaces (APIs) called by Windows Portable Executable (PE) files for detecting malware from the large and imbalanced gray list. Our effort is based on associative classifiers due to their high interpretability as well as their capability of discovering interesting relationships among API calls. We first adapt several different post-processing techniques of associative classification, including rule pruning and rule re-ordering, for building effective associative classifiers from large collections of training data. In order to help the virus analysts detect malware from the imbalanced gray list, we then develop the Hierarchical Associative Classifier (HAC). HAC constructs a two-level associative classifier to maximize precision and recall of the minority (malware) class: in the first level, it uses high precision rules of majority (benign file samples) class and low precision rules of minority class to achieve high recall; and in the second level, it ranks the minority class files and optimizes the precision. Finally, since our case studies are based on a large and real data collection obtained from the Anti-virus Lab of Kingsoft corporation, including 8,000,000 malware, 8,000,000 benign files, and 100,000 file samples from the gray list, we empirically examine the sampling strategy to build the classifiers for such a large data collection to avoid over-fitting and achieve great effectiveness as well as high efficiency. Promising experimental results demonstrate the effectiveness and efficiency of the HAC classifier. HAC has already been incorporated into the scanning tool of Kingsoft’s Anti-Virus software.     

Metamorphic Malware Detection Using Code Metrics

ABSTRACT

Malware is becoming more and more aggressive and new techniques are emerging to allow malicious code to evade detection by antiviruses. Metamorphic malware is a particularly insidious kind of virus that changes its form at each infection. In this article, a technique for detecting metamorphic viruses is proposed that is based on identifying specific features of the assembly code, such as the instructions that change the contents of the registers, the instructions that change the control flow, and the potential code fragmentation. Such features have been derived by the analysis of a large dataset of malware. The experimentation suggests that the proposed technique produces very high precision (over 97%) in recognizing metamorphic malware, and allows also for distinguishing among different families of malware.     

Using ontologies to perform threat analysis and develop defensive strategies for mobile security

Existing studies on the detection of mobile malware have focused mainly on static analyses performed to examine the code-structure signature of viruses, rather than the dynamic behavioral aspects. By contrast, the unidentified behavior of new mobile viruses using the self-modification, polymorphic, and mutation techniques for variants have largely been ignored. The problem of precision regarding malware variant detection has become one of the key concerns in mobile security. Accordingly, the present study proposed a threat risk analysis model for mobile viruses, using a heuristic approach incorporating both malware behavior analysis and code analysis to generate a virus behavior ontology associated with the Protégé platform. The proposed model can not only explicitly identify an attack profile in accordance with structural signature of mobile viruses, but also overcome the uncertainty regarding the probability of an attack being successful. This model is able to achieve this by extending frequent episode rules to investigate the attack profile of a given malware, using specific event sequences associated with the sandbox technique for mobile applications (apps) and hosts. For probabilistic analysis, defense evaluation metrics for each node were used to simulate the results of an attack. The simulations focused specifically on the attack profile of a botnet to assess the threat risk. The validity of the proposed approach was demonstrated numerically by using two malware cyber-attack examples. Overall, the results presented in this paper prove that the proposed scheme offers an effective countermeasure, evaluated using a set of security metrics, for mitigating network threats by considering the interaction between the attack profiles and defense needs.     

A statistical model for undecidable viral detection

This paper presents a statistical model of the malware detection problem. Where Chess and White (An undetectable computer virus. In: Virus Bulletin Conference, 2000) just partially addressed this issue and gave only existence results, we give here constructive results of undetectable malware. We show that any existing detection techniques can be modelled by one or more statistical tests. Consequently, the concepts of false positive and non detection are precisely defined. The concept of test simulability is then presented and enables us to gives constructive results how undetectable malware could be developped by an attacker. Practical applications of this statistical model are proposed. Finally, we give a statistical variant of Cohen’s undecidability results of virus detection.     

Fragmented malware through RFID and its defenses

Malware, in essence, is an infiltration to one’s computer system. Malware is created to wreak havoc once it gets in through weakness in a computer’s barricade. Anti-virus companies and operating system companies are working to patch weakness in systems and to detect infiltrators. However, with the advance of fragmentation, detection might even prove to be more difficult. Malware detection relies on signatures to identify malware of certain shapes. With fragmentation, functionality and size can change depending on how many fragments are used and how the fragments are created. In this paper we present a robust malware detection technique, with emphasis on detecting fragmentation malware attacks in RFID systems that can be extended to detect complex obfuscated and mutated malware. After a particular fragmented malware has been first identified, it can be analyzed to extract the signature, which provides a basis for detecting variants and mutants of similar types of malware in the future. Encouraging experimental results on a limited set of recent malware are presented.     

Titans’ revenge: Detecting Zeus via its own flaws

Malware is one of the main threats to the Internet security in general, and to commercial transactions in particular. However, given the high level of sophistication reached by malware (e.g. usage of encrypted payload and obfuscation techniques), malware detection tools and techniques still call for effective and efficient solutions. In this paper, we address a specific, dreadful, and widely diffused financial malware: Zeus.The contributions of this paper are manifold: first, we propose a technique to break the encrypted malware communications, extracting the keystream used to encrypt such communications; second, we provide a generalization of the proposed keystream extraction technique. Further, we propose Cronus, an IDS that specifically targets Zeus malware. The implementation of Cronus has been experimentally tested on a production network, and its high quality performance and effectiveness are discussed. Finally, we highlight some principles underlying malware—and Zeus in particular—that could pave the way for further investigation in this field.     

Acquisition of hierarchy-structured probabilistic decision tables and rules from data

Abstract: The paper is concerned with the creation of predictive models from data within the framework of the variable precision rough set model. It is focused on two aspects of the model derivation: computation of uncertain, in general, rules from information contained in probabilistic decision tables and forming hierarchies of decision tables with the objective of reduction or elimination of decision boundaries in the resulting classifiers. A new technique of creation of a linearly structured hierarchy of decision tables is introduced and compared to tree‐structured hierarchy. It is argued that the linearly structured hierarchy has significant advantages over tree‐structured hierarchy.     

Deciphering malware’s use of TLS (without decryption)

The use of TLS by malware poses new challenges to network threat detection because traditional pattern-matching techniques can no longer be applied to its messages. However, TLS also introduces a complex set of observable data features that allow many inferences to be made about both the client and the server. We show that these features can be used to detect and understand malware communication, while at the same time preserving the privacy of the benign uses of encryption. These data features also allow for accurate malware family attribution of network communication, even when restricted to a single, encrypted flow. To demonstrate this, we performed a detailed study of how TLS is used by malware and enterprise applications. We provide a general analysis on millions of TLS encrypted flows, and a targeted study on 18 malware families composed of thousands of unique malware samples and tens-of-thousands of malicious TLS flows. Importantly, we identify and accommodate for the bias introduced by the use of a malware sandbox. We show that the performance of a malware classifier is correlated with a malware family’s use of TLS, i.e., malware families that actively evolve their use of cryptography are more difficult to classify. We conclude that malware’s usage of TLS is distinct in an enterprise setting, and that these differences can be effectively used in rules and machine learning classifiers.