A comprehensive survey on deep learning based malware detection techniques

Recent theoretical and practical studies have revealed that malware is one of the most harmful threats to the digital world. Malware mitigation techniques have evolved over the years to ensure security. Earlier, several classical methods were used for detecting malware embedded with various features like the signature, heuristic, and others. Traditional malware detection techniques were unable to defeat new generations of malware and their sophisticated obfuscation tactics. Deep Learning is increasingly used in malware detection as DL-based systems outperform conventional malware detection approaches at finding new malware variants. Furthermore, DL-based techniques provide rapid malware prediction with excellent detection rates and analysis of different malware types. Investigating recently proposed Deep Learning-based malware detection systems and their evolution is hence of interest to this work. It offers a thorough analysis of the recently developed DL-based malware detection techniques. Furthermore, current trending malwares are studied and detection techniques of Mobile malware (both Android and iOS), Windows malware, IoT malware, Advanced Persistent Threats (APTs), and Ransomware are precisely reviewed.     

A Survey on malware analysis and mitigation techniques

In recent days, malwares are advanced, sophisticatedly engineered to attack the target. Most of such advanced malwares are highly persistent and capable of escaping from the security systems. This paper explores such an advanced malware type called Advanced Persistent Threats (APTs). APTs pave the way for most of the Cyber espionages and sabotages. APTs are highly sophisticated, target specific and operate in a stealthy mode till the target is compromised. The intention of the APTs is to deploy target specific automated malwares in a host or network to initiate an on-demand attack based on continuous monitoring. Encrypted covert communication and advanced, sophisticated attack techniques make the identification of APTs more challenging. Conventional security systems like antivirus, anti-malware systems which depend on signatures and static analysis fail to identify these APTs. The Advanced Evasive Techniques (AET) used in APTs are capable of bypassing the stateful firewalls housed in the enterprise choke points at ease. Hence, this paper presents a detailed study on sophisticated attack and evasion techniques used by the contemporary malwares. Furthermore, existing malware analysis techniques, application hardening techniques and CPU assisted application security schemes are also discussed. Finally, the study concludes by presenting the System and Network Security Design (SNSD) using existing mitigation techniques.     

Software transformations to improve malware detection

Malware is code designed for a malicious purpose, such as obtaining root privilege on a host. A malware detector identifies malware and thus prevents it from adversely affecting a host. In order to evade detection, malware writers use various obfuscation techniques to transform their malware. There is strong evidence that commercial malware detectors are susceptible to these evasion tactics. In this paper, we describe the design and implementation of a malware transformer that reverses the obfuscations performed by a malware writer. Our experimental evaluation demonstrates that this malware transformer can drastically improve the detection rates of commercial malware detectors.     

Nonnegative matrix factorization and metamorphic malware detection

Metamorphic malware change their internal code structure by adopting code obfuscation technique while maintaining their malicious functionality during each infection. This causes change of their signature pattern across each infection and makes signature based detection particularly difficult. In this paper, through static analysis, we use similarity score from matrix factorization technique called Nonnegative Matrix Factorization for detecting challenging metamorphic malware. We apply this technique using structural compression ratio and entropy features and compare our results with previous eigenvector-based techniques. Experimental results from three malware datasets show this is a promising technique as the accuracy detection is more than 95%.

     

Music classification as a new approach for malware detection

Each year, a huge number of malicious programs are released which causes malware detection to become a critical task in computer security. Antiviruses use various methods for detecting malware, such as signature-based and heuristic-based techniques. Polymorphic and metamorphic malwares employ obfuscation techniques to bypass traditional detection methods used by antiviruses. Recently, the number of these malware has increased dramatically. Most of the previously proposed methods to detect malware are based on high-level features such as opcodes, function calls or program’s control flow graph (CFG). Due to new obfuscation techniques, extracting high-level features is tough, fallible and time-consuming; hence approaches using program’s bytes are quicker and more accurate. In this paper, a novel byte-level method for detecting malware by audio signal processing techniques is presented. In our proposed method, program’s bytes are converted to a meaningful audio signal, then Music Information Retrieval (MIR) techniques are employed to construct a machine learning music classification model from audio signals to detect new and unseen instances. Experiments evaluate the influence of different strategies converting bytes to audio signals and the effectiveness of the method.

     

The blockchain potential in computer virology: leveraging combinatorial techniques of k-ary codes

The threat landscape is continuously evolving and attackers are improving their tactics and techniques. From worms and viruses, initially introduced in 1982, to advanced, targeted and persistent attacks that have emerged in recent years, many verdicts demonstrate that no architecture is invulnerable. Nowadays, malware and cyberthreats are penetrating many platforms and the growth is exponential and a corporate and politically-driven outbreak has surfaced worldwide. A continuous back-and-forth between vulnerabilities and controls directs the evolution of the information age. Besides, intelligent technologies are a dual-use and a new class of smart cyberthreats is arisen. This paper presents a state of the art in computer virology and explores how we leveraged the blockchain technology to create a new form of malware offering a new aspect to the cyber-vector.

     

A general definition of malware

We propose a general, formal definition of the concept of malware (malicious software) as a single sentence in the language of a certain modal logic. Our definition is general thanks to its abstract formulation, which, being abstract, is independent of—but nonetheless generally applicable to—the manifold concrete manifestations of malware. From our formulation of malware, we derive equally general and formal definitions of benware (benign software), anti-malware (“antibodies” against malware), and medware (medical software or “medicine” for affected software). We provide theoretical tools and practical techniques for the detection, comparison, and classification of malware and its derivatives. Our general defining principle is causation of (in)correctness.     

Metamorphic Malware Detection Using Code Metrics

ABSTRACT

Malware is becoming more and more aggressive and new techniques are emerging to allow malicious code to evade detection by antiviruses. Metamorphic malware is a particularly insidious kind of virus that changes its form at each infection. In this article, a technique for detecting metamorphic viruses is proposed that is based on identifying specific features of the assembly code, such as the instructions that change the contents of the registers, the instructions that change the control flow, and the potential code fragmentation. Such features have been derived by the analysis of a large dataset of malware. The experimentation suggests that the proposed technique produces very high precision (over 97%) in recognizing metamorphic malware, and allows also for distinguishing among different families of malware.     

Intelligent and behavioral-based detection of malware in IoT spectrum sensors

The number of Cyber-Physical Systems (CPS) available in industrial environments is growing mainly due to the evolution of the Internet-of-Things (IoT) paradigm. In such a context, radio frequency spectrum sensing in industrial scenarios is one of the most interesting applications of CPS due to the scarcity of the spectrum. Despite the benefits of operational platforms, IoT spectrum sensors are vulnerable to heterogeneous malware. The usage of behavioral fingerprinting and machine learning has shown merit in detecting cyberattacks. Still, there exist challenges in terms of (i) designing, deploying, and evaluating ML-based fingerprinting solutions able to detect malware attacks affecting real IoT spectrum sensors, (ii) analyzing the suitability of kernel events to create stable and precise fingerprints of spectrum sensors, and (iii) detecting recent malware samples affecting real IoT spectrum sensors of crowdsensing platforms. Thus, this work presents a detection framework that applies device behavioral fingerprinting and machine learning to detect anomalies and classify different botnets, rootkits, backdoors, ransomware and cryptojackers affecting real IoT spectrum sensors. Kernel events from CPU, memory, network, file system, scheduler, drivers, and random number generation have been analyzed, selected, and monitored to create device behavioral fingerprints. During testing, an IoT spectrum sensor of the ElectroSense platform has been infected with ten recent malware samples (two botnets, three rootkits, three backdoors, one ransomware, and one cryptojacker) to measure the detection performance of the framework in two different network configurations. Both supervised and semi-supervised approaches provided promising results when detecting and classifying malicious behaviors from the eight previous malware and seven normal behaviors. In particular, the framework obtained 0.88–0.90 true positive rate when detecting the previous malicious behaviors as unseen or zero-day attacks and 0.94–0.96 F1-score when classifying them.

     

IoT-oriented high-efficient anti-malware hardware focusing on time series metadata extractable from inside a processor core

We aim to improve the efficiency of our previously proposed anti-malware hardware; it is a hardware-implemented malware detection mechanism that uses information inside the processor. We previously evaluated a prototype, but, due to its prototypical nature, there remain limitations, such as only detecting certain behaviors, high power consumption, and a tendency to bloat the training model. In this paper, we propose a circuit and a learning method to achieve high efficiency, low power consumption, and light weight for the model. In considering these three issues, we focus on time-series metadata obtained by transforming the processor information. To improve efficiency, we implement predictive detection to predict the behavior of metadata in the malware detection component. This lets the model detect malware within less than 19% of the number of execution cycles of the conventional method. To reduce power consumption, we implement a sampling circuit that interrupts the input to the detection circuit at regular intervals, reducing the system’s uptime by 99% while maintaining judgment accuracy. Finally, for a light weight, we focus on the training process of the metadata generator based on a machine-learning model. By applying sampling learning and feature dimensionality reduction in the training process, a metadata generator approximately 16% smaller than the previous version is created.

     

A new machine learning-based method for android malware detection on imbalanced dataset

Nowadays, malware applications are dangerous threats to Android devices, users, developers, and application stores. Researchers are trying to discover new methods for malware detection because the complexity of malwares, their continuous changes, and damages caused by their attacks have increased. One of the most important challenges in detecting malware is to have a balanced dataset. In this paper, a detection method is proposed to identify malware to improve accuracy and reduce error rates by preprocessing the used dataset and balancing it. To attain these purposes, the static analysis is used to extract features of the applications. The ranking methods of features are used to preprocess the feature set and the low-effective features are removed. The proposed method also balances the dataset by using the techniques of undersampling, the Synthetic Minority Oversampling Technique (SMOTE), and a combination of both methods, which have not yet been studied among detection methods. Then, the classifiers of K-Nearest Neighbor (KNN), Support Vector Machine, and Iterative Dichotomiser 3 are used to create the detection model. The performance of KNN with SMOTE is better than the performance of the other classifiers. The obtained results indicate that the criteria of precision, recall, accuracy, F-measure, and Matthews Correlation Coefficient are over 97%. The proposed method is effective in detecting 99.49% of the malware’s existing in the used dataset and new malware.

     

The poset of retracts of a free monoid

The set of retracts of a free monoid F with the partial order of inclusion is investigated. This poset is a lattice if and only if F is generated by three or fewer elements. For a finitely generated free monoid F it is shown non-constructively that, for every submonoid S of F, the intersection of all retracts of F containing S is regular. A regular expression can be constructed for this intersection when S is regular. The submonoid generated by the set of all retracts of F contained in the regular submonoid S is also regular and constructable. This allows the decision to be made whether or not any given pair of retracts has a supremum or an infimum in the poset of retracts of F. The procedure yields regular expressions for such suprema and infima when they exist.     

Structural analysis of binary executable headers for malware detection optimization

In the context of the OpenDAVFI project (a fork of the French initiative DAVFI for giving birth to a new generation, open antivirus engine which has been funded by the French Government), different AV filters have been developped and chained to detect both known and unknown malware very accurately while requiring a very limited number of updates. While most AV software use different static and dynamic detection techniques which are mostly based on the general concept of (static or heuristic) signature, we have observed that many malware do not comply to the Microsoft specifications with respect to the MZ-PE format. In this technical correspondence, we present structural analysis tests which have been implemented in the DAVFI/OpenDAVFi project. These tests accurately detect malware and therefore greatly reduce the number of malware that have to be analyzed by subsequent modules in our detection chain.     

Malware classification based on call graph clustering

Each day, anti-virus companies receive tens of thousands samples of potentially harmful executables. Many of the malicious samples are variations of previously encountered malware, created by their authors to evade pattern-based detection. Dealing with these large amounts of data requires robust, automatic detection approaches. This paper studies malware classification based on call graph clustering. By representing malware samples as call graphs, it is possible to abstract certain variations away, enabling the detection of structural similarities between samples. The ability to cluster similar samples together will make more generic detection techniques possible, thereby targeting the commonalities of the samples within a cluster. To compare call graphs mutually, we compute pairwise graph similarity scores via graph matchings which approximately minimize the graph edit distance. Next, to facilitate the discovery of similar malware samples, we employ several clustering algorithms, including k-medoids and Density-Based Spatial Clustering of Applications with Noise (DBSCAN). Clustering experiments are conducted on a collection of real malware samples, and the results are evaluated against manual classifications provided by human malware analysts. Experiments show that it is indeed possible to accurately detect malware families via call graph clustering. We anticipate that in the future, call graphs can be used to analyse the emergence of new malware families, and ultimately to automate implementation of generic detection schemes.     

Approximate matching of XML document with regular hedge grammar

A regular hedge grammar is a formal method to specify XML schema. An XML document can be viewed as an ordered labeled tree. Given an XML schema, finding the sequence of changes with minimum cost is not just of theoretical interest. This problem can be modeled, as given, as an ordered labeled tree (forest) F and as a regular hedge grammar P. We consider the minimum edit distance required to transform the forest F into F′ so that F′ is exactly matched by P. By introducing a leaf forest, we design an algorithm for solving this problem in time O(F ² P(F?+?log P)), where F is the size of the forest and P is the size of the grammar. To our knowledge, this is the first algorithm to transform an XML document (ordered labeled tree) to conform to the schema (tree grammar).     

Android malware detection through hybrid features fusion and ensemble classifiers: The AndroPyTool framework and the OmniDroid dataset

Cybersecurity has become a major concern for society, mainly motivated by the increasing number of cyber attacks and the wide range of targeted objectives. Due to the popularity of smartphones and tablets, Android devices are considered an entry point in many attack vectors. Malware applications are among the most used tactics and tools to perpetrate a cyber attack, so it is critical to study new ways of detecting them. In these detection mechanisms, machine learning has been used to build classifiers that are effective in discerning if an application is malware or benignware. However, training such classifiers require big amounts of labelled data which, in this context, consist of categorised malware and benignware Android applications represented by a set of features able to describe their behaviour. For that purpose, in this paper we present OmniDroid, a large and comprehensive dataset of features extracted from 22,000 real malware and goodware samples, aiming to help anti-malware tools creators and researchers when improving, or developing, new mechanisms and tools for Android malware detection. Furthermore, the characteristics of the dataset make it suitable to be used as a benchmark dataset to test classification and clustering algorithms or new representation techniques, among others. The dataset has been released under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License and was built using AndroPyTool, our automated framework for dynamic and static analysis of Android applications. Finally, we test a set of ensemble classifiers over this dataset and propose a malware detection approach based on the fusion of static and dynamic features through the combination of ensemble classifiers. The experimental results show the feasibility and potential usability (for the machine learning, soft computing and cyber security communities) of our automated framework and the publicly available dataset.     

MAPAS: a practical deep learning-based android malware detection system

A lot of malicious applications appears every day, threatening numerous users. Therefore, a surge of studies have been conducted to protect users from newly emerging malware by using machine learning algorithms. Albeit existing machine or deep learning-based Android malware detection approaches achieve high accuracy by using a combination of multiple features, it is not possible to employ them on our mobile devices due to the high cost for using them. In this paper, we propose MAPAS, a malware detection system, that achieves high accuracy and adaptable usages of computing resources. MAPAS analyzes behaviors of malicious applications based on API call graphs of them by using convolution neural networks (CNN). However, MAPAS does not use a classifier model generated by CNN, it only utilizes CNN for discovering common features of API call graphs of malware. For efficiently detecting malware, MAPAS employs a lightweight classifier that calculates a similarity between API call graphs used for malicious activities and API call graphs of applications that are going to be classified. To demonstrate the effectiveness and efficiency of MAPAS, we implement a prototype and thoroughly evaluate it. And, we compare MAPAS with a state-of-the-art Android malware detection approach, MaMaDroid. Our evaluation results demonstrate that MAPAS can classify applications 145.8% faster and uses memory around ten times lower than MaMaDroid. Also, MAPAS achieves higher accuracy (91.27%) than MaMaDroid (84.99%) for detecting unknown malware. In addition, MAPAS can generally detect any type of malware with high accuracy.

     

ELF-Miner: using structural knowledge and data mining methods to detect new (Linux) malicious executables

Linux malware can pose a significant threat—its (Linux) penetration is exponentially increasing—because little is known or understood about Linux OS vulnerabilities. We believe that now is the right time to devise non-signature based zero-day (previously unknown) malware detection strategies before Linux intruders take us by surprise. Therefore, in this paper, we first do a forensic analysis of Linux executable and linkable format (ELF) files. Our forensic analysis provides insight into different features that have the potential to discriminate malicious executables from benign ones. As a result, we can select a features’ set of 383 features that are extracted from an ELF headers. We quantify the classification potential of features using information gain and then remove redundant features by employing preprocessing filters. Finally, we do an extensive evaluation among classical rule-based machine learning classifiers—RIPPER, PART, C4.5 Rules, and decision tree J48—and bio-inspired classifiers—cAnt Miner, UCS, XCS, and GAssist—to select the best classifier for our system. We have evaluated our approach on an available collection of 709 Linux malware samples from vx heavens and offensive computing. Our experiments show that ELF-Miner provides more than 99% detection accuracy with less than 0.1% false alarm rate.     

Multifamily malware models

When training a machine learning model, there is likely to be a tradeoff between accuracy and the diversity of the dataset. Previous research has shown that if we train a model to detect one specific malware family, we generally obtain stronger results as compared to a case where we train a single model on multiple diverse families. However, during the detection phase, it would be more efficient to have a single model that can reliably detect multiple families, rather than having to score each sample against multiple models. In this research, we conduct experiments based on byte n-gram features to quantify the relationship between the generality of the training dataset and the accuracy of the corresponding machine learning models, all within the context of the malware detection problem. We find that neighborhood-based algorithms generalize surprisingly well, far outperforming the other machine learning techniques considered.

     

Shadow attacks: automatically evading system-call-behavior based malware detection

Contemporary malware makes extensive use of different techniques such as packing, code obfuscation, polymorphism, and metamorphism, to evade signature-based detection. Traditional signature-based detection technique is hard to catch up with latest malware or unknown malware. Behavior-based detection models are being investigated as a new methodology to defeat malware. This kind of approaches typically relies on system call sequences/graphs to model a malicious specification/pattern. In this paper, we present a new class of attacks, namely ??shadow attacks??, to evade current behavior-based malware detectors by partitioning one piece of malware into multiple ??shadow processes??. None of the shadow processes contains a recognizable malicious behavior specification known to single-process-based malware detectors, yet those shadow processes as an ensemble can still fulfill the original malicious functionality. To demonstrate the feasibility of this attack, we have developed a compiler-level prototype tool, AutoShadow, to automatically generate shadow-process version of malware given the source code of original malware. Our preliminary result has demonstrated the effectiveness of shadow attacks in evading several behavior-based malware analysis/detection solutions in real world. With the increasing adoption of multi-core computers and multi-process programs, malware writers may exploit more such shadow attacks in the future. We hope our preliminary study can foster more discussion and research to improve current generation of behavior-based malware detectors to address this great potential threat before it becomes a security problem of the epidemic proportions.