A defense framework against malware and vulnerability exploits

Current anti-malware tools have proved to be insufficient in combating ever-evolving malware attacks and vulnerability exploits due to inevitable vulnerabilities present in the complex software used today. In addition, the performance penalty incurred by anti-malware tools is magnified when security approaches designed for desktops are migrated to modern mobile devices, such as tablets and laptops, due to their relatively limited processing capabilities and battery capacities. In this paper, we propose a fine-grained anomaly detection defense framework that offers a cost-efficient way to detect malicious behavior and prevent vulnerability exploits in resource-constrained computing platforms. In this framework, a trusted third party (e.g., the publisher) first tests a new application by running it in a heavily monitored testing environment that emulates the target system and extracts a behavioral model from its execution paths. Extensive security policies are enforced during this process. In case of a violation, the program is denied release to the user. If the application passes the tests, the user can download the behavioral model along with the tested application binary. At run-time, the application is monitored against the behavioral model. In the unlikely event that a new execution path is encountered, conservative but lightweight security policies are applied. To reduce overhead at the user end, the behavioral model may be further reduced by the publisher through static analysis. We have implemented the defense framework using a netbook with the Intel Atom processor and evaluated it with a suite of 51 real-world Linux viruses and malware. Experiments demonstrate that our tool achieves a very high coverage (98 %) of considered malware and security threats. The four antivirus tools we compare our tool against were found to have poor virus coverage, especially of obfuscated viruses. By removing safe standard library blocks from the behavioral model, we reduce the model size by 8.4 \(\times \) and the user’s run-time overhead by 23 %.