基于Spark Streaming的僵尸主机检测算法

随着宽带互联网的广泛应用,产生了同时针对互联网服务提供商和用户的新型威胁,僵尸网络。僵尸网络通过多类传播和感染程序,构建一个可一对多控制的网络,操控大量僵尸主机发起DDoS攻击、发送垃圾邮件、偷窃敏感数据和钓鱼等恶意行为。本文的目标是基于一种分布式实时处理框架,提出一种分布式的僵尸主机检测算法,该算法能够充分利用网络流量的统计数据IPFIX,在无需深度包解析的情况下,能够识别僵尸主机行为。同时,本文使用该算法实现了IPFIXScanner原型系统,系统的鲁棒性和可扩展性是设计该系统的核心原则。实验表明,IPFIXScanner原型在使用指定僵尸家族样本训练的情况下,对于特定类的僵尸主机能够获得较高的检出率和较低的误报率。在核心交换机上的测试结果表明,IPFIXScanner能够进行分布式的实时检测,加速比接近线性,验证了Spark Streaming引擎在分布式流处理方面的优势,以及用于僵尸主机检测方面的可行性。     

基于数值分析的异常扫描行为监测系统

提出了一种基于数值分析的异常扫描行为监测方法, 以Netflow网管数据为基础, 设计开发了监测系统, 实现了对网络中主流网络蠕虫病毒、IRC僵尸木马的传播爆发以及黑客恶意扫描探测等异常行为的实时监测, 取得良好效果, 大幅提升了网络运营单位的网络安全支撑服务能力。     

基于神经网络的僵尸网络检测

目前主流的僵尸网络检测方法主要利用网络流量分析技术,这往往需要数据包的内部信息,或者依赖于外部系统提供的信息或僵尸主机的恶意行为,并且大多数方法不能自动存储僵尸网络的流量特征,不具有联想记忆功能.为此提出了一种基于BP神经网络的僵尸网络检测方法,通过大量的僵尸网络和正常流量样本训练BP神经网络分类器,使其学会辨认僵尸网络的流量,自动记忆僵尸流量特征,从而有效检测出被感染的主机.该神经网络分类器以主机对为分析对象,提取2个主机间通信的流量特征,将主机对的特征向量作为输入,有效地区分出正常主机和僵尸主机.实验表明,该方法的检测率达到99%,误报率在1%以下,具有良好的性能.     

基于snort僵尸网络检测系统的设计与实现

为了准确检测僵尸网络等恶意流量,提出以snort为核心模块,通过对网络流量的抓取、分析,以及后端数据库和前端页面的相关设计,实现了一个入侵检测系统。该系统可以实时的监测网络流量从而快速检测出网络攻击行为,及时地发出警告信息,该系统具有良好的扩展性和可移植性。     

基于主机行为异常的P2P僵尸网络在线检测方法

僵尸网络已经成为当前最为严重的网络威胁之一,其中P2P僵尸网络得到迅速发展,其自身的通信特征给检测带来巨大的挑战.针对P2P僵尸网络检测技术的研究已经引起研究人员的广泛关注.提出一种P2P僵尸网络在线检测方法,首先采用信息熵技术发现网络流量中的异常点,然后通过分析P2P僵尸网络中主机的行为异常,利用统计学中的假设检验技术,从正常的网络流量数据中识别出可疑P2P僵尸主机,同时根据僵尸主机通信模式的相似性进行最终确认.实验结果表明该方法能够有效实现P2P僵尸网络的在线检测.     

分布式流数据加载和查询技术优化

分布式流查询是一种基于数据流的实时查询计算方法,近年来得到了广泛的关注和快速发展。综述了分布式流处理框架在实时关系型查询上取得的研究成果;对涉及分布式数据加载、分布式流计算框架、分布式流查询的产品进行了分析和比较;提出了基于Spark Streaming和Apache Kafka构建的分布式流查询模型,以并发加载多个文件源的形式,设计内存文件系统实现数据的快速加载,相较于基于Apache Flume的加载技术提速1倍以上。在Spark Streaming的基础上,实现了基于Spark SQL的分布式流查询接口,并提出了自行编码解析SQL语句的方法,实现了分布式查询。测试结果表明,在查询语句复杂的情况下,自行编码解析SQL的查询效率具有明显的优势。     

增强的Netflow数据采集系统设计与实现

基于端口号进行网络流量分析的Netflow技术无法准确地回答网络流分别由哪些应用协议组成及其比例等问题,也不能提供web用户行为数据来反映用户对大多数网站的访问情况。为改进上述缺陷,提出一种增强的Netflow数据采集系统。结合flow和深度包检测技术进行精确的应用协议识别,对web用户行为数据进行采集与动态存储,为网络流量监测与分析提供全面的数据支撑,设计并实现该系统,验证了其有效性和可行性。     

一种基于深度强化学习的Spark Streaming参数优化方法

Spark Streaming作为主流的开源分布式流分析框架,性能优化是目前的研究热点之一。在Spark Streaming性能优化中,业务场景下的配置参数优化是其性能提升的重要因素。在Spark Streaming系统中,可配置的参数有200多个,对参数调优人员的经验要求较高,未经优化的参数配置会影响流作业执行性能。因此,针对Spark Streaming的参数配置优化问题,提出一种基于深度强化学习的Spark Streaming参数优化方法（DQN-SSPO）,将Spark Streaming参数优化配置问题转化为深度强化学习模型训练中的最大回报获得问题,并提出权重状态空间转移方法来增加模型训练获得高反馈奖励的概率。在3种典型的流分析任务上进行实验,结果表明经参数优化后Spark Streaming上的流作业性能在总调度时间上平均缩减27.93%,在总处理时间上平均缩减42%。     

使用Spark Streaming的自适应实时DDoS检测和防御技术

分布式拒绝服务(distributed denial of service,DDoS)攻击是重要的安全威胁,网络速度的不断提高给传统的检测方法带来了新的挑战。以Spark等为代表的大数据处理技术,给网络安全的高速检测带来了新的契机。提出了一种基于Spark Streaming框架的自适应实时DDoS检测防御技术,通过对滑动窗口内源簇进行分组,并根据与各分组内源簇比例的偏差统计,检测出DDoS攻击流量。通过感知合法的网络流量,实现了对DDoS攻击的自适应快速检测和有效响应。实验结果表明,该技术可极大地提升检测能力,为保障网络服务性能和安全检测的可扩展性提供了一种可行的解决方案。     

一种新颖的P2P僵尸网络检测技术

针对当前僵尸网络向P2P方向发展的趋势,在对P2P僵尸网络本质的理解和把握的基础上,提出了一种新颖的P2P僵尸网络检测技术。对于某个被监视的网络,关注其内部每台主机的通信行为和网络恶意活动。把这些通信行为和网络恶意活动分类,找出具有相似或相关通信和网络恶意行为的主机。根据我们对定义的理解,这些主机就属于某个P2P僵尸网络。     

基于流量摘要的僵尸网络检测

随着僵尸网络的日益进化,检测和防范僵尸网络攻击成为网络安全研究的重要任务.现有的研究很少考虑到僵尸网络中的时序模式,并且在实时僵尸网络检测中效果不佳,也无法检测未知的僵尸网络.针对这些问题,本文提出了基于流量摘要的僵尸网络检测方法,首先将原始流数据按照源主机地址聚合,划分适当的时间窗口生成流量摘要记录,然后构建决策树、随机森林和XGBoost机器学习分类模型.在CTU-13数据集上的实验结果表明,本文提出的方法能够有效检测僵尸流量,并且能够检测未知僵尸网络,此外,借助Spark技术也能满足现实应用中快速检测的需要.     

一种新颖的P2P僵尸网络检测技术

针对当前僵尸网络向P2P方向发展的趋势,在对P2P僵尸网络本质的理解和把握的基础上,提出了一种新颖的P2P僵尸网络检测技术。对于某个被监视的网络,关注其内部每台主机的通信行为和网络恶意活动。把这些通信行为和网络恶意活动分类,找出具有相似或相关通信和网络恶意行为的主机。根据我们对定义的理解,这些主机就属于某个P2P僵尸网络。     

中小型局域网中P2P僵尸网络的检测

传统的僵尸网络大多是基于IRC协议的集中式结构,但越来越多的僵尸网络开始转向了分布式的P2P结构,针对IRC信道的检测方法已经不适用于新型的P2P僵尸网络。提出一种面向中小型局域网,根据流量统计特性和恶意攻击活动相结合的P2P僵尸网络检测方法。这种方法对采用随机端口,数据加密等新型手段的Botnets可以进行有效检测。     

Identifying botnets by capturing group activities in DNS traffic

Botnets have become the main vehicle to conduct online crimes such as DDoS, spam, phishing and identity theft. Even though numerous efforts have been directed towards detection of botnets, evolving evasion techniques easily thwart detection. Moreover, existing approaches can be overwhelmed by the large amount of data needed to be analyzed. In this paper, we propose a light-weight mechanism to detect botnets using their fundamental characteristics, i.e., group activity. The proposed mechanism, referred to as BotGAD (botnet group activity detector) needs a small amount of data from DNS traffic to detect botnet, not all network traffic content or known signatures. BotGAD can detect botnets from a large-scale network in real-time even though the botnet performs encrypted communications. Moreover, BotGAD can detect botnets that adopt recent evasion techniques. We evaluate BotGAD using multiple DNS traces collected from different sources including a campus network and large ISP networks. The evaluation shows that BotGAD can automatically detect botnets while providing real-time monitoring in large scale networks.     

Bloccess: Enabling Fine-Grained Access Control Based on Blockchain

Botnets pose significant threats to cybersecurity. The infected Internet of Things (IoT) devices are used to launch unsupported malicious activities on target entities to disrupt their operations and services. To address this danger, we propose a machine learning-based method, for detecting botnets by analyzing network traffic data flow including various types of botnet attacks. Our method uses a hybrid model where a Variational AutoEncoder (VAE) is trained in an unsupervised manner to learn latent representations that describe the benign traffic data, and one-class classifier (OCC) for detecting anomaly (also called novelty detection). The main aim of this research is to learn the discriminating representations of the normal data in low dimensional latent space generated by VAE, and thus improve the predictive power of the OCC to detect malicious traffic. We have evaluated the performance of our model, and compared it against baseline models using a real network based dataset, containing popular IoT devices, and presenting a wide variety of attacks from two recent botnet families Mirai and Bashlite. Tests showed that our model can detect botnets with a satisfactory performance.

     

Detecting P2P botnets by discovering flow dependency in C&C traffic

Botnets are widely used by attackers and they have evolved from centralized structures to distributed structures. Most of the modern P2P bots launch attacks in a stealthy way and the detection approaches based on the malicious traffic of bots are inefficient. In this paper, an approach that aims to detect Peer-to-Peer (P2P) botnets is proposed. Unlike previous works, the approach is independent of any malicious traffic generated by bots and does not require bots’ information provided by external systems. It detects P2P bots by focusing on the instinct characteristics of their Command and Control (C&C) communications, which are identified by discovering flow dependencies in C&C traffic. After discovering the flow dependencies, our approach distinguishes P2P bots and normal hosts by clustering technique. Experimental results on real-world network traces merged with synthetic P2P botnet traces indicate that 1) flow dependency can be used to detect P2P botnets, and 2) the proposed approach can detect P2P botnets with a high detection rate and a low false positive rate.     

面向规避僵尸网络流量检测的对抗样本生成

基于机器学习的僵尸网络流量检测是现阶段网络安全领域比较热门的研究方向,然而生成对抗网络(generative adversarial networks,GAN)的出现使得机器学习面临巨大的挑战.针对这个问题,在未知僵尸网络流量检测器模型结构和参数的假设条件下,基于生成对抗网络提出了一种新的用于黑盒攻击的对抗样本生成方法...     

A novel method of mining network flow to detect P2P botnets

Botnets are a serious threat to cyber-security. As a consequence, botnet detection has become an important research topic in network protection and cyber-crime prevention. P2P botnets are one of the most malicious zombie networks, as their architecture imitates P2P software. Characteristics of P2P botnets include (1) the use of multiple controllers to avoid single-point failure; (2) the use of encryption to evade misuse detection technologies; and (3) the capacity to evade anomaly detection, usually by initiating numerous sessions without consuming substantial bandwidth. To overcome these difficulties, we propose a novel data mining method. First, we identify the differences between P2P botnet behavior and normal network behavior. Then, we use these differences to tune the data-mining parameters to cluster and distinguish normal Internet behavior from that lurking P2P botnets. This method can identify a P2P botnet without breaking the encryption. Furthermore, the detection system can be deployed without altering the existing network architecture, and it can detect the existence of botnets in a complex traffic mix before they attack. The experimental results reveal that the method is effective in recognizing the existence of botnets. Accordingly, the results of this study will be of value to information security academics and practitioners.