Human perspective to anomaly detection for cybersecurity

Traditionally signature-based network Intrusion Detection Systems (IDS) rely on inputs from domain experts and can only identify the attacks if they occur as individual event. IDS generate large number of alerts and it becomes very difficult for human users to go through each message. Previous researches have proposed analytics based approaches to analyze IDS alert patterns based on anomaly detection models, multi-steps models or probabilistic approaches. However, due to the complexities of network intrusions, it is impossible to develop all possible attack patterns or to avoid false positives. With the advance in technologies and popularity of networks in our daily life, it is becoming more and more difficult to detect network intrusions. However, no matter how rapid the technologies change, the human behaviors behind the cyber attacks stay relatively constant. This provides us an opportunity to develop an improved system to detect the unusual cyber attacks. In this paper, we developed four network intrusion models based on consideration of human factors. We then tested these models on ITOC Cyber Defense Competition (CDX) 2009 data. Our results are encouraging. These Models are not only able to recognize most network attacks identified by SNORT log alerts, they are also able to distinguish the non-attack network traffic that was potentially missed by SNORT as indicated by ground truth validation of the data.     

入侵检测系统中报警验证模块的设计与实现

传统入侵检测系统虽然可以根据特征匹配的方法检测出攻击企图,却无法验证攻击企图是否成功,生成的报警不仅数量巨大而且误警率很高。该文提出一种结合漏洞扫描工具对入侵检测系统生成的报警进行验证的方法,根据被攻击主机是否包含能使攻击成功的漏洞来判定攻击能否成功,对攻击的目标主机不存在对应漏洞的报警降低优先级,从而提高报警质量。说明了报警验证模型各部分的设计和实现方法,系统运行结果显示该方法能有效地压缩报警量,降低误警率,帮助管理员从大量数据中找到最应该关注的真实报警。     

Risk balance defense approach against intrusions for network server

The paper presents a new defense approach based on risk balance to protect network servers from intrusion activities. We construct and implement a risk balance system, which consists of three modules, including a comprehensive alert processing module, an online risk assessment module, and a risk balance response decision-making module. The alert processing module improves the information quality of intrusion detection system (IDS) raw alerts by reducing false alerts rate, forming alert threads, and computing general parameters from the alert threads. The risk assessment module provides accurate evaluation of risks accordingly to alert threads. Based on the risk assessment, the response decision-making module is able to make right response decisions and perform very well in terms of noise immunization. Having advantages over conventional intrusion response systems, the risk balancer protects network servers not by directly blocking intrusion activities but by redirecting related network traffics and changing service platform. In this way, the system configurations that favor attackers are changed, and attacks are stopped with little impact on services to users. Therefore, the proposed risk balance approach is a good solution to not only the trade-off between the effectiveness and the negative effects of responses but also the false response problems caused by both IDS false-positive alerts and duplicated alerts.     

IDS alerts correlation using grammar-based approach

Intrusion Detection System (IDS) is a security technology that attempts to identify intrusions. Defending against multi-step intrusions which prepare for each other is a challenging task. In this paper, we propose a novel approach to alert post-processing and correlation, the Alerts Parser. Different from most other alert correlation methods, our approach treats the alerts as tokens and uses modified version of the LR parser to generate parse trees representing the scenarii in the alerts. An Attribute Context-Free Grammar (ACF-grammar) is used for representing the multi-step attacks. Attack scenarii information and prerequisites/consequences knowledge are included together in the ACF-grammar enhancing the correlation results. The modified LR parser depends on these ACF-grammars to generate parse trees. The experiments were performed on two different sets of network traffic traces, using different open-source and commercial IDS sensors. The discovered scenarii are represented by Correlation Graphs (CGs). The experimental results show that Alerts Parser can work in parallel, effectively correlate related alerts with low false correlation rate, uncover the attack strategies, and generate concise CGs.     

攻击案例综合学习系统研究

随着入侵检测系统在安全领域的广泛应用,入侵报警学习和分析已经成为一个研究热点。针对目前入侵报警泛滥和知识贫乏等问题,设计了一个完整的攻击案例学习系统框架。该学习系统分为两个阶段：入侵报警精简和典型攻击案例挖掘。前者利用改进的密度聚类方法实现相似报警聚合以及报警聚类的自动精简表示,后者利用序列模式挖掘方法挖掘频繁入侵事件序列。进一步提出一种基于入侵执行顺序约束关系的攻击案例评估算法实现典型攻击案例的自动筛选。最后,利用真实入侵报警数据测试了该攻击案例学习系统,结果表明该系统能够实现高效报警精简和典型攻击案例的准确学习。     

Decentralized multi-dimensional alert correlation for collaborative intrusion detection

The growth in coordinated network attacks such as scans, worms and distributed denial-of-service (DDoS) attacks is a profound threat to the security of the Internet. Collaborative intrusion detection systems (CIDSs) have the potential to detect these attacks, by enabling all the participating intrusion detection systems (IDSs) to share suspicious intelligence with each other to form a global view of the current security threats. Current correlation algorithms in CIDSs are either too simple to capture the important characteristics of attacks, or too computationally expensive to detect attacks in a timely manner. We propose a decentralized, multi-dimensional alert correlation algorithm for CIDSs to address these challenges. A multi-dimensional alert clustering algorithm is used to extract the significant intrusion patterns from raw intrusion alerts. A two-stage correlation algorithm is used, which first clusters alerts locally at each IDS, before reporting significant alert patterns to a global correlation stage. We introduce a probabilistic approach to decide when a pattern at the local stage is sufficiently significant to warrant correlation at the global stage. We then implement the proposed two-stage correlation algorithm in a fully distributed CIDS. Our experiments on a large real-world intrusion data set show that our approach can achieve a significant reduction in the number of alert messages generated by the local correlation stage with negligible false negatives compared to a centralized scheme. The proposed probabilistic threshold approach gains a significant improvement in detection accuracy in a stealthy attack scenario, compared to a naive scheme that uses the same threshold at the local and global stages. A large scale experiment on PlanetLab shows that our decentralized architecture is significantly more efficient than a centralized approach in terms of the time required to correlate alerts.     

入侵检测报警关联技术

报警关联技术分析不同安全产品产生的报警,从中识别出真正有意义的攻击警报,并减少大量的误报警,降低安全管理员的工作量。该文介绍了报警关联的基本模型和主要技术,分析了主要的关联方法,探讨了报警关联技术的发展方向。这些讨论对应用或发展报警关联技术都有参考价值。     

Network Security Alerts Management Architecture for Signature-Based Intrusions Detection Systems within a NAT Environment

Internet is providing essential communication between an infinite number of people and is being increasingly used as a tool for commerce. At the same time, security is becoming a tremendously important issue to deal with. Different network security solutions exist and contribute to enhanced security. From these solutions, Intrusion detection systems (IDS) have become one of the most common countermeasures for monitoring safety in computer systems and networks. The purpose of IDSs is distinguishing between intruders and normal users. However, IDSs report a massive number of isolated alerts. These isolated alerts represent low-level security-related events. Many of these isolated alerts are logically involved in a single multi-stage intrusion incident and a security officer often wants to analyze the complete incident instead of each individual simple alert. Another problem is that IDSs cannot work correctly with an environment managed with a NAT technique (Network Address Translation) since the host information (IP address and port number) are affected by the NAT devices. In order to address these limitations, the paper proposes a well-structured model to manage the massive number of isolated alerts and includes the NAT information in the IDS analysis. In fact, our solution permits to determine the real identities of entities implicated in security issues and abstracts the logical relation between alerts in order to support automatic correlation of those alerts involved in the same intrusion and to construct comprehensible attacks scenarios.     

A decision support system for constructing an alert classification model

As the rapid growth of network attacking tools, patterns of network intrusion events change gradually. Although many researches had been proposed to analyze network intrusion behaviors in accordance with low-level network data, they still suffer a large mount of false alerts and result in difficulties for network administrators to discover useful information from these alerts. To reduce the load of administrators, by collecting and analyzing unknown attack sequences systematically, administrators can do the duty of fixing the root causes. Due to the different characteristics of each intrusion, none of analysis methods can correlate IDS alerts precisely and discover all kinds of real intrusion patterns. Therefore, an alert-based decision support system is proposed in this paper to construct an alert classification model for on-line network behavior monitoring. The architecture of decision support system consists of three phases: Alert Preprocessing Phase, Model Constructing Phase and Rule Refining Phase. The Alert Processing Phase is used to transform IDS alerts into alert transactions with specific data format as alert subsequences, where an alert sequence is a kind of well-aggregated alert transaction format to discover intrusion behaviors. Besides, the Model Constructing Phase is used to construct three kinds of rule classes: normal rule classes, intrusion rule classes and suspicious rule classes, to filter false alert patterns and analyze each existing or unknown alert patterns; each rule class represents a set of classification rules. Normal rule class, a set of false alert classification rules, can be trained by using sequential pattern mining approach in an attack-free environment. Intrusion rule classes, a set of known intrusion classification rules, and suspicious rule classes, a set of novel intrusion classification rules, can be trained in a simulated attacking environment using several well-known rootkits and labeling by experts. Finally, the Rule Refining Phase is used to change the classification flags of alert sequence across different time intervals. According to the urgent situations of different levels, Network administrators can do event protecting or vulnerability repairing, even or cause tracing of attacks. Therefore, the decision support system can prevent attacks effectively, find novel attack patterns exactly and reduce the load of administrators efficiently.     

增量挖掘实时报警关联研究

入侵检测技术通过实时获取网络攻击报警信息,对网络安全实施检测、分析和动态防御,有效弥补了防火墙的不足。通过有效处理网络报警信息提高入侵检测的检测率、精确度是当前入侵检测技术研究的重要课题之一。提出了一种实时的增量挖掘入侵检测报警关联方法。该方法使报警事件的聚合操作和报警关联分析控制在小规模数据范围内进行,有效克服了一些数据挖掘算法应用到入侵检测过程中存在的多遍扫描、误报率高和报警信息关联度低问题。实验结果表明,该方法不但可以处理大容量实时网络报警信息,而且在报警信息关联分析和报警事件约减都体现了良好的性能。     

Reducing false positives in anomaly detectors through fuzzy alert aggregation

In this paper we focus on the aggregation of IDS alerts, an important component of the alert fusion process. We exploit fuzzy measures and fuzzy sets to design simple and robust alert aggregation algorithms. Exploiting fuzzy sets, we are able to robustly state whether or not two alerts are “close in time”, dealing with noisy and delayed detections. A performance metric for the evaluation of fusion systems is also proposed. Finally, we evaluate the fusion method with alert streams from anomaly-based IDS.     

基于可拓学的网络安全报警分析技术研究*

用户的网络管理需要建立一种新型的综合网络安全管理解决方案,即统一网络安全管理。特别关注于其中的一个关键功能——报警分析。其思路是以IDS报警为中心,将报警分析过程分解为包含报警评估与报警相关的两级关联分析模式。为了有效克服现今IDS报警分析技术中存在的问题和局限,顺应网络安全管理的统一化趋势,引入在解决矛盾问题方面极具优势的可拓学,以保证网络安全报警分析各种功能在技术上的实现。     

基于可拓学的网络安全报警分析技术研究*

用户的网络管理需要建立一种新型的综合网络安全管理解决方案,即统一网络安全管理。特别关注于其中的一个关键功能——报警分析。其思路是以IDS报警为中心,将报警分析过程分解为包含报警评估与报警相关的两级关联分析模式。为了有效克服现今IDS报警分析技术中存在的问题和局限,顺应网络安全管理的统一化趋势,引入在解决矛盾问题方面极具优势的可拓学,以保证网络安全报警分析各种功能在技术上的实现。     

报警信息关联模型的构建和实现

协同和分布式的网络攻击对传统的网络安全防护提出了巨大的挑战,同时也对分布式入侵检测技术提出了更高的要求,而有效融合多种入侵检测系统报警信息能够提高告警的准确性。首先给出了五维度报警信息关联的定义;然后设计与实现了带有实时响应机制的层次化关联模型,该模型具有较广泛的适用性,每一层都可以作为一个单独的模块完成相应的功能;最后给出了报警信息融合模块的实现。实验证明：报警信息融合可以降低误报、漏报率,并能识别攻击意图,达到预警的目的。     

Improving the quality of alerts and predicting intruder’s next goal with Hidden Colored Petri-Net

Intrusion detection systems (IDS) often provide poor quality alerts, which are insufficient to support rapid identification of ongoing attacks or predict an intruder’s next likely goal. In this paper, we propose a novel approach to alert postprocessing and correlation, the Hidden Colored Petri-Net (HCPN). Different from most other alert correlation methods, our approach treats the alert correlation problem as an inference problem rather than a filter problem. Our approach assumes that the intruder’s actions are unknown to the IDS and can be inferred only from the alerts generated by the IDS sensors. HCPN can describe the relationship between different steps carried out by intruders, model observations (alerts) and transitions (actions) separately, and associate each token element (system state) with a probability (or confidence). The model is an extension to Colored Petri-Net (CPN). It is so called “hidden” because the transitions (actions) are not directly observable but can be inferred by looking through the observations (alerts). These features make HCPN especially suitable for discovering intruders’ actions from their partial observations (alerts) and predicting intruders’ next goal. Our experiments on DARPA evaluation datasets and the attack scenarios from the Grand Challenge Problem (GCP) show that HCPN has promise as a way to reducing false positives and negatives, predicting intruder’s next possible action, uncovering intruders’ intrusion strategies after the attack scenario has happened, and providing confidence scores.     

分布式IDS的报警关联定义

网络规模越来越大。传统的IDS往往存在漏报误报率高、报警太低级的问题,因而不能及时准确反映整个系统的安全态势。网络管理人员不得不面对海量的原始报警信息,如大海捞针般地寻找可能的安全威胁和攻击来源。本文首先讨论了现有IDS的不足;之后给出了报警关联的定义。本文的研究成果已经在“网络安全监控与预警系统”（十五863项目）中得到应用,对分布式入侵检测系统的报警关联设计有重要的参考价值。     

一种基于信息熵的IDS告警预处理方法

针对目前入侵检测系统存在的海量重复告警、误报率偏高、告警质量低下等问题，提出一种基于信息熵的IDS告警预处理方法，用于减少误告警，聚合相似告警，生成代表单步攻击意图的超告警。首先，对IDS告警进行特征提取，用告警密度、告警周期值、源IP对应的目的IP数与攻击源威胁度这4个特征的信息熵融合结果表示一条告警所具有的特征信息量。通过与误告警的特征向量进行互雷尼信息熵的计算，从而识别出误告，并且去除误告。然后对误告去除后的告警按照IP对应关系，划分为2类：一种源IP对应一种目的IP的告警以及一种源IP对应多种目的IP的告警。分别对2类告警进行特征统计，构造5维特征信息熵向量，采用DBSCAN算法将信息量相同或者相似的告警进行聚类。最后对各个类别的告警进行动态时间窗口划分，并构建出代表单步攻击意图的超告警。实验结果表明，基于信息熵的告警预处理方法误告去除率为87.43%，告警聚合率达到98.63%，具有较好的误告去除效果以及较高的告警聚合率。     

Detecting and preventing replay attacks in industrial automation networks operated with profinet IO

Modern industrial facilities consist of controllers, actuators and sensors that are connected via traditional IT equipment. The ongoing integration of these systems into the communication network yields to new threats and attack possibilities. In industrial networks, often distinct communication protocols like Profinet IO (PNIO) are used. These protocols are often not supported by typical network security tools. In this work, we present two attack techniques that allow to take over the control of a PNIO device, enabling an attacker to replay previously recorded traffic. We model attack detection rules and propose an intrusion detection system (IDS) for industrial networks which is capable of detecting those replay attacks by correlating alerts from traditional IT IDS with specific PNIO alarms. As an additional effort, we introduce defense in depth mechanisms in order to prevent those attacks from taking effect in the physical world. Thereafter, we evaluate our IDS in a physical demonstrator and compare it with another IDS dedicated to securing PNIO networks. In a conceptual design, we show how network segmentation with flow control allows for preventing some, but not all of the attacks.     

Alert correlation in collaborative intelligent intrusion detection systems—A survey

As complete prevention of computer attacks is not possible, intrusion detection systems (IDSs) play a very important role in minimizing the damage caused by different computer attacks. There are two intrusion detection methods: namely misuse- and anomaly-based. A collaborative, intelligent intrusion detection system (CIIDS) is proposed to include both methods, since it is concluded from recent research that the performance of an individual detection engine is rarely satisfactory. In particular, two main challenges in current collaborative intrusion detection systems (CIDSs) research are highlighted and reviewed: CIDSs system architectures and alert correlation algorithms. Different CIDSs system, architectures are explained and compared. The use of CIDSs together with other multiple security systems raise certain issues and challenges in, alert correlation. Several different techniques for alert correlation are discussed. The focus will be on correlation of CIIDS alerts. Computational, Intelligence approaches, together with their applications on IDSs, are reviewed. Methods in soft computing collectively provide understandable, and autonomous solutions to IDS problems. At the end of the review, the paper suggests fuzzy logic, soft computing and other AI techniques, to be exploited to reduce the rate of false alarms while keeping the detection rate high. In conclusion, the paper highlights opportunities for an integrated solution to large-scale CIIDS.     

分布式入侵检测中的数据融合模型

针对传统的入侵检测系统存在的误警率高、存在告警洪流、告警孤立等缺点,引入了数据融合方法,提出了一个分布式入侵检测中的数据融合模型。该模型对告警进行分类,采用D-S理论对多IDS告警进行融合,基于前提和后果的方法对告警进行关联,最后量化系统受威胁程度,提供了一个解决上述问题的框架和方法。