Hierarchical Kohonenen net for anomaly detection in network security.

A novel multilevel hierarchical Kohonen Net (K-Map) for an intrusion detection system is presented. Each level of the hierarchical map is modeled as a simple winner-take-all K-Map. One significant advantage of this multilevel hierarchical K-Map is its computational efficiency. Unlike other statistical anomaly detection methods such as nearest neighbor approach, K-means clustering or probabilistic analysis that employ distance computation in the feature space to identify the outliers, our approach does not involve costly point-to-point computation in organizing the data into clusters. Another advantage is the reduced network size. We use the classification capability of the K-Map on selected dimensions of data set in detecting anomalies. Randomly selected subsets that contain both attacks and normal records from the KDD Cup 1999 benchmark data are used to train the hierarchical net. We use a confidence measure to label the clusters. Then we use the test set from the same KDD Cup 1999 benchmark to test the hierarchical net. We show that a hierarchical K-Map in which each layer operates on a small subset of the feature space is superior to a single-layer K-Map operating on the whole feature space in detecting a variety of attacks in terms of detection rate as well as false positive rate.     

A self-organizing network for hyperellipsoidal clustering (HEC)

We propose a self-organizing network for hyperellipsoidal clustering (HEC). It consists of two layers. The first employs a number of principal component analysis subnetworks to estimate the hyperellipsoidal shapes of currently formed clusters. The second performs competitive learning using the cluster shape information from the first. The network performs partitional clustering using the proposed regularized Mahalanobis distance, which was designed to deal with the problems in estimating the Mahalanobis distance when the number of patterns in a cluster is less than or not considerably larger than the dimensionality of the feature space during clustering. This distance also achieves a tradeoff between hyperspherical and hyperellipsoidal cluster shapes so as to prevent the HEC network from producing unusually large or small clusters. The significance level of the Kolmogorov-Smirnov test on the distribution of the Mahalanobis distances of patterns in a cluster to the cluster center under the Gaussian cluster assumption is used as a compactness measure. The HEC network has been tested on a number of artificial data sets and real data sets, We also apply the HEC network to texture segmentation problems. Experiments show that the HEC network leads to a significant improvement in the clustering results over the K-means algorithm with Euclidean distance. Our results on real data sets also indicate that hyperellipsoidal shaped clusters are often encountered in practice.     

Human behavior clustering for anomaly detection

This paper aims to address the problem of modeling human behavior patterns captured in surveillance videos for the application of online normal behavior recognition and anomaly detection. A novel framework is developed for automatic behavior modeling and online anomaly detection without the need for manual labeling of the training data set. The framework consists of the following key components. 1) A compact and effective behavior representation method is developed based on spatial-temporal interest point detection. 2) The natural grouping of behavior patterns is determined through a novel clustering algorithm, topic hidden Markov model (THMM) built upon the existing hidden Markov model (HMM) and latent Dirichlet allocation (LDA), which overcomes the current limitations in accuracy, robustness, and computational efficiency. The new model is a four-level hierarchical Bayesian model, in which each video is modeled as a Markov chain of behavior patterns where each behavior pattern is a distribution over some segments of the video. Each of these segments in the video can be modeled as a mixture of actions where each action is a distribution over spatial-temporal words. 3) An online anomaly measure is introduced to detect abnormal behavior, whereas normal behavior is recognized by runtime accumulative visual evidence using the likelihood ratio test (LRT) method. Experimental results demonstrate the effectiveness and robustness of our approach using noisy and sparse data sets collected from a real surveillance scenario.     

An efficient hyperellipsoidal clustering algorithm for resource-constrained environments

Clustering has been widely used as a fundamental data mining tool for the automated analysis of complex datasets. There has been a growing need for the use of clustering algorithms in embedded systems with restricted computational capabilities, such as wireless sensor nodes, in order to support automated knowledge extraction from such systems. Although there has been considerable research on clustering algorithms, many of the proposed methods are computationally expensive. We propose a robust clustering algorithm with low computational complexity, suitable for computationally constrained environments. Our evaluation using both synthetic and real-life datasets demonstrates lower computational complexity and comparable accuracy of our approach compared to a range of existing methods.     

Self-adaptive and dynamic clustering for online anomaly detection

As recent Internet threats are evolving more rapidly than ever before, one of the major challenges in designing an intrusion detection system is to provide early and accurate detection of emerging threats. In this study, a novel framework is developed for fully unsupervised training and online anomaly detection. The framework is designed so that an initial model is constructed and then it gradually evolves according to the current state of online data without any human intervention. In the framework, a self-organizing map (SOM) that is seamlessly combined with K-means clustering is transformed into an adaptive and dynamic algorithm suitable for real-time processing. The performance of the proposed approach is evaluated through experiments using the well-known KDD Cup 1999 data set and further experiments using the honeypot data recently collected from Kyoto University. It is shown that the proposed approach can significantly increase the detection rate while the false alarm rate remains low. In particular, it is capable of detecting new types of attacks at the earliest possible time.     

Comments on "A self-organizing network for hyperellipsoidal clustering (HEC)" [and reply].

In the above paper by Mao-Jain (ibid., vol.7 (1996)), the Mahalanobis distance is used instead of Euclidean distance as the distance measure in order to acquire the hyperellipsoidal clustering. We prove that the clustering cost function is a constant under this condition, so hyperellipsoidal clustering cannot be realized. We also explains why the clustering algorithm developed in the above paper can get some good hyperellipsoidal clustering results. In reply, Mao-Jain state that the Wang-Xia failed to point out that their HEC clustering algorithm used a regularized Mahalanobis distance instead of the standard Mahalanobis distance. It is the regularized Mahalanobis distance which plays an important role in realizing hyperellipsoidal clusters. In conclusion, the comments made by Wang-Xia together with this response provide some new insights into the behavior of their HEC clustering algorithm. It further confirms that the HEC algorithm is a useful tool for understanding the structure of multidimensional data.     

Local anomaly detection for mobile network monitoring

Huge amounts of operation data are constantly collected from various parts of communication networks. These data include measurements from the radio connections and system logs from servers. System operators and developers need robust, easy to use decision support tools based on these data. One of their key applications is to detect anomalous phenomena of the network. In this paper we present an anomaly detection method that describes the normal states of the system with a self-organizing map (SOM) identified from the data. Large deviation in the data samples from the SOM nodes is detected as anomalous behavior. Large deviation has traditionally been detected using global thresholds. If variation of the data occurs in separate parts of the data space, the global thresholds either fail to reveal anomalies or reveal false anomalies. Instead of one global threshold, we can use local thresholds, which depend on the local variation of the data. We also present a method to find an adaptive threshold using the distribution of the deviations. Our anomaly detection method can be used both in exploration of history data or comparison of unforeseen data against a data model derived from history data. It is applicable to wide range of processes that produce multivariate data. In this paper we present examples of this method applied to server log data and radio interface data from mobile networks.     

一种基于聚类的异常检测方法

利用数据挖掘技术对网络中的海量数据进行分析从而发现入侵行为已成为目前异常检测研究的重点.为了进一步提高入侵行为检测的质量,提出了一种改进的异常检测算法.该方法首先将训练数据集转换为标准的单位特征度量空间,然后利用改进算法对数据进行划分,以找到聚类中心.最后对改进算法进行了性能分析与比较,实验结果表明:算法具有良好的稳定...     

网络入侵检测系统中的警报聚类

到目前为止,网络管理员对入侵检测系统(IDS)所产生的警报还是以在辅助工具下的手工操作进行整理,从而得到一个高级别的攻击描述。为了有效融合多种入侵检测系统报警信息,提高警告的准确性,警报聚类自动分析工具被建议使用来产生高级别的攻击描述。除此之外,警报聚类自动分析工具还可以有效地分析威胁,融合不同的信息源,例如来自于不同IDS中的信息源。该文提出了新的警报聚类系统,以便把来自于多种IDS所产生的警报进行警报聚类,产生攻击描述。实验结果表明,通过警报聚类模块有效地总结攻击可以产生高级别的警报,并大幅度地减少了要提交给管理员的警报数量。此外,以这些高级别警报为基础还可以进一步地进行威胁分析。     

Reservoir-based network traffic stream summarization for anomaly detection

Summarization is an important intermediate step for expediting knowledge discovery tasks such as anomaly detection. In the context of anomaly detection from data stream, the summary needs to represent both anomalous and normal data. But streaming data has distinct characteristics, such as one-pass constraint, for which conducting data mining operations are difficult. Existing stream summarization techniques are unable to create summary which represent both normal and anomalous instances. To address this problem, in this paper, a number of hybrid summarization techniques are designed and developed using the concept of reservoir for anomaly detection from network traffic. Experimental results on thirteen benchmark data streams show that the summaries produced from stream using pairwise distance (PSSR) and template matching (TMSSR) techniques can retain more anomalies than existing stream summarization techniques, and anomaly detection technique can identify the anomalies with high true positive and low false positive rate.     

Accuracy improving guidelines for network anomaly detection systems

An unprecedented growth in computer and communication systems in the last two decades has resulted in a proportional increase in the number and sophistication of network attacks. In particular, the number of previously-unseen attacks has increased exponentially in the last few years. Due to the rapidly evolving nature of network attacks, a considerable paradigm shift has taken place in the intrusion detection community. The main focus is now on Network Anomaly Detection Systems (NADSs) which model and flag deviations from normal/benign behavior of a network and can hence detect previously-unseen attacks. Contemporary NADS borrow concepts from a variety of theoretical fields (e.g., Information theory, stochastic and machine learning, signal processing, etc.) to model benign behavior. These NADSs, however, fall short of achieving acceptable performance levels as therefore widespread commercial deployments. Thus, in this paper, we firstly evaluate the performance of eight prominent network-based anomaly detectors under malicious portscan attacks to identify which NADSs perform better than others and why. These NADSs are evaluated on three criteria: accuracy (ROC curves), scalability (with respect to varying normal and attack traffic rates, and deployment points) and detection delay. These criteria are evaluated using two independently collected datasets with complementary strengths. We then propose novel methods and promising guidelines to improve the accuracy and scalability of existing and future anomaly detectors. Experimental analysis of the proposed guidelines is also presented for the proof of concept.     

一种基于动态聚类的异常入侵检测方法

运用数据挖掘方法进行入侵检测已经成为网络安全领域的一个重要研究方向。提出一种动态聚类的数据挖掘方法进行异常入侵检测,该方法将不同用户行为的特征动态聚集,根据各个子的类支持度与预设的检测阈值比较来区分正常与异常。由于动态聚类算法在每次聚类过程中都检验归类的合理性,因此获得很好的聚类效果。实时检测试验得到了较高的检测率和较低的误报率。     

An adaptive method for anomaly detection in symmetric network traffic

Symmetry is an obvious phenomenon in two-way communications. In this paper, we present an adaptive nonparametric method that can be used for anomaly detection in symmetric network traffic. Two important features are emphasized in this method: (i) automatic adjustment of the detection threshold according to the traffic conditions; and (ii) timely detection of the end of an anomalous event. Source-end defense against SYN flooding attacks is used to illustrate the efficacy of this method. Experiments on real traffic traces show that this method has high detection accuracy and low detection delays, and excels at detecting low intensity attacks.     

试论计算机网络安全的入侵检测技术

信息技术突飞猛进的发展,促使计算机走进千家万户,已然成为公众生活和生产活动不可或缺的工具,但与此同时计算机网络安全问题也备受关注,这无疑对计算机网络安全防范技术提出了更高的要求,尤其是入侵检测技术.对此,本文以计算机网络入侵途径为切入点,阐述了入侵检测技术的实践应用,并就其发展趋势作了展望.     

Ad Hoc网络自适应安全加权分簇算法

为解决Ad Hoc网络分簇过程中恶意节点被选为簇首带来的安全隐患,保障Ad Hoc网络的正确分簇和稳定运行,提出基于节点相关度、相对移动性、剩余能量值、安全评估度量值多方面因素的自适应安全加权分簇算法。安全评估度量参数由外部入侵检测系统和内部节点信任度共同计算得到,确保安全因素在分簇过程中的准确性;基于该算法给出相应分簇管理过程。仿真结果表明,该算法能够改善分簇性能,提高Ad Hoc网络的安全性。     

A hybrid deep network based approach for crowd anomaly detection

Multimedia Tools and Applications - In this paper, we present a hybrid deep network based approach for crowd anomaly detection in videos. For improved performance, the proposed approach exploits...     

自适应多趟聚类在检测无线传感器网络安全中的应用

在KSummary算法的基础上,引入层次和密度聚类方法,提出自适应多趟聚类方法。依次获得聚类个数k,聚类初始中心和最终聚类。将算法应用于无线传感器网络数据中,可以很好地发现数据中的离群点,从而找到传感器节点安全上存在的隐患。实验结果和分析表明：此算法不但可获得稳定、收敛的聚类结果,还能很好地发现离群点。     

基于网络处理器的多维统计异常检测系统

基于网络处理器开发的网络设备能够很好地解决灵活性和高性能之间的矛盾.基于网络处理器IXP2400自身的特点,设计了多维异常检测系统.该系统可以有效地检测和防御DDOS攻击.根据TCP/IP协议簇,对数据包进行多维解析,统计以及异常标记.仿真和硬件实验的验证数据表明,该系统能准确无误地按照设计目标一一分解数据包,并标记出异常值,从而为后续的网络安全的研究和防御工作提供可靠的数据保证.     

Combining sketches and wavelet analysis for multi time-scale network anomaly detection

With the rapid development and the increasing complexity of computer and communication systems and networks, traditional security technologies and measures can not meet the demand for integrated and dynamic security solutions. In this scenario, the use of Intrusion Detection Systems has emerged as a key element in network security.In this paper we address the problem proposing a wavelet-based technique able to detect network anomalies almost in real-time. In more detail, our approach is based on the combined use of sketches and wavelet analysis to reveal the anomalies in data collected at the router level. Moreover, to improve the detection rate we propose a multi time-scale analysis. The performance analysis, presented in this paper, demonstrates the effectiveness of the proposed method.