基于生成对抗网络的系统日志级异常检测算法

针对大规模软件系统自动化异常检测任务中异常样本过少且异常反馈不及时的问题，提出一种基于生成对抗网络（GAN）与注意力机制的日志级异常检测算法。首先，通过日志模板将非结构化的日志转化为结构化的事件，每一个事件包含了日志的时间戳、签名与变量。其次，以滑动窗口的方式划分解析的事件序列，将产生的事件模式与下一时刻的事件组成真实的数据样本集。然后，将真实的事件模式作为训练样本输入来训练基于注意力机制的生成对抗网络，通过对抗学习的机制训练基于循环神经网络（RNN）的生成器直至收敛。最后，生成器通过输入的流式事件模式生成在新到来的事件模式下的正常与异常事件分布，并在系统管理员设置阈值的情况下，自动判断下一时刻的特定日志为正常事件或是异常事件。实验结果表明，提出的以门控循环单元网络为注意力权重并且用长短时记忆（LSTM）网络来解析事件模式的异常检测算法，比仅使用门控循环单元网络时的算法精准率提高了21.7%；此外，与日志级异常检测算法LogGAN相比，所提算法比LogGAN的异常检测精准率提升了7.8%。     

基于生成对抗网络的系统日志级异常检测算法

针对大规模软件系统自动化异常检测任务中异常样本过少且异常反馈不及时的问题,提出一种基于生成对抗网络（GAN）与注意力机制的日志级异常检测算法。首先,通过日志模板将非结构化的日志转化为结构化的事件,每一个事件包含了日志的时间戳、签名与变量。其次,以滑动窗口的方式划分解析的事件序列,将产生的事件模式与下一时刻的事件组成真实的数据样本集。然后,将真实的事件模式作为训练样本输入来训练基于注意力机制的生成对抗网络,通过对抗学习的机制训练基于循环神经网络（RNN）的生成器直至收敛。最后,生成器通过输入的流式事件模式生成在新到来的事件模式下的正常与异常事件分布,并在系统管理员设置阈值的情况下,自动判断下一时刻的特定日志为正常事件或是异常事件。实验结果表明,提出的以门控循环单元网络为注意力权重并且用长短时记忆（LSTM）网络来解析事件模式的异常检测算法,比仅使用门控循环单元网络时的算法精准率提高了21.7%;此外,与日志级异常检测算法LogGAN相比,所提算法比LogGAN的异常检测精准率提升了7.8%。     

A new intrusion detection system using support vector machines and hierarchical clustering

Whenever an intrusion occurs, the security and value of a computer system is compromised. Network-based attacks make it difficult for legitimate users to access various network services by purposely occupying or sabotaging network resources and services. This can be done by sending large amounts of network traffic, exploiting well-known faults in networking services, and by overloading network hosts. Intrusion Detection attempts to detect computer attacks by examining various data records observed in processes on the network and it is split into two groups, anomaly detection systems and misuse detection systems. Anomaly detection is an attempt to search for malicious behavior that deviates from established normal patterns. Misuse detection is used to identify intrusions that match known attack scenarios. Our interest here is in anomaly detection and our proposed method is a scalable solution for detecting network-based anomalies. We use Support Vector Machines (SVM) for classification. The SVM is one of the most successful classification algorithms in the data mining area, but its long training time limits its use. This paper presents a study for enhancing the training time of SVM, specifically when dealing with large data sets, using hierarchical clustering analysis. We use the Dynamically Growing Self-Organizing Tree (DGSOT) algorithm for clustering because it has proved to overcome the drawbacks of traditional hierarchical clustering algorithms (e.g., hierarchical agglomerative clustering). Clustering analysis helps find the boundary points, which are the most qualified data points to train SVM, between two classes. We present a new approach of combination of SVM and DGSOT, which starts with an initial training set and expands it gradually using the clustering structure produced by the DGSOT algorithm. We compare our approach with the Rocchio Bundling technique and random selection in terms of accuracy loss and training time gain using a single benchmark real data set. We show that our proposed variations contribute significantly in improving the training process of SVM with high generalization accuracy and outperform the Rocchio Bundling technique.     

面向不平衡数据集的煤矿监测系统异常数据识别方法

异常数据识别对于煤矿安全监测系统具有重要作用,但安全监测系统中异常数据一般只占数据总量的1%左右,不平衡性是此类数据的固有特点。目前多数机器学习算法在不平衡数据集上的分类预测准确率和灵敏度都相对较差。为了能准确识别异常数据,以煤矿分布式光纤竖井变形监测系统采集的数据为研究对象,提出了一种面向不平衡数据集、基于去重复下采样(RDU)、合成少数类过采样技术(SMOTE)和随机森林(RF)分类算法的煤矿监测系统异常数据识别方法。该方法利用RDU算法对多数类数据进行下采样,去除重复样本;利用SMOTE算法对少数类异常数据进行过采样,通过合成新的异常数据来改善数据集的不平衡性;并利用优化后的数据集训练RF分类算法,得到异常数据识别模型。在6个真实数据集上的对比实验结果表明,该方法的异常数据识别准确率平均值达到99.3%,具有较好的泛化性和较强的鲁棒性。     

Stochastic subset selection for learning with kernel machines

Kernel machines have gained much popularity in applications of machine learning. Support vector machines (SVMs) are a subset of kernel machines and generalize well for classification, regression, and anomaly detection tasks. The training procedure for traditional SVMs involves solving a quadratic programming (QP) problem. The QP problem scales super linearly in computational effort with the number of training samples and is often used for the offline batch processing of data. Kernel machines operate by retaining a subset of observed data during training. The data vectors contained within this subset are referred to as support vectors (SVs). The work presented in this paper introduces a subset selection method for the use of kernel machines in online, changing environments. Our algorithm works by using a stochastic indexing technique when selecting a subset of SVs when computing the kernel expansion. The work described here is novel because it separates the selection of kernel basis functions from the training algorithm used. The subset selection algorithm presented here can be used in conjunction with any online training technique. It is important for online kernel machines to be computationally efficient due to the real-time requirements of online environments. Our algorithm is an important contribution because it scales linearly with the number of training samples and is compatible with current training techniques. Our algorithm outperforms standard techniques in terms of computational efficiency and provides increased recognition accuracy in our experiments. We provide results from experiments using both simulated and real-world data sets to verify our algorithm.     

基于改进单类支持向量机的工业控制网络入侵检测方法

针对单类支持向量机（OCSVM）入侵检测方法无法检测内部异常点和离群点导致决策函数偏离训练样本的问题,提出了一种结合具有噪声的密度聚类（DBSCAN）方法和K-means方法的OCSVM异常入侵检测算法。首先通过DBSCAN算法,剔除训练数据中的离群点,消除离群点的影响;然后利用K-means划分数据类簇的方法筛选出内部异常点;最后利用OCSVM算法为每一个类簇建立单分类器用于检测异常数据。工控网络数据集上的实验结果表明,该组合分类器能够利用无异常数据样本检测出工控网络入侵,并且提高了OCSVM方法的检测效果。在气体管道网络数据集入侵检测实验中,所提方法的总体检测率为91.81%;而原始OCSVM算法则为80.77%。     

基于轻量化重构网络的表面缺陷视觉检测

基于深度学习的方法在某些工业产品的表面缺陷识别和分类方面表现出优异的性能, 然而大多数工业产品缺陷样本稀缺, 而且特征差异大, 导致这类需要大量缺陷样本训练的检测方法难以适用. 提出一种基于重构网络的无监督缺陷检测算法, 仅使用容易大量获得的无缺陷样本数据实现对异常缺陷的检测. 提出的算法包括两个阶段: 图像重构网络训练阶段和表面缺陷区域检测阶段. 训练阶段通过一种轻量化结构的全卷积自编码器设计重构网络, 仅使用少量正常样本进行训练, 使得重构网络能够生成无缺陷重构图像, 进一步提出一种结合结构性损失和L1损失的函数作为重构网络的损失函数, 解决自编码器检测算法对不规则纹理表面缺陷检测效果较差的问题; 缺陷检测阶段以重构图像与待测图像的残差作为缺陷的可能区域, 通过常规图像操作即可实现缺陷的定位. 对所提出的重构网络的无监督缺陷检测算法的网络结构、训练像素块大小、损失函数系数等影响因素进行了详细的实验分析, 并在多个缺陷图像样本集上与其他同类算法做了对比, 结果表明重构网络的无监督缺陷检测算法有较强的鲁棒性和准确性. 由于重构网络的无监督缺陷检测算法的轻量化结构, 检测1024 × 1024像素图像仅仅耗时2.82 ms, 适合工业在线检测.     

基于免疫优势多克隆网络的异常检测

为实现无监督异常检测,提出一种用于网络数据训练学习的免疫优势多克隆网络聚类算法。根据抗体抗原亲合度,通过免疫优势、克隆、交叉、非一致变异、禁忌克隆和克隆死亡等人工免疫系统算子,实现抗体网络的进化学习和自适应调节。以一个小规模的网络映射原始数据集的内在结构,利用基于凝聚的层次聚类方法对网络结构进行分析,从而获得描述正常和异常行为的数据特征。仿真结果表明,该算法适用于大规模、无标识数据的异常检测,并能检测出未知攻击。     

FRaC: a feature-modeling approach for semi-supervised and unsupervised anomaly detection

Anomaly detection involves identifying rare data instances (anomalies) that come from a different class or distribution than the majority (which are simply called “normal” instances). Given a training set of only normal data, the semi-supervised anomaly detection task is to identify anomalies in the future. Good solutions to this task have applications in fraud and intrusion detection. The unsupervised anomaly detection task is different: Given unlabeled, mostly-normal data, identify the anomalies among them. Many real-world machine learning tasks, including many fraud and intrusion detection tasks, are unsupervised because it is impractical (or impossible) to verify all of the training data. We recently presented FRaC, a new approach for semi-supervised anomaly detection. FRaC is based on using normal instances to build an ensemble of feature models, and then identifying instances that disagree with those models as anomalous. In this paper, we investigate the behavior of FRaC experimentally and explain why FRaC is so successful. We also show that FRaC is a superior approach for the unsupervised as well as the semi-supervised anomaly detection task, compared to well-known state-of-the-art anomaly detection methods, LOF and one-class support vector machines, and to an existing feature-modeling approach.     

基于支持向量数据描述的系统调用轨迹异常检测

通过监控主机系统调用的执行,可以在很大程度上发现入侵行为。建立支持向量数据描述(SVDD)的异常检测模型,通过对正常状态应用程序系统调用轨迹进行训练,使偏离正常模式的活动都被认为是潜在的入侵。通过对IE进程系统调用轨迹的优化处理,只利用少量的训练样本,试验获得了对异常样本99%的检测率,而虚警率不到1%。