Advanced probabilistic approach for network intrusion forecasting and detection

Recently, as damage caused by Internet threats has increased significantly, one of the major challenges is to accurately predict the period and severity of threats. In this study, a novel probabilistic approach is proposed effectively to forecast and detect network intrusions. It uses a Markov chain for probabilistic modeling of abnormal events in network systems. First, to define the network states, we perform K-means clustering, and then we introduce the concept of an outlier factor. Based on the defined states, the degree of abnormality of the incoming data is stochastically measured in real-time. The performance of the proposed approach is evaluated through experiments using the well-known DARPA 2000 data set and further analyzes. The proposed approach achieves high detection performance while representing the level of attacks in stages. In particular, our approach is shown to be very robust to training data sets and the number of states in the Markov model.     

On online high-dimensional spherical data clustering and feature selection

Motivated by the high demand to construct compact and accurate statistical models that are automatically adjustable to dynamic changes, in this paper, we propose an online probabilistic framework for high-dimensional spherical data modeling. The proposed framework allows simultaneous clustering and feature selection in online settings using finite mixtures of von Mises distributions (movM). The unsupervised learning of the resulting model is approached using Expectation Maximization (EM) for parameter estimation along with minimum message length (MML) to determine the optimal number of mixture components. The gradient stochastic descent approach is considered for incremental updating of model parameters, also. Through empirical experiments, we demonstrate the merits of the proposed learning framework on diverse high dimensional datasets and challenging applications.     

A Hybrid Deployable Dynamic Traffic Assignment Framework for Robust Online Route Guidance

Randomness in time-dependent origin-destination (O-D) demands and/or network supply conditions, and the computational tractability of potential solution methodologies are two major concerns for the online deployment of dynamic traffic assignment (DTA) under real-time traffic management systems. Most existing DTA models ignore these concerns and/or make unrealistic assumptions, precluding their online applicability. In this paper, a hybrid approach consisting of offline and online strategies is proposed to address the online stochastic dynamic traffic assignment problem. The basic idea is to address the computationally intensive components offline, while efficiently and effectively reacting to the unfolding conditions online. The offline component seeks a robust initial solution vis-à-vis randomness in O-D demands using historical O-D demand data. Termed the offline a priori solution, it is updated dynamically online based on unfolding O-D demands and incidents. The framework circumvents the need for accurate O-D demand and incident likelihood prediction models online, while exploiting historical O-D demand and incident data offline. Results of simulation experiments highlight the robustness of the hybrid approach with respect to online variations in O-D demand, its ability to address incident situations effectively, and its online efficiency.     

Online detection of bearing incipient fault with semi-supervised architecture and deep feature representation

Although researchers have made substantial progress in bearing fault detection and diagnosis recently, incipient fault detection, especially online detection, is still at an initial stage. Generally speaking, online detection of incipient faults is still subject to the following challenges: (1) improving discriminative ability of incipient fault features; (2) adaptive recognition of the distribution inconsistency that exists in online sequential data; (3) achieving automatic detections with avoiding manual adjustment of detection criterion; and (4) reducing false alarm rate. To address these challenges, this paper presents a new approach for bearing incipient fault online detection using semi-supervised architecture and deep feature representation. This approach is simple and effective. First, we extract deep features using stacked denoising auto-encoder from the target bearing's normal state data and an auxiliary bearing's fault state data. Second, we introduce safe semi-supervised support vector machine (S4VM), a kind of semi-supervised classifier, to identify the sequentially arrived data of the target bearing as normal or anomalous. To update the classifier effectively, we use the principal curve to generate synthetic fault data for keeping data classes balanced during online condition monitoring. Finally, we propose a new fault alarm criterion based on S4VM generalization error upper bound to adaptively recognize the occurrence of an incipient fault. The experimental results on three datasets (IEEE PHM Challenge 2012, IMS and XJTU-SY) demonstrate the effectiveness and high reliability of the proposed approach.     

Distributed anomaly detection for industrial wireless sensor networks based on fuzzy data modelling

Modern infrastructure increasingly depends on large computerized systems for their reliable operation. Supervisory Control and Data Acquisition (SCADA) systems are being deployed to monitor and control large scale distributed infrastructures (e.g. power plants, water distribution systems). A recent trend is to incorporate Wireless Sensor Networks (WSNs) to sense and gather data. However, due to the broadcast nature of the network and inherent limitations in the sensor nodes themselves, they are vulnerable to different types of security attacks. Given the critical aspects of the underlying infrastructure it is an extremely important research challenge to provide effective methods to detect malicious activities on these networks. This paper proposes a robust and scalable mechanism that aims to detect malicious anomalies accurately and efficiently using distributed in-network processing in a hierarchical framework. Unsupervised data partitioning is performed distributively adapting fuzzy c-means clustering in an incremental model. Non-parametric and non-probabilistic anomaly detection is performed through fuzzy membership evaluations and thresholds on observed inter-cluster distances. Robust thresholds are determined adaptively using second order statistical knowledge at each evaluation stage. Extensive experiments were performed and the results demonstrate that the proposed framework achieves high detection accuracy compared to existing data clustering approaches with more than 96% less communication overheads opposed to a centralized approach.     

On optimization of expertise matching with various constraints

This paper studies the problem of expertise matching with various constraints. Expertise matching, which aims to find the alignment between experts and queries, is a common problem in many applications such as conference paper-reviewer assignment, product-reviewer alignment, and product-endorser matching. Most existing methods formalize this problem as an information-retrieval problem and focus on finding a set of experts for each query independently. However, in real-world systems, various constraints are often needed to be considered. For example, in order to review a paper, it is desirable that there is at least one senior reviewer to guide the reviewing process. An important question is: “Can we design a framework to efficiently find the optimal solution for expertise matching under various constraints?” This paper explores such an approach by formulating the expertise matching problem in a constraint-based optimization framework. In the proposed framework, the problem of expertise matching is linked to a convex cost flow problem, which guarantees an optimal solution under various constraints. We also present an online matching algorithm to support incorporating user feedbacks in real time. The proposed approach has been evaluated on two different genres of expertise matching problems, namely conference paper-reviewer assignment and teacher-course assignment. Experimental results validate the effectiveness of the proposed approach. Based on the proposed method, we have also developed an online system for paper-reviewer suggestions, which has been used for paper-reviewer assignment in a top conference and feedbacks from the conference organizers are very positive.     

Adaptive anomaly detection with evolving connectionist systems

Anomaly detection holds great potential for detecting previously unknown attacks. In order to be effective in a practical environment, anomaly detection systems have to be capable of online learning and handling concept drift. In this paper, a new adaptive anomaly detection framework, based on the use of unsupervised evolving connectionist systems, is proposed to address these issues. It is designed to adapt to normal behavior changes while still recognizing anomalies. The evolving connectionist systems learn a subject's behavior in an online, adaptive fashion through efficient local element tuning. Experiments with the KDD Cup 1999 network data and the Windows NT user profiling data show that our adaptive anomaly detection systems, based on Fuzzy Adaptive Resonance Theory (ART) and Evolving Fuzzy Neural Networks (EFuNN), can significantly reduce the false alarm rate while the attack detection rate remains high.     

Anomaly Detection Based on Discrete Wavelet Transformation for Insider Threat Classification

Unlike external attacks, insider threats arise from legitimate users who belong to the organization. These individuals may be a potential threat for hostile behavior depending on their motives. For insider detection, many intrusion detection systems learn and prevent known scenarios, but because malicious behavior has similar patterns to normal behavior, in reality, these systems can be evaded. Furthermore, because insider threats share a feature space similar to normal behavior, identifying them by detecting anomalies has limitations. This study proposes an improved anomaly detection methodology for insider threats that occur in cybersecurity in which a discrete wavelet transformation technique is applied to classify normal vs. malicious users. The discrete wavelet transformation technique easily discovers new patterns or decomposes synthesized data, making it possible to distinguish between shared characteristics. To verify the efficacy of the proposed methodology, experiments were conducted in which normal users and malicious users were classified based on insider threat scenarios provided in Carnegie Mellon University’s Computer Emergency Response Team (CERT) dataset. The experimental results indicate that the proposed methodology with discrete wavelet transformation reduced the false-positive rate by 82% to 98% compared to the case with no wavelet applied. Thus, the proposed methodology has high potential for application to similar feature spaces.     

On-line generation association rules over data streams

In order to efficiently trace the changes of association rules over an online data stream, this paper proposes a method of generating association rules directly over the changing set of currently frequent itemsets. While all of the currently frequent itemsets in an online data stream are monitored by the estDec method, all the association rules of every frequent itemset in the prefix tree of the estDec method are generated by the proposed method in this paper. For this purpose, a traversal stack is introduced to efficiently enumerate all association rules in the prefix tree. This online implementation can avoid the drawbacks of the conventional two-step approach. In addition, the prefix tree itself can be utilized as an index structure for finding the current support of the antecedent of an association rule. Finally, the performance of the proposed method is analyzed by a series of experiments to identify its various characteristics.     

Online learning of contexts for detecting suspicious behaviors in surveillance videos

One of the main incentives for implementing video-based surveillance systems is the urban security. During the last years, several approaches for automatic detection of suspicious events have been proposed. Those methods usually require a training stage before starting their operation. This means that previous to run time a representative dataset of interest events, that may occur in the future, must be available. Nevertheless, most real surveillance systems lack of that information, so many of those proposals results impractical.In this paper, a context online learning scheme for detecting suspicious behaviors on surveillance videos is proposed. Contextual information, which is inferred from videos of people in a scenario, allows detecting suspicious behaviors before an eventual criminal's final attack occur. The main attribute of the proposed approach is the capacity to start up its operation with a reduced training dataset. By an incremental learning process, which uses new data obtained during the online operation, the proposed scheme improves the performance over time achieving a better adaptation to conditions of each scenario.The proposed scheme was validated on two datasets. The first of them includes threats against a parked truck and its driver. The second testing dataset is composed of night assault scenes recorded in an urban environment. The experimental results demonstrate that the proposed method is able to learn incrementally from a reduced initial dataset, achieving a performance similar to batch-type systems trained with all data simultaneously and outperforming five state-of-the-art algorithms over violence detection.     

一种基于深度迁移学习的滚动轴承早期故障在线检测方法

近年来,深度学习技术已在滚动轴承故障检测和诊断领域取得了成功应用,但面对不停机情况下的早期故障在线检测问题,仍存在着早期故障特征表示不充分、误报警率高等不足.为解决上述问题,本文从时序异常检测的角度出发,提出了一种基于深度迁移学习的早期故障在线检测方法.首先,提出一种面向多域迁移的深度自编码网络,通过构建具有改进的最大...     

A novel two-stage phased modeling framework for early fraud detection in online auctions

Reported dollar losses from online auction fraud were over $43M in 2008 in the US (NW3C, 2009). In general, reputation systems provided by online auction sites are the most common countermeasure available for buyers to evaluate a seller’s credit. Unfortunately, feedback score mechanisms are too easily manipulated, creating falsely overrated reputations. In addition, existing research on online auction fraud shows that a more complicated reputation management system could weaken the motivation of committing a fraud. However, very few of the previous work addresses the most important issue of a fraud detection mechanism is to discover a fraudster before he defrauds as early as possible. Therefore, developing an effective early fraud detection mechanism is necessary to prevent fraud for online auction participants.This paper proposes a novel two-stage phased modeling framework that integrates hybrid-phased models with a successive filtering procedure to identify latent fraudsters by examining the phased features of potential fraudsters’ lifecycles. This framework improves the performance of identifying latent fraudsters disguising as legitimate accounts with diverse features. In addition, a composite of measuring attributes we devised in this study is also helpful in modeling fraudulent behavior. To demonstrate the effectiveness of the proposed methods, real transaction data were collected from Yahoo!Taiwan (http://tw.bid.yahoo.com/) for training and testing. The experimental results show that the true positive rate of detecting fraudsters is over 93% on average. Furthermore, the proposed framework can significantly improve the precision and the success rate of fraud detection; the experimental results also show that the fraud detection models constructed by conventional methods are ineffective in detecting latent fraudsters.     

物联网安全威胁与安全模型

随着物联网应用的发展和普及利用,针对物联网的攻击事件日益增多且危害严重。目前面对物联网安全问题主要采用被动补救的方式,缺乏对物联网安全的体系化思考和研究。本论文首先介绍物联网系统架构和各实体的发展,然后分析物联网面临的多层次安全威胁,包括各实体自身的安全威胁,也包括跨域的安全威胁。其中,实体自身安全威胁涉及到云平台、设备端、管道、云端交互。物联网跨域安全威胁包含4个方面:多域级联攻击、物理域的冲突与叠加、信息域对物理域进行非预期的控制、信息域对物理域输入的理解不全面。在此基础上,论文研究了基于PDRR网络安全体系的物联网安全模型,包含安全防护、安全检测、响应、恢复4个维度。安全防护包含认证、授权与访问控制、通信加密等技术,需要考虑物联网种类繁多,规模巨大,异构等特点进行设计与实施。安全检测需要对各实体进行入侵检测、在线安全监测、脆弱性检测以及恶意代码检测。其中,在线安全监测获取系统内部设备、应用程序的行为、状态、是否存在已知脆弱性等。脆弱性检测偏向于对未知脆弱性进行深度挖掘。在响应阶段,除了配合相关部门机关完成安全行动资源配置、态势感知等响应工作外,还需要进行入侵事件的分析与响应,漏洞...     

Consistency anomalies in multi-tier architectures: automatic detection and prevention

Modern transaction systems, consisting of an application server tier and a database tier, offer several levels of isolation providing a trade-off between performance and consistency. While it is fairly well known how to identify qualitatively the anomalies that are possible under a certain isolation level, it is much more difficult to detect and quantify such anomalies during run-time of a given application. In this paper, we present a new approach to detect and quantify consistency anomalies for arbitrary multi-tier application running under any isolation levels ensuring at least read committed. In fact, the application can run even under a mixture of isolation levels. Our detection approach can be online or off-line and for each detected anomaly, we identify exactly the transactions and data items involved. Furthermore, we classify the detected anomalies into patterns showing the business methods involved as well as analyzing the types of cycles that occur. Our approach can help designers to either choose an isolation level where the anomalies do not occur or to change the transaction design to avoid the anomalies. Furthermore, we provide an option in which the occurrence of anomalies can be automatically reduced during run-time. To test the effectiveness and efficiency of our approach, we have conducted a set of experiments using a wide range of benchmarks.     

Bayesian changepoint and time-varying parameter learning in regime switching volatility models

This paper proposes a combined state and piecewise time-varying parameter learning technique in regime switching volatility models using multiple changepoint detection. This approach is a Sequential Monte Carlo method for estimating GARCH & EGARCH based volatility models with an unknown number of changepoints. Modern auxiliary particle filtering techniques are used to calculate the posterior densities and online forecasts. This approach also automatically deals with the common ancestral path dependence problem faced in these type volatility models. The model is tested on Borsa Istanbul (BIST) formerly known as Istanbul Stock Exchange (ISE) market data using daily log returns. A full structural changepoint specification is defined in which all parameters of the conditional variance of the volatility models are dynamic. Finally, it is shown with simulation experiments that the proposed approach partitions the series into several regimes and learns the parameters of each regime's volatility model in parallel with the multiple changepoint detection process.     

A Relax-and-Cut framework for large-scale maximum weight connected subgraph problems

Finding maximum weight connected subgraphs within networks is a fundamental combinatorial optimization problem both from theoretical and practical standpoints. One of the most prominent applications of this problem appears in Systems Biology and it corresponds to the detection of active subnetworks within gene interaction networks.Due to its importance, several modeling and algorithmic strategies have been proposed for tackling the maximum weight connected subgraph problem (MWCS) over the last years; the most effective strategies typically depend on the use of integer linear programming (ILP). Nonetheless, this implies that large-scale networks (such as those appearing in Systems Biology) can become burdensome; moreover, not all practitioners may have access to an ILP solver. In this paper, a unified modeling and algorithmic scheme is designed to solve the MWCS and some of its application-oriented variants with cardinality-constraints or budget-constraints. The proposed framework is based on a general node-based model which is tackled by a Relax-and-Cut scheme, i.e., Lagrangian relaxation combined with constraint generation; this yields a heuristic procedure capable of providing both dual and primal bounds. The approach is enhanced by additional valid inequalities, lifted valid inequalities, primal heuristics and variable-fixing procedures.Computational results on instances from the literature, as well as on additional large-scale instances, show that the proposed framework is competitive with respect to the existing approaches and it allows to find improved solutions for some unsolved instances from literature. The effect of initializing a Branch-and-Cut approach with information from the Relax-and-Cut is also investigated. The implemented approach is made available online.     

Adaptive learning for event modeling and characterization

Adaptive learning of specific patterns or events of interest has been an area of significant research for various applications in the last two decades. In developing diagnostic evaluation and safety monitoring applications of a propulsion system, it is critical to detect, characterize and model events of interest. It is a challenging task since the detection system should allow adaptive characterization of potential events of interest and correlate them to learn new models for future detection for online health monitoring and diagnostic evaluation. In this paper, a novel framework is established using a hierarchical adaptive clustering approach with fuzzy membership functions to characterize specific events of interest from the measured and processed features. Raw engine measurement data is first analyzed using the wavelet transform to provide features for localization of frequency information for use in the classification system. A method combining hierarchical and fuzzy k-means clustering is then applied to a set of selected measurements and computed features to determine the events of interest during engine operations. Experimental results have shown that the proposed approach is effective and computationally efficient to detect, characterize and model new events of interest from data collected through continuous operations.     

Multiframe Many–Many Point Correspondence for Vehicle Tracking in High Density Wide Area Aerial Videos

This paper presents a novel framework for tracking thousands of vehicles in high resolution, low frame rate, multiple camera aerial videos. The proposed algorithm avoids the pitfalls of global minimization of data association costs and instead maintains multiple object-centric associations for each track. Representation of object state in terms of many to many data associations per track is proposed and multiple novel constraints are introduced to make the association problem tractable while allowing sharing of detections among tracks. Weighted hypothetical measurements are introduced to better handle occlusions, mis-detections and split or merged detections. A two-frame differencing method is presented which performs simultaneous moving object detection in both. Two novel contextual constraints of vehicle following model, and discouragement of track intersection and merging are also proposed. Extensive experiments on challenging, ground truthed data sets are performed to show the feasibility and superiority of the proposed approach. Results of quantitative comparison with existing approaches are presented, and the efficacy of newly introduced constraints is experimentally established. The proposed algorithm performs better and faster than global, 1–1 data association methods.     

Novelty detection in wildlife scenes through semantic context modelling

Novelty detection is an important functionality that has found many applications in information retrieval and processing. In this paper we propose a novel framework that deals with novelty detection in multiple-scene image sets. Working with wildlife image data, the framework starts with image segmentation, followed by feature extraction and classification of the image blocks extracted from image segments. The labelled image blocks are then scanned through to generate a co-occurrence matrix of object labels, representing the semantic context within the scene. The semantic co-occurrence matrices then undergo binarization and principal component analysis for dimension reduction, forming the basis for constructing one-class models on scene categories. An algorithm for outliers detection that employs multiple one-class models is proposed. An advantage of our approach is that it can be used for novelty detection and scene classification at the same time. Our experiments show that the proposed approach algorithm gives favourable performance for the task of detecting novel wildlife scenes, and binarization of the semantic co-occurrence matrices helps increase the robustness to variations of scene statistics.