一种Hadoop集群下的行为异常检测方法

随着分布式计算技术的发展,Hadoop成为大规模数据处理领域的典型代表,由于安全机制相对薄弱,缺少用户行为活动的监控,容易受到隐藏的安全威胁,如数据泄露等。结合主成分分析计算的特点,基于MapReduce对其做并行化处理,克服了传统主成分分析计算的缺点,提高了模型训练效率。提出了一种基于并行化主成分分析的异常行为检测方法,即比较当前用户的行为模式是否与历史行为模式相匹配作为判定用户行为异常与否的度量标准。实验表明该方法能够较好地发现用户的异常行为。

An abnormal behavior detection method in Hadoop cluster

With the development of distributed computing technology, Hadoop, as a typical representative in the field of massive data processing, is vulnerable to hidden security threats, such as data breaches, due to weak security mechanism and lack of user activity monitoring. By combining with the characteristics of the principal component analysis, we perform parallel process through MapReduce to overcome the disadvantage of principal component analysis and improve the training efficiency. We propose an abnormal behavior detection method in Hadoop cluster, namely we compare the current user behavior patterns with historical behavior patterns to see if they match, which is taken as a metric for anomaly behavior detection. Experimental results indicate that our method can detect users' anomaly behavior effectively.