| 基于大数据的网络日志分析技术 |
| |
| 引用本文： | 应毅,任凯,刘亚军.基于大数据的网络日志分析技术[J].计算机科学,2018,45(Z11):353-355. |
| |
| 作者姓名： | 应毅 任凯 刘亚军 |
| |
| 作者单位： | 三江学院计算机科学与工程学院 南京210012,南京大学金陵学院 南京210089,东南大学计算机科学与工程学院 南京210096 |
| |
| 基金项目： | 本文受江苏省高等学校自然科学研究面上项目(17KJB520033)资助 |
| |
| 摘 要： | 传统的日志分析技术在处理海量数据时存在计算瓶颈。针对该问题,研究了基于大数据技术的日志分析方案:由多台计算机完成日志文件的存储、分析、挖掘工作,建立了一个基于Hadoop开源框架的并行网络日志分析引擎,在MapReduce模型下重新实现了IP统计算法和异常检测算法。实验证明,在数据密集型计算中使用大数据技术可以明显提高算法的执行效率和增加系统的可扩展性。
|
| 关 键 词： | 大数据 Hadoop MapReduce 日志分析 异常检测 |
Network Log Analysis Technology Based on Big Data |
| |
| YING Yi,REN Kai and LIU Ya-jun.Network Log Analysis Technology Based on Big Data[J].Computer Science,2018,45(Z11):353-355. |
| |
| Authors: | YING Yi REN Kai and LIU Ya-jun |
| |
| Affiliation: | College of Computer Science and Technology,Sanjiang University,Nanjing 210012,China,Jinling College,Nanjing University,Nanjing 210089,China and School of Computer Science and Engineering,Southeast University,Nanjing 210096,China |
| |
| Abstract: | There exists a calculation bottleneck when traditional log analysis technology processes the massive data.To solve this problem,a log analysis solution based on big data technology was proposed in this paper.In this solution,the storage and analysis,mining tasks of Log files will be decomposed on multiple computers.The open source framework Hadoop is used to establish a parallel network log analysis engine.IP statistics and outlier detection algorithm was rea-lized with MapReduce model.Empirical studies show that the use of big data technology in data-intensive computing can significantly improve the execution efficiency of algorithms and the scalability of system. |
| |
| Keywords: | Big data Hadoop MapReduce Log analysis Outlier detection |
|
| 点击此处可从《计算机科学》浏览原始摘要信息 |
|
点击此处可从《计算机科学》下载全文 |
|
