基于大数据的网络安全与情报分析

随着IT技术和通信技术的发展，网络环境日趋复杂，云计算和虚拟化等技术的应用，使得主机边界、网络边界也变得动态和模糊。同时，网络攻击频繁，隐蔽性、持续性、趋利性等高级网络威胁增多。而传统网络安全与情报分析技术受数据来源单一、处理能力有限、部署依赖于物理环境等因素的限制，导致对威胁情报的获取、分析、利用能力不足，且对网络安全态势的感知与预测能力有限，不能有效解决当前和未来所面临的网络安全挑战。作者以大数据技术给网络安全与情报分析研究带来的挑战与机遇为线索，回顾大数据的内涵，分析当前网络安全与情报分析面临的困境，梳理大数据和网络安全与情报分析的关系，阐述大数据技术对传统安全分析方法的改变。大数据技术在安全领域应用形成大数据安全分析这一新型安全应对方法，通过紧扣安全数据自身的特点和安全分析的目标，应用大数据分析的方法和技术，解决网络安全与情报分析中的实际问题。一方面，批量数据处理技术、流式数据处理技术、交互式数据查询技术等大数据处理技术解决了高性能网络流量的实时还原与分析、海量历史日志数据分析与快速检索、海量文本数据的实时处理与检索等网络安全与情报分析中的数据处理问题；另一方面，大数据技术应用到安全可视分析、安全事件关联、用户行为分析中，形成大数据交互式可视分析、多源事件关联分析、用户实体行为分析、网络行为分析等一系列大数据安全分析研究分支，以应对当前的网络安全挑战。大数据安全分析技术在APT攻击检测、网络异常检测、网络安全态势感知、网络威胁情报分析等方面已经得到应用，但是，当前的网络安全形势仍不容乐观：高级网络威胁与攻击的有效检测方法缺乏;未知复杂网络攻击与威胁预测能力不足；缺乏度量网络安全态势评估结果的评价体系，关键资产与网络整体的态势评估指标体系不完善，网络安全态势感知评估方法缺少针对性；网络威胁情报信息分析的新型数据源数据获取难度大，缺乏威胁情报共享标准，尚未建成规模化、一体化的现代威胁情报中心和开放的威胁情报综合服务平台。围绕这些问题，需要研究高级网络威胁发现方法、复杂网络攻击预测方法、大规模网络安全态势感知技术、威胁情报数据采集与共享技术，并在高级网络威胁早期检测、隐蔽性和持续性网络通信行为检测、基于大数据分析的网络特征提取技术、综合威胁情报的高级网络威胁预测、非公开网络情报采集等关键技术上实现突破，以提升大数据对网络信息安全的支撑能力，增强网络信息安全风险感知、预警和处置能力。

Big Data Analytics for Network Security and Intelligence

With the development of IT and communication technology,the network environment is becoming more and more complicated, and the perimeters of host and network become dynamic and fuzzy due to the application of cloud computing and visualization technology.At the same time,network attacks become more frequent and advanced network threats with evasive and persistent behavior and profit-chasing are also increasing.However,due to the limit of data source and process ability and device deployment relied on physical environment,traditional network security and intelligence techniques are inefficient on the acquisition capability,analytical abilities and utilize capacity of threat intelligence,and the perception and prediction ability of the network security situation is limited,so it cannot solve the current and future network security challenges efficiently.The chances and challenges caused by big data for network security and intelligence analysis are took as this paper clue.first of all,the connotation of big data is reviewed,and the current dilemmas in network security and intelligence analysis are analyzed,and then the relationship between big data with network security and intelligence analysis is explored,and the changes of traditional security analysis brought by big data technologies are parsed.Big data security analysis,a new security method is formed after the big data technologies was applied in cyber security field.The value of big data security analysis embodies in solving practical problems in network security and intelligence analysis through the methods and technologies of big data analysis under sticking to the purpose of security analysis and the character of security data itself.On the one hand,big data processing technology,such as bulk data processing technology,streaming data processing technology,interactive data query technology,can solve the issues of data processing in the high-performance network traffic real-time restore and analysis,massive historical log data analysis and rapid retrieval,massive text data real-time processing.On the other hand,big data technologies are applied in security visual analysis,security event association and network user behavior analysis,a series research branch of big data security analysis are formed,such as big data interactive visual analysis,multi-source event correlation analysis,user entity behavior analysis,network behavior analysis and so on.Big data security analysis technologies have applied in APT attack detection,network anomaly detection,network security situation perception,network threat intelligence analysis,etc.Even though some achievements have been made in big data based network security and intelligence analysis,the current network security situation is still not optimistic.The effective detection method of advanced network threats and attacks is lacking.The detection and prediction result to unknown complex network attacks is undesirable.The measurement system for evaluation methods of network security situation awareness is needed,and large-scale network security situation awareness indicator system for key assets and entire network is incomplete yet,and the evaluation methods are no pertinence.It is difficult to acquire data from new type data sources of threat information and the standards of threat intelligence sharing are needed to be researched further.Large-scale and integrate threat intelligence center and open service platform aren''t yet build.Around the above problems,it needs to be studied that the methods of advanced network threat discovery,complex network attack prediction,large-scale network security situational awareness and threat information collection and sharing technology and needs some key technical breakthroughs such as early detection of advanced network threats,concealment of continuous network communication behavior detection,big data analytics based network feature extraction technology,intelligent-based advanced network threat forecast,non-public network intelligence collection,so as to improve the big data supporting capability for network information security and enhance network information security risk perception and disposal capacity.