Towards adaptive character frequency-based exclusive signature matching scheme and its applications in distributed intrusion detection

Network intrusion detection systems (NIDSs), especially signature-based NIDSs, are being widely deployed in a distributed network environment with the purpose of defending against a variety of network attacks. However, signature matching is a key limiting factor to limit and lower the performance of a signature-based NIDS in a large-scale network environment, in which the cost is at least linear to the size of an input string. The overhead network packets can greatly reduce the effectiveness of such detection systems and heavily consume computer resources. To mitigate this issue, a more efficient signature matching algorithm is desirable. In this paper, we therefore develop an adaptive character frequency-based exclusive signature matching scheme (named ACF-EX) that can improve the process of signature matching for a signature-based NIDS. In the experiment, we implemented the ACF-EX scheme in a distributed network environment, evaluated it by comparing with the performance of Snort. In addition, we further apply this scheme to constructing a packet filter that can filter out network packets by conducting exclusive signature matching for a signature-based NIDS, which can avoid implementation issues and improve the flexibility of the scheme. The experimental results demonstrate that, in the distributed network environment, the proposed ACF-EX scheme can positively reduce the time consumption of signature matching and that our scheme is promising in constructing a packet filter to reduce the burden of a signature-based NIDS.     

A fast pattern matching algorithm with multi-byte search unit for high-speed network security

A signature-based intrusion detection system identifies intrusions by comparing the data traffic with known signature patterns. In this process, matching of packet strings against signature patterns is the most time-consuming step and dominates the overall system performance. Many signature-based network intrusion detection systems (NIDS), e.g., the Snort, employ one or multiple pattern matching algorithms to detect multiple attack types. So far, many pattern matching algorithms have been proposed. Most of them use single-byte standard unit for search, while a few algorithms such as the Modified Wu-Manber (MWM) algorithm use typically two-byte unit, which guarantees better performance than others even as the number of different signatures increases. Among those algorithms, the MWM algorithm has been known as the fastest pattern matching algorithm when the patterns in a rule set rarely appear in packets. However, the matching time of the MWM algorithm increases as the length of the shortest pattern in a signature group decreases.In this paper, by extending the length of the shortest pattern, we minimize the pattern matching time of the algorithm which uses multi-byte unit. We propose a new pattern matching algorithm called the L⁺¹-MWM algorithm for multi-pattern matching. The proposed algorithm minimizes the performance degradation that is originated from the dependency on the length of the shortest pattern. We show that the L⁺¹-MWM algorithm improves the performance of the MWM algorithm by as much as 20% in average under various lengths of shortest patterns and normal traffic conditions. Moreover, when the length of the shortest pattern in a rule set is less than 5, the L⁺¹-MWM algorithm shows 38.87% enhancement in average. We also conduct experiments on a real campus network and show that 12.48% enhancement is obtained in average. In addition, it is shown that the L⁺¹-MWM algorithm provides a better performance than the MWM algorithm by as much as 25% in average under various numbers of signatures and normal traffic conditions, and 20.12% enhancement in average with real on-line traffic.     

入侵检测中一种节约内存的多模式匹配算法

模式匹配既是网络入侵检测系统（NIDS）的关键,也是NIDS中消耗资源最多的部分。随着网络速度和入侵检测规则的持续增长,模式匹配正在成为NIDS的性能瓶颈。提出了一种基于非确定有限自动机结构的Aho-Corasick算法,通过压缩状态表,把状态和状态变迁存储在一个单一向量中,显著降低了内存需求,获得了良好的cache性能。测试表明,与其他Aho-Corasick 算法相比,MEAC的内存消耗平均减少了92.3%~98.4%,同时保持了Aho-Corasick算法的良好性能。     

基于有状态Bloom filter引擎的高速分组检测

越来越多的网络安全技术通过分析网络分组中的内容来检测报文中是否含有恶意攻击代码.为了能够在线检测攻击,部署在路由器中的分组检测模块对于分组检测的速度也提出了越来越高的要求.虽然在这个领域已有很多研究工作,然而在性能、可扩展性和适用性方面还有很多可研究的空间.提出了一种基于有状态Bloom filter引擎的高速分组检测方法State-Based Bloom filter engine(SABFE).通过并行查找Bloom filter和前缀寄存器堆,以及利用多个并行的Bloom filter引擎进行流并行检测,达到了较高的吞吐性能.同时,利用快速查找表和前缀寄存器堆保存当前子串的匹配状态来检测长的规则.分析和模拟实验表明:该方法在规则长度增加时依然保持了较高的吞吐性能,可以实现线速的分组检测,同时,极大地减少了硬件资源开销,提高了可扩展性.     

Improving the performance of passive network monitoring applications with memory locality enhancements

Passive network monitoring is the basis for a multitude of systems that support the robust, efficient, and secure operation of modern computer networks. Emerging network monitoring applications are more demanding in terms of memory and CPU resources due to the increasingly complex analysis operations that are performed on the inspected traffic. At the same time, as the traffic throughput in modern network links increases, the CPU time that can be devoted for processing each network packet decreases. This leads to a growing demand for more efficient passive network monitoring systems in which runtime performance becomes a critical issue.In this paper we present locality buffering, a novel approach for improving the runtime performance of a large class of CPU and memory intensive passive monitoring applications, such as intrusion detection systems, traffic characterization applications, and NetFlow export probes. Using locality buffering, captured packets are being reordered by clustering packets with the same port number before they are delivered to the monitoring application. This results in improved code and data locality, and consequently, in an overall increase in the packet processing throughput and decrease in the packet loss rate. We have implemented locality buffering within the widely used libpcap packet capturing library, which allows existing monitoring applications to transparently benefit from the reordered packet stream without modifications. Our experimental evaluation shows that locality buffering improves significantly the performance of popular applications, such as the Snort IDS, which exhibits a 21% increase in the packet processing throughput and is able to handle 67% higher traffic rates without dropping any packets.     

基于网络处理器的多模式串匹配研究

深度数据包检查是网络入侵检测系统的性能瓶颈。该文分析入侵规则集中模式串的分布特点,对多模式匹配算法FNP进行改进,研究在多核多线程体系结构的网络处理器上高效实现模式串匹配的方法。在Intel IXP2800网络处理器上的仿真实验结果表明,改进算法在规模为10 K的模式串集合上能达到6 Gb/s的吞吐量,具有几乎线性的加速比。     

多模式匹配算法在网络入侵自动检测中的应用

为了可以对计算机网络安全进行有效的保护,同时提高计算机网络入侵检测的功能及其效率,提出多模式匹配算法在网络入侵自动检测中的应用.首先快速检测引擎初始化,快速有效地区别规则集合;其次构造模式匹配链表,读取系统配置文件的规则;最后检测网络数据包,确保计算机网络安全.通过实验结果的对比,可以明显看出,相比于传统算法,多模式匹...     

Scheduling Protocols for Switches with Large Envelopes

Traditionally, switches make scheduling decisions on the granularity of a packet. However, this is becoming increasingly difficult since network bandwidth is growing rapidly whereas packet sizes remain largely unchanged. Therefore the service time of an individual packet is decreasing rapidly. In this paper we study switches that make scheduling decisions on the granularity of an envelope which can be much larger than a packet in size. For an output-queued switch with envelope size E, each output chooses one input every E time steps and transmits packets from this chosen input during the next E steps. For an input-queued switch with envelope size E, one matching from the inputs to the outputs is computed every E steps and only the input–output pairs that are defined by this matching are allowed to transmit packets during the next E steps. Traditional switches correspond to envelope size E = 1 and almost all previous scheduling work deals with this case exclusively. We first show how some stable protocols for scheduling networks of output-queued switches with E = 1 fail for arbitrary E when these protocols are generalized in the most straightforward manner. We then present an extremely simple protocol that does guarantee network stability for output-queued switches for any E ≥ 1. For input-queued switches we first present a max-weight matching protocol that is stable for a single switch with arbitrary E. We then present a more complex protocol that achieves stability for a network of input-queued switches for any E ≥ 1.     

基于深度包检测的防火墙系统设计

随着互联网的飞速发展,防火墙作为网络安全防护的重要手段已经成为了人们研究的重点。为了能够高效地过滤无关数据报文、抵御恶意攻击、保障网络的安全稳定运行,在研究深度包检测技术的基础上,提出了一种基于现场可编程门阵列、三态内容可寻址寄存器架构而实现的硬件防火墙系统。测试表明,该系统能够满足实际要求。     

一种基于深度报文检测的FSM状态表压缩技术

针对深度报文检测中正则表达式模式匹配的状态表爆炸问题,提出并实现了一种集合交割的预编码方法(SI-precode),在正则表达式转换成DFA前对所有输入符号进行预编码,通过压缩输入,减少FSM中输入符号的种类,从而压缩状态转移表的空间.证明了预编码生成的状态机的正确性及其与原状态机的同态性.采用L7-filter模式进行实验表明SI-precode不仅提高了正则表达式的编译速度,针对单模式状态机,其状态转移表空间比不进行预编码压缩了87%～97%,50个模式的多模式状态机可压缩59%.预编码在软硬件结合体系结构下进行协议识别时不会对性能造成影响;对纯软件结构性能降低2%～4%.     

A platform-based SoC design and implementation of scalable automaton matching for deep packet inspection

String matching plays a central role in packet inspection applications such as intrusion detection, anti-virus, anti-spam and Web filtering. Since they are computation and memory intensive, software matching algorithms are insufficient to meet the high-speed performance. Thus, offloading packet inspection to a dedicated hardware seems inevitable. This paper presents a scalable automaton matching (SAM) coprocessor that uses Aho-Corasick (AC) algorithm with two parallel acceleration techniques, root-indexing and pre-hashing. The root-indexing can match multiple bytes in one single matching, and the pre-hashing can be used to avoid bitmap AC matching which is a cycle-consuming operation. In the platform-based SoC implementation of the Xilinx ML310 FPGA, the proposed hardware architecture can achieve almost 10.7 Gbps and support over 10,000 patterns for virus, which is the largest pattern set from among the existing works. On the average, the performance of SAM is 7.65 times faster than the original bitmap AC. Furthermore, SAM is feasible for either internal or external memory architecture. The internal memory architecture provides high performance, while the external memory architecture provides high scalability in term of the number of patterns.     

Hierarchical management protocol for constructing a QoS communication path in wireless Ad Hoc networks

An Ad Hoc network consists of mobile hosts that can dynamically construct a wireless network without base stations. Due to the limited communication range, a source host usually needs other hosts to relay messages to the destination in a multi-hop manner. Consequently, establishing a routing path from the source to the destination is a basic requirement for providing communication service between any pair of mobile hosts. This study proposes a two-level management approach for efficiently constructing and maintaining a QoS routing path in Ad Hoc wireless networks, significantly reducing the quantity of control packets. In the first phase, the mobile hosts are partitioned into a number of complete graphs, each represented by a Supernode managed by an agent. The Ad Hoc network topology is thus transformed to an Agent-based Graph (AG). In the second phase, some agents of a larger degree than neighboring agents are selected as core nodes. The core nodes then virtually construct a Core Graph (CG). The proposed two-level hierarchical management and bandwidth-looking-ahead technologies can efficiently establish and maintain a QoS communication path at a low control packet cost. Simulation results indicate that the proposed management model significantly reduces the number of control packets in areas with very large numbers of mobile hosts.     

大规模多串匹配算法的访存行为分析

随着网络带宽的日益增长,病毒和非法信息的形式越来越多,网络安全系统处理的压力越来越大。多串匹配算法作为大部分网络安全系统中的核心扫描部分其性能尤为重要。从微处理器体系结构的角度,用模拟的方法分析了SBOM、AC、WM等三种精确多串匹配算法在大规模规则库的情况下,其性能影响的各种因素,特别是其访存行为特征,并从算法原理上解释了访存性为是如何被影响的。指出当规则库规模增到5 000时,由Cache失效引起的性能损失占全部开销的近10%,而且比重随着规则库规模增大而继续变大。     

A regular expression matching engine with hybrid memories

A key technique of network security inspection is by using the regular expression matching to locate the specific fingerprints of networking applications or attacks in the packet flows, and accordingly identify the underlying applications or attacks. However, due to the surge of various networking applications and attacks in recent years, even more fingerprints need to be investigated in this process, which leads to a high demand on a large memory space for regular expression matching. In addition, with the frequent upgrading of the network links nowadays, the network flow rate also increases dramatically. As a result, it demands the fast operation of regular expression matching accordingly with the enhanced throughput for network inspection. However, due to the limited space of the fast memory, the requirements on fast operations and large memory space are conflicting. On addressing this challenge, in this paper, we propose to use hybrid memory for regular expression matching. In specific, by investigating on the transition table state access probability through the Markov theory, it can be observed that there exist a number of states which are much more frequently accessed than others. Therefore, we devise a matching engine which is suitable for FPGA implementation with two-level memories, where the first-level memory uses the on-chip memory of FPGA to cache the frequently accessed state transitions, and the second-level memory, composed of slow and cheap DRAM, stores the whole state transitions. Furthermore, the L7-filter's regular expression patterns have been applied to obtain the state access probability, and different quantities of memory assignment approaches have also been investigated to evaluate the throughput.     

Hot Potato Worm Routing via Store-and-Forward Packet Routing

The theory of worm routing (rather than packet routing) has recently attracted increased attention as an abstraction of the underlying communication mechanisms in many parallel machines. Routing the worms in the hot potato style is a desired form of communication in high-speed optical interconnection networks. In this work, we develop a simple method for the design of parallel hot potato worm routing algorithms. Our basic approach is to simulate known packet routing algorithms, so that in each step worms are moved around instead of packets. By plugging in known results for packet routing, we get the fastest (so far) deterministic batch worm routing algorithms. Although the results are given for permutation routing on the mesh and the hypercube, the general method can be applied to many other networks and to more general communication patterns as well. Moreover, once better routing algorithms are found for the underlying network, the worm routing algorithms improve as well.     

运用差分演化算法实现多维包匹配的研究

互联网的发展已经使网速的瓶颈由链路速度转移到核心网络设备的包处理速度上,而包处理的核心工作是包匹配。传统方法难以做到包匹配速度适应核心网络设备数据包线速转发。提出了一种新的包匹配算法,该算法对差分演化算法进行了改进,并结合了改进算法和传统的包匹配算法。在适应值处理上运用统计学方法,从而增加了分析问题的客观性。数值实验表明,新算法与传统算法相比,在速度、存储空间以及更新时间等性能上得到了有效改善,另外新算法的包匹配的时间性能与规则数目只有很弱的相关性,从而适合处理多维和大规模问题。新算法把演化算法运用于多域大规模规则库的网络数据包的转发,并且数据包还能做到线速转发。新算法具有普适性,适用于防火墙、差别服务路由器等网络设备。     

Off-chip communication architectures for high throughput network processors

In this paper, we propose a new interconnection mechanism for network line cards. We project that the packet storage needs for the next-generation networks will be much higher. Such that the number of memory modules required to store the packets will be more than that can be directly connected to the network processor (NPU). In other words, the NPU I/O pins are limited and they do not scale well with the growing number of memory modules and processing elements employed on the network line cards. As a result, we propose to explore more suitable off-chip interconnect and communication mechanisms that will replace the existing systems and that will provide extraordinary high throughput. In particular, we investigate if the packet-switched k-ary n-cube networks can be a solution. To the best of our knowledge, this is the first time, the k-ary n-cube networks are used on a board. We investigate multiple k-ary n-cube based interconnects and include a variation of 2-ary 3-cube interconnect called the 3D-mesh. All of the k-ary n-cube interconnects include multiple, highly efficient techniques to route, switch, and control packet flows in order to minimize congestion spots and packet loss within the interconnects. We explore the tradeoffs between implementation constraints and performance. Performance results show that k-ary n-cube topologies significantly outperform the existing line card interconnects and they are able to sustain higher traffic loads. Furthermore, the 3D-mesh reaches the highest performance results of all interconnects and allows future scalability to adopt more memories and/or processors to increase the line card’s processing power.     

基于混合模式匹配算法的网络入侵检测

为了提升中央处理单元(CPU)和图形处理单元(GPU)协同检测网络入侵的性能,本文提出了一种具有数据包有效载荷长度约束的CPU/GPU混合模式匹配算法(LHPMA)。在分析CPU/GPU混合模式匹配算法(HPMA)的基础上,设计了长度约束分离算法(LBSA)对传入数据包进行提前分类。利用CPU中的预过滤缓冲区对较长数据包进行快速预过滤,结合全匹配缓冲区将较短数据包直接分配给GPU进行全模式匹配,通过减少有效载荷长度的多样性,提升了CPU/GPU协同检测网络入侵的性能。实验结果表明,LHPMA增强了HPMA的处理性能,充分发挥了GPU并行处理较短数据包的优势,并且LHPMA提高了网络入侵检测的吞吐量。     

A novel path protection scheme for MPLS networks using multi-path routing

Multi-protocol label switching (MPLS) is an evolving network technology that is used to provide traffic engineering (TE) and high speed networking. Internet service providers, which support MPLS technology, are increasingly demanded to provide high quality of service (QoS) guarantees. One of the aspects of QoS is fault tolerance. It is defined as the property of a system to continue operating in the event of failure of some of its parts. Fault tolerance techniques are very useful to maintain the survivability of the network by recovering from failure within acceptable delay and minimum packet loss while efficiently utilizing network resources.In this paper, we propose a novel approach for fault tolerance in MPLS networks. Our approach uses a modified (k, n) threshold sharing scheme with multi-path routing. An IP packet entering MPLS network is partitioned into n MPLS packets, which are assigned to node/link disjoint LSPs across the MPLS network. Receiving MPLS packets from k out of n LSPs are sufficient to reconstruct the original IP packet. The approach introduces no packet loss and no recovery delay while requiring reasonable redundant bandwidth. In addition, it can easily handle single and multiple path failures.     

General memory efficient packet matching FPGA architecture for future high-speed networks

Packet classification (matching) is one of the critical operations in networking widely used in many different devices and tasks ranging from switching or routing to a variety of monitoring and security applications like firewall or IDS. To satisfy the ever-growing performance demands of current and future high-speed networks, specially designed hardware accelerated architectures implementing packet classification are necessary. These demands are now growing to such an extent, that in order to keep up with the rising throughputs of network links, the FPGA accelerated architectures are required to perform matching of multiple packets in every single clock cycle. To meet this requirement a simple replication approach can be utilized – instantiate multiple copies of a processing pipeline matching incoming packets in parallel. However, simple replication of pipelines inseparably brings a significant increase in utilization of FPGA resources of all types, which is especially costly for rather scarce on-chip memories used in matching tables.We propose and examine a unique parallel hardware architecture for hash-based exact match classification of multiple packets in each clock cycle that offers a reduction of memory replication requirements. The core idea of the proposed architecture is to exploit the basic memory organization structure present in all modern FPGAs, where hundreds of individual block or distributed memory tiles are available and can be accessed (addressed) independently. This way, we are able to maintain a rather high throughput of matching multiple packets per clock cycle even without fully replicated memory resources in matching tables. Our results show that the designed approach can use on-chip memory resources very efficiently and even scales exceptionally well with increased capacities of match tables. For example, the proposed architecture is able to achieve a throughput of more than 2 Tbps (over 3 000 Mpps) with an effective capacity of more than 40 000 IPv4 flow records at the cost of only a few hundred block memory tiles (366 BlockRAM for Xilinx or 672 M20K for Intel FPGAs) utilizing only a small fraction of available logic resources (around 68 000 LUTs for Xilinx or 95 000 ALMs for Intel).