视频监控存储系统的设计与实现

针对文件系统存储视频数据时存储效率低和检索性能差的问题,提出一种基于裸盘设备的视频数据专用存储方案,实现了视频数据的高效存储和秒级检索。针对多路视频流的并发写入控制,采用单线程非阻塞I/O复用技术和"乒乓缓冲"机制,避免多路视频流并发写入时的阻塞问题;针对视频监控系统媒体转发服务器的时钟漂移问题,采用基于两级索引的回馈定位技术,提高系统对服务器钟差的容错能力。系统测试结果表明,相对于文件存储,"时间索引+数据块"存储方式的视频检索效率不受数据规模大小影响,多路并发写入时,单线程非阻塞I/O复用和"乒乓缓冲"机制能有效避免阻塞现象的发生。     

面向海量高清视频数据的高性能分布式存储系统

通过对视频监控数据的特点和传统存储方案进行分析,提出一种高性能分布式存储系统解决方案.不同于传统的基于文件存储的方式,设计了一种逻辑卷结构,将非结构化的视频流数据以此结构进行组织并直接写入RAW磁盘设备,解决了传统存储方案中随机磁盘读写和磁盘碎片导致存储性能下降的问题.该方案将元数据组织为两级索引结构,分别由状态管理器和存储服务器管理,极大地减少了状态管理器需要管理元数据的数量,消除了性能瓶颈,并提供精确到秒级的检索精度.此外,该方案灵活的存储服务器分组策略和组内互备关系使得存储系统具备容错能力和线性扩展能力.系统测试结果表明,该方案在成本低廉的PC服务器上实现了单台服务器能同时记录400路1080P视频流,写入速度是本地文件系统的2.5倍.     

一种网络视频监控系统的专用存储方案

针对网络视频监控系统中传统文件系统存储方案存储效率低和检索性能差的缺陷,提出一种基于裸磁盘设备的录像存储方案。根据监控系统的数据存储特点,采用B＋树管理录像段索引信息,设计一种磁盘逻辑存储结构,并给出一种基于图像组的数据缓存机制。系统测试结果表明,与传统的文件系统存储方案相比,该方案在监控系统的512Kb／s和1Mb／s典型存储码率下,录像存储效率分别提高了43．6％和30．3％,录像检索耗时降至35ms以下。     

异构环境中HDFS数据块调度算法的设计与实现

针对Hadoop分布式文件系统(HDFS)的写性能在执行效率上的不足,提出一种在节点性能异构环境中对HDFS数据块进行并发传输的调度算法。该算法实时监控HDFS集群中每个节点的资源状态和内存缓存队列,动态地将接收节点与转发节点进行配对传输,使全系统节点的网卡和磁盘并发工作,缩短了所有副本写入分布式文件系统的时间。将数据写入磁盘后请求下一个数据块,保证数据安全性,同时也使得各个节点获得与自身性能相匹配的副本数,使性能异构的系统能达到较高的写入速度。性能测试结果表明,使用该算法的分布式文件系统的写入性能较原始的HDFS提高了1倍。     

基于负载均衡的多源流数据实时存储系统

物联网感知流数据多以时序数据为主,具有数据量大、连续到达、多来源等特点。现有的基于HBase的交通流数据存储系统在数据写入并发量大时,仍然存在存储效率低与系统可用性不高的问题。针对该问题,设计并实现了基于负载均衡的多源流数据实时存储系统。该系统将数据代理扩展为集群架构,提出了一种基于负载均衡的任务调度算法,实现了任务与数据代理之间的按序匹配,使数据代理集群负载均衡地处理任务,实现数据并行存储到HBase数据库中。实验对比结果表明:该系统使各数据代理的数据分配比例维持在0.3~0.4,同时以约1.5倍于单数据代理的速度将数据写入HBase数据库。     

分布式大规模监控视频存储系统THNVR

通过分析网络视频监控应用中的存储需求特点,提出了一种采用低成本SATA硬盘的高性能存储解决方案。该方案使用定长文件,并将结构化与非结构化数据分别存储和检索,彻底克服了传统解决方案中磁盘碎片导致存储性能下降的问题。该方案还采用了自适应的缓冲技术,对磁盘访问进行了有效的规整和调度,充分利用了磁盘的写入带宽。此外,该方案还实现了容错、扩展等重要功能。性能分析显示,该方案在普通计算机和SATA磁盘上能够同时记录256路D1或512路CIF视频数据流。     

大容量数据采集存储系统设计与实现

为解决现有数字示波器和数据存储记录仪对复杂系统覆盖性和可靠性测试不适用问题,设计了基于PCIe总线架构的大容量数据采集存储系统,保证多路高速测量时完整保留测试数据便于后续分析和回顾。给出了基于主控单元片上系统的硬件框架和PCIe总线存储架构设计,多种数据存储方式满足不同数据量的需求。设计了系统通信协议,HPS与FPGA之间的控制指令和交互方式。对数据存储流程中HPS获取采样数据过程,HPS将采样数据写入磁盘过程,以及数据写入磁盘的数据格式进行了设计。测试结果表明,系统硬件、软件设计方案可行,运行稳定,在相同采样速率和分辨率下能存储记录数据量远优于现有数字示波器和数据存储记录仪,目前已投入多款产品的研制和生产过程,取得良好测试效果。     

基于组合学的大规模磁盘队列轮换休息机制的编码设计

在大规模存储的系统中,运用组合学方法实现了磁盘队列轮换休息机制的编码算法,该机制能极大的提高磁盘的平均无故障时间和延长使用寿命.具体通过圆排列映射的数学建模方法,连续两次应用抽屉(鸽巢)原理,设计了从N个磁盘中随机取出M个磁盘(M＜N),让其中一个或数个磁盘断电休眠,用剩余磁盘还原休眠磁盘的组合学编码算法.     

多路视频监控系统的存储调度策略

采用传统的“轮巡式”视频监控系统在多路存储情况下,存在着存储的效率低下、因硬盘空间不足改变存储路径而造成的存储“抖动”等问题。该文介绍了旨在提高多路存储效率的多磁盘存储调度算法和解决“抖动”问题的存储预分配算法及其设计思路和实现方法。实际应用表明,多磁盘存储调度算法能够很好地利用多硬盘大容量的特点,根据各个硬盘的容量和被访问的次数,将多路存储“平均”分配于多个硬盘中,充分提高存储访问的效率,并通过资源预约方式、预分配算法很好地解决了存储“抖动”问题。     

基于热度的Hadoop快速副本复制算法

在云存储中心, 由于节点失效带来的文件数据块副本丢失不仅会影响系统的可靠性, 还会影响文件的并发访问效率. 针对Hadoop中默认的副本复制方法存在的问题, 即副本复制过程某些节点数据传输过于集中, 负载不均衡, 磁盘I/O吞吐率低, 提出一种基于热度的快速副本复制算法. 该算法优先复制热度高的数据块, 合理选择数据块复制的源节点和目的节点. 仿真结果表明, 该算法平衡了系统的工作负载, 提高了磁盘I/O吞吐率, 显著降低用户请求平均响应时间.     

视频服务器中变比特率视频流的I/O调度

讨论了视频服务器中变化特率（ＶＢＲ）视频流的Ｉ／Ｏ调度问题．其中重点讨论了Ｉ／Ｏ调度中的接纳控制算法．首先提出了Ｂｉ－ＳＣＡＮ磁盘调度算法,并将其应用到ＶＢＲ视频流的Ｉ／Ｏ调度．接着讨论了两种确定性接纳控制算法．确定性接纳控制能提供确保的服务质量,但系统资源的利用率很低．为此提出了一种基于统计复用的统计接纳控制算法,该算法将ＶＢＲ视频数据复杂的统计特性转化为磁盘提交的数据块数量的统计特性,使问题得     

Storage System for Supporting More Video Streams in Video Server

Video server needs a storage system with large bandwidth in order to provide concurrently more users with the real time retrieval requests for video streams. So, the storage system generally has the structure of disk array, which consists of multiple disks. When the storage system serves multiple video stream requests, it's bottlenecks come from the seeking delay caused by the random movement of disk head and from unbalanced disk access due to disk load unbalance among multiple disks.This paper presents a novel placement and retrieval policy. The new policy retrieves the requested data through sequential movement of disk heads and maintaining disk load balance so that it can diminish the bottlenecks on retrieving and can provide the concurrent real time retrieval services for more users simultaneously. In addition, the novel policy reduces the startup latency for the requests. The correctness of the novel placement and retrieval policy is analyzed with theoretical views. Performance analysis of the novel placement and retrieval policy is provided with simulations.     

Inter-Object Layer Clustering for scalable video streaming

In this work, we develop an efficient storage technique to support real-time streaming of layer encoded video in a single hard disk. The size of a single hard disk drive will soon be able to hold multi-tera bytes and is going to handle relatively larger number of files. We expect that disk layout in a single disk will be rather critical issue in determining the efficiency of the storage system. We propose a novel storage technique, Inter-Object Layer Clustering for layer encoded video objects. In Inter-Object Layer Clustering, storage is partitioned into two regions: lower layer partition and upper layer partition. Lower and upper layer partition harbor the lower layer and upper layer data blocks across all video objects and cluster them together. We develop an elaborate performance model for this placement scheme. We examine the performance of the proposed technique using analytical formulation as well as a physical experiment. We found that clustering the layers across all objects brings 100% increase in the number of concurrent sessions compared to the case where file is stored in temporal order when the clients’ access bandwidth is narrow. Inter-Object Layer Clustering shows 15% performance improvement compared to the clustering of layers within the objects.     

视频服务器中视频流存储分配算法的研究

飞速发展的网络和存储技术使为大量的用同时提供视频点播功能的ＶＯＤ系统成为可能，目前已有一批这样的系统问题，但这些系统为了保证一定数目的并发点播流，都预留了大量的设备资源。     

连续媒体服务器存储系统中的容错算法

连续媒体服务器(如VOD服务器)要对大量连续媒体数据(如声频、视频)进行管理,按一定速率为用户提供连续的媒体服务。因此,在这样的系统中,作为存储设备的磁盘阵列要具有高可靠性和一定的容错能力。文章提出一种基于奇偶检验的数据重构恢复算法,以保证系统中只有一个盘出现故障时,能使服务器及时重构出故障盘上的数据,并且算法充分利用了媒体流内在特性———回放时数据的连续性,与目前使用的标准故障恢复算法相比,大大减少了磁盘在线故障后数据重构过程的系统开销。最后通过分析、比较证明了算法的有效性。     

基于存储熵的存储负载均衡算法

在分布式存储系统中,一般都是以磁盘空间利用率（DU）来判断各存储节点的负载均衡程度,当所有节点的磁盘空间利用率相等时,是整个分布式存储系统的存储负载均衡点。但是在实际的应用场景中,磁盘I/O速率比较低的存储节点和可靠性比较低的存储节点往往成为影响整个存储系统数据读写性能的瓶颈,因此在异构分布式存储系统中,特别是各存储节点磁盘I/O速率和可靠性差异较大的分布式存储系统中,如果仅仅以磁盘空间利用率作为存储负载均衡的判定条件,则其数据的读写效率必然受到限制。从读写效率的角度提出一种度量分布式存储系统中存储负载均衡的新思路。根据负载均衡理论和熵理论给出存储熵（SE）的定义,并提出一种基于存储熵的负载均衡算法,该算法通过系统负载判定、单节点负载判定和负载迁移实现了对分布式存储系统存储负载的量化调整,并通过实验与基于磁盘空间利用率的负载均衡算法进行了对比分析,验证了该算法对分布式存储系统中存储负载具有良好的均衡性,有效地控制了系统负载失衡的问题,提高了分布式存储系统的整体读写效率。     

Design and evaluation of a generic software architecture foron-demand video servers

Introduces the design, implementation and evaluation of a generic software architecture for on-demand video servers. We describe different key components for controlling the storage and network devices within the server. The interactive collaborations between these software components are also illustrated. The experimental results indicate a very promising direction in exploring the right combinations of these software components. The server is thus able to increase the number of concurrent video accesses with the same hardware configuration. For instance, with the right combinations, the system achieved about 80% of the storage system bandwidth of four disks, about 70% of the storage system bandwidth of six disks, and generally reached the maximal achieved SCSI bandwidth when eight disks are used over two SCSI buses. Our research and experimental results are based on video servers currently under construction across a variety of hardware platforms, including SMP, DMP and clusters of PCs or workstations. The most advanced prototype server is based on an SGI shared-memory multiprocessor with a mass storage system consisting of RAID-3 disk arrays. With all the enabling/management schemes, we were able to further investigate interesting research issues by considering the user's access profiles for taking advantage of popular video titles. The results were significant, with a range of 60% improvement given a 512 kByte block size. In addition to the experimental results, theoretical performance models were also developed that closely match to our collected experimental results     

视频服务器存储子系统的I/O优化

视频服务器存储子系统的I／O性能决定了视频服务器的总体性能。本文提出的一种新的实时磁盘调度算法（LLF－Window)能有效地服务各种编码格式的视频流，并对传统的SCSI总线不均衡调度机制进行了改造，实验结果表明，新的磁盘调度算法和改造后的SCSI总线调度机制有效地改进了视频服务器存储子系统的I／O性能，保证了视频流的连续播放。     

A buffer-inventory-based dynamic scheduling algorithm for multimedia-on-demand servers

We present a producer-consumer model of multimedia-on-demand (MOD) servers. The producer retrieves media data from a disk and places it into a set of buffers, while the consumer sends out the data in the buffers to the users. We develop for the producer a buffer-inventory-based dynamic scheduling (BIDS) algorithm that guarantees non-zero inventory and non-overflow of data in the buffers to meet the continuity requirement and no-loss of data for each media stream. The algorithm can deal with heterogeneous me dia streams as well as the transient circumstances upon service completions and arrivals of new requests. To smooth out the impact of bursty data of variable-bit-rate media streams and therefore increase the maximum admissible load of requests, we also introduce into the scheduling scheme a time-scale-dependent peak consumption rate and a virtual cycle time. Based on BIDS, an effective admission control mechanism can be easily established by checking two simple conditions respectively on the overall system load and buffer size. Our algorithm is very easy to implement. Experiments carried out with an actual disk system and real video stream data verify that it is more robust compared to static scheduling algorithms previously proposed in the literature, especially when handling variable-bit-rate media streams.