基于 Hadoop 的分布式视频处理

中国科学院计算机网络信息中心与青海湖保护区管理局合作,共同建设了青海湖野外网络视频监控。如何高效地处理每天产出的超过 100GB 的视频数据成为了一个难题。现在的视频处理系统采用计算和存储相分离的架构,这需要配置较高的专门服务器进行支撑,本文基于廉价扩展性能较好的 Hadoop 平台对视频处理做出了分布式的实现,并对单个视频文件做出了分布式转码的实现。同时,本文将基于 Hadoop 的分布式视频处理的实现和基于 HTCondor 的分布式视频的批处理实现进行了对比,实验证明,在不损失视频处理效率的条件下,基于 Hadoop 的分布式视频处理的实现拥有分布式文件系统支撑、完善的任务监控等优势。

Distributed Video Processing Based on Hadoop

The Computer Network Information Center of the Chinese Academy of Sciences, in cooperation with the Qinghai Lake Nature Reserve Administration, has built a Qinghai Lake outdoor video surveillance network. How to efficiently handle the daily output of more than 100GB of video data has become an issue. The present video processing system adopts the architecture of computing and storage separation, which needs the support of highly specialized servers. This paper proposes a distributed implementation of video processing based on the Hadoop platform with better scalability and low cost. In this paper, the implementation of the distributed video transcoding is presented. At the same time, the paper compares the Hadoop-based implementation with the HTCondor-based batch implementation of distributed video. The experiment shows that the distributed video processing system based on Hadoop has the advantages of distributed file system support and perfect task monitoring without losing the efficiency of video processing.