海量视频人脸提取与识别并行框架设计及优化

近年来,监控设备大量应用于城市智能化建设中,而其产生的海量视频数据,亟待一种快速高效的解决方案。随着大数据处理技术的发展,使得处理海量视频数据成为可能。本文将视频数据集解耦合实现任务的并行处理,通过Spark读取数据流的同时获取关键帧的方式解决了解耦视频数据引起数据倍增问题,并对图片特征数据进行优化,进而在Spark上实现了具有高可扩展性并行处理海量视频数据的框架。本文在天河二号云平台上进行部署实验,分析实验结果表明随着处理节点个数增加本框架可以获得近线性的加速比。

Parallel framework for extraction and recognition for large-scale video

In recent years, a large number of monitoring equipment has been used in intelligent urban construction, and generate massive video data which are in urgent need of a fast and efficient processing. With the development of large data-processing technology, it is possible to handle massive video data. This paper decouples video files in order to realizing the parallelism of tasks, resolves the problem of multiplied data after decoupling video files and optimizes the feature of picture, this paper realizes the better scalable framework of processing video data on Spark. By deploying and experimenting over KylinCloud on Tianhe-2, the experimental results show that the framework gets linear speed-up ratio with the increase of processing node.