基于Spark的实时视频分析系统

视频监控技术在交通管理、公共安全、智慧城市等方面有着广泛的应用前景，且向着智能识别、实时处理、大数据分析的方向发展. 本文针对大规模实时视频监控提出了新的解决方案. 基于Spark streaming流式计算、分布式存储及OLAP框架，使多路视频处理在可扩展性、容错性及数据多维聚合分析上具有明显的优势. 系统根据视频处理算法划分为单机处理与分布式处理. 并将视频图像处理与数据分析耦合，利用Kafka消息队列与Spark streaming完成对多路视频输出数据的进一步操作. 结合分布式存储方案，并利用OLAP框架实现对海量数据实时多维聚合分析与高效实时查询.

Scalable Real-Time Video Analysis System Based on Spark

The video surveillance technology has a wide application prospect in traffic management, public safety, intelligent city, and is developing towards intelligent recognition, real-time processing, and large data analysis. In this paper, we propose a new system for large-scale real-time video surveillance. The system is based on Spark streaming, distributed storage and OLAP framework so that multi-channel video processing has obvious advantages in scalability, fault tolerance and data analysis of the multi-dimensional polymer. According to video processing algorithm, the processing module is divided into single machine processing and distributed processing. The video processing is separated from the data analysis, and the further operation of the multi-channel video output data is completed by using Kafka message queue and Spark streaming. Combining the distributed storage technology with OLAP framework, the system achieves real-time multi-dimensional data analysis and high-performance real-time query.