An FPGA stereo matching unit based on fuzzy logic

Stereo matching is one of the most used algorithms in real-time image processing applications such as positioning systems for mobile robots, three-dimensional building mapping and recognition, detection and three-dimensional reconstruction of objects. In order to improve the performance, stereo matching algorithms often have been implemented in dedicated hardware such as FPGA or GPU devices. In this paper an FPGA stereo matching unit based on fuzzy logic is described. The proposed algorithm consists of three stages. First, three similarity parameters inherent to each pixel contained in the input stereo pair are computed. Then, the similarity parameters are sent to a fuzzy inference system which determines a fuzzy-similarity value. Finally, the disparity value is defined as the index which maximizes the fuzzy-similarity values (zero up to d_max). Dense disparity maps are computed at a rate of 76 frames per second for input stereo pairs of 1280 × 1024 pixel resolution and a maximum expected disparity equal to 15. The developed FPGA architecture provides reduction of the hardware resource demand compared to other FPGA-based stereo matching algorithms: near to 72.35% for logic units and near to 32.24% for bits of memory. In addition, the developed FPGA architecture increases the processing speed: near to 34.90% pixels per second and outperforms the accuracy of most of real-time stereo matching algorithms in the state of the art.     

Stereo disparity optimization with depth change constraint based on a continuous video

Three-dimensional reconstruction based on stereo vision technology is an important research direction in the field of computer vision, and has a wide range of applications in industrial measurement, medical image reconstruction, cultural relic preservation, robot navigation, virtual reality and other fields. However, the three-dimensional reconstruction of moving objects usually has poor accuracy, low efficiency and poor visualization effect due to the image noise, motion blur, complex and time-consuming calculation etc. In this article, a disparity optimization method based on depth change constraint is proposed, which utilizes the correlation of the adjacent frames in the continuous video sequence to eliminate mismatches and correct the wrong disparity values by introducing a depth change constraint threshold. The experiments on the video images which are taken by a binocular stereo vision system demonstrate that our method of removing incorrect matches bears satisfactory results and it can greatly improve the effect of the three-dimensional reconstruction of the moving objects.     

Video-based, real-time multi-view stereo

We investigate the problem of obtaining a dense reconstruction in real-time, from a live video stream. In recent years, multi-view stereo (MVS) has received considerable attention and a number of methods have been proposed. However, most methods operate under the assumption of a relatively sparse set of still images as input and unlimited computation time. Video based MVS has received less attention despite the fact that video sequences offer significant benefits in terms of usability of MVS systems. In this paper we propose a novel video based MVS algorithm that is suitable for real-time, interactive 3d modeling with a hand-held camera. The key idea is a per-pixel, probabilistic depth estimation scheme that updates posterior depth distributions with every new frame. The current implementation is capable of updating 15 million distributions/s. We evaluate the proposed method against the state-of-the-art real-time MVS method and show improvement in terms of accuracy.     

基于卷积神经网络的视频图像超分辨率重建方法

为了进一步增强视频图像超分辨率重建的效果,研究利用卷积神经网络的特性进行视频图像的空间分辨率重建,提出了一种基于卷积神经网络的视频图像重建模型。采取预训练的策略用于重建模型参数的初始化,同时在多帧视频图像的空间和时间维度上进行训练,提取描述主要运动信息的特征进行学习,充分利用视频帧间图像的信息互补进行中间帧的重建。针对帧间图像的运动模糊,采用自适应运动补偿加以处理,对通道进行优化输出得到高分辨率的重建图像。实验表明,重建视频图像在平均客观评价指标上均有较大提升（PSNR +0.4 dB / SSIM +0.02）,并且有效减少了图像在主观视觉效果上的边缘模糊现象。与其他传统算法相比,在图像评价的客观指标和主观视觉效果上均有明显的提升,为视频图像的超分辨率重建提供了一种基于卷积神经网络的新颖架构,也为进一步探索基于深度学习的视频图像超分辨率重建方法提供了思路。     

Real-time stereo on GPGPU using progressive multi-resolution adaptive windows

We introduce a new GPGPU-based real-time dense stereo matching algorithm. The algorithm is based on a progressive multi-resolution pipeline which includes background modeling and dense matching with adaptive windows. For applications in which only moving objects are of interest, this approach effectively reduces the overall computation cost quite significantly, and preserves the high definition details. Running on an off-the-shelf commodity graphics card, our implementation achieves a 36 fps stereo matching on 1024 × 768 stereo video with a fine 256 pixel disparity range. This is effectively same as 7200 M disparity evaluations per second. For scenes where the static background assumption holds, our approach outperforms all published alternative algorithms in terms of the speed performance, by a large margin. We envision a number of potential applications such as real-time motion capture, as well as tracking, recognition and identification of moving objects in multi-camera networks.     

Real-time video watermarking system on the compressed domain for high-definition video contents: Practical issues

Everyday, we encounter high-quality multimedia contents from HDTV broadcasting, DVD, and high-speed Internet services. These contents are, unhappily, processed and distributed without protection. This paper proposes a practical video watermarking technique on the compressed domain that is real-time and robust against video processing attacks. In particular, we focus on video processing that is commonly used in practice such as downscaling resolution, framerate changing, and transcoding. Most previous watermarking algorithms are unable to survive when these processings are strong or composite. We extract low frequency coefficients of frames in fast by partly decoding videos and apply a quantization index modulation scheme to embed and detect the watermark. On an Intel architecture computer, we implement a prototype system and measure performance against video processing attacks frequently occur in the real world. Simulation results show that our video watermarking system satisfies real-time requirements and is robust to protect the copyright of HD video contents.     

Consistent Depth Maps Recovery from a Video Sequence

This paper presents a novel method for recovering consistent depth maps from a video sequence. We propose a bundle optimization framework to address the major difficulties in stereo reconstruction, such as dealing with image noise, occlusions, and outliers. Different from the typical multi-view stereo methods, our approach not only imposes the photo-consistency constraint, but also explicitly associates the geometric coherence with multiple frames in a statistical way. It thus can naturally maintain the temporal coherence of the recovered dense depth maps without over-smoothing. To make the inference tractable, we introduce an iterative optimization scheme by first initializing the disparity maps using a segmentation prior and then refining the disparities by means of bundle optimization. Instead of defining the visibility parameters, our method implicitly models the reconstruction noise as well as the probabilistic visibility. After bundle optimization, we introduce an efficient space-time fusion algorithm to further reduce the reconstruction noise. Our automatic depth recovery is evaluated using a variety of challenging video examples.     

Stereoscopic visualization of laparoscope image using depth information from 3D model

Laparoscopic surgery is indispensable from the current surgical procedures. It uses an endoscope system of camera and light source, and surgical instruments which pass through the small incisions on the abdomen of the patients undergoing laparoscopic surgery. Conventional laparoscope (endoscope) systems produce 2D colored video images which do not provide surgeons an actual depth perception of the scene. In this work, the problem was formulated as synthesizing a stereo image of the monocular (conventional) laparoscope image by incorporating into them the depth information from a 3D CT model. Various algorithms of the computer vision including the algorithms for the feature detection, matching and tracking in the video frames, and for the reconstruction of 3D shape from shading in the 2D laparoscope image were combined for making the system. The current method was applied to the laparoscope video at the rate of up to 5 frames per second to visualize its stereo video. A correlation was investigated between the depth maps calculated with our method with those from the shape from shading algorithm. The correlation coefficients between the depth maps were within the range of 0.70–0.95 (P < 0.05). A t-test was used for the statistical analysis.     

基于多任务管理系统的高清视频处理

提出了一种基于多任务管理系统的高清视频处理技术,具有提升高清视频处理实时性,优化计算资源利用率,降低高清视频处理应用设计难度的特点。首先,介绍了面向异构多核计算环境的多任务管理系统,用于多种类型任务的调度执行及计算资源的负载均衡。在此基础上,设计了一种软件流水线,将对于高清视频的复杂而重复的处理过程分解成多类型的任务,提交至多任务管理系统。最后,对基于多任务管理系统的高清视频处理技术进行了实验验证。结果表明,异构多核环境下,高清视频处理的计算性能提升了3.7倍。     

A real-time video watermarking system with buffer sharing for video-on-demand service

Real-time watermarking for streaming video (such as VOD service) requires significant amounts of computing resources. To address this issue, we present a scalable watermarking scheme integrated in a parallel MPEG-2 engine. A content-based block selection algorithm is proposed to efficiently embed the pseudo-random watermark signatures into DCT blocks. Our watermark scheme also provides a robust way to synchronize the watermarked video to the original source at detectors and is very resilient against cumulative and temporal attack.We optimize the parallel watermark engine to achieve real-time watermarking performance. We found that the system throughput could suffer significant degradation when processing high-level MPEG-2 video (such as HDTV) due to inefficient management of memory space. Therefore, we investigated an efficient buffer management scheme consisting of two methods: First we reduced the transmission buffer in slave nodes by frames sharing between frames in the Group-of-Picture (GOP) level. Then we further reduce the buffer space by a dynamic on-demand allocation on the slave side. By solving the memory-shortage bottleneck, the proposed system can support real-time watermarking for multiple high-resolution (up to 1404 × 960) video.