A real-time multitarget tracking system with robust multichannel CNN-UM algorithms

This paper introduces a tightly coupled topographic sensor-processor and digital signal processor (DSP) architecture for real-time visual multitarget tracking (MTT) applications. We define real-time visual MTT as the task of tracking targets contained in an input image flow at a sampling-rate that is higher than the speed of the fastest maneuvers that the targets make. We utilize a sensor-processor based on the cellular neural network universal machine architecture that permits the offloading of the main image processing tasks from the DSP and introduces opportunities for sensor adaptation based on the tracking performance feedback from the DSP. To achieve robustness, the image processing algorithms running on the sensor borrow ideas from biological systems: the input is processed in different parallel channels (spatial, spatio-temporal and temporal) and the interaction of these channels generates the measurements for the digital tracking algorithms. These algorithms (running on the DSP) are responsible for distance calculation, state estimation, data association and track maintenance. The performance of the proposed system is studied using actual hardware for different video flows containing rapidly moving maneuvering targets.