Spatio-temporal object relationships in image sequences using adjacency matrices

In this work, we bring together object tracking and digital watermarking to solve the spatio-temporal object adjacency problem in image sequences. Spatio-temporal relationships are established by embedding objects with unique digital watermarks and then by propagating the watermark frame by frame. Watermark propagation is accomplished by an existing object tracking module so that a tracked object acquires its watermark from the correspondences established by the object tracker. The spatio-temporally marked image sequences can then be searched to establish spatial and temporal adjacency among objects without using traditional spatio-temporal graphs. Borrowing from graph theory, we construct binary adjacency matrices among tracked objects and develop interpretation rules to establish a track history for each object. Track history can be used to determine the arrival of new objects in frames or the changing of spatial and temporal positions of objects with respect to each other as they move through time and space.