Optically based direct manipulation for augmented reality

Augmented reality (AR) constitutes a very powerful three-dimensional user interface for many “hands-on” application scenarios. To fully exploit the AR paradigm, the computer must not only augment the real world, but also accept feedback from it. In this paper, we present an optical approach for collecting such feedback by analyzing video sequences to track users and the objects they work with. Our system can be set up in any room after quickly placing a few known optical targets in the scene. We present two demonstration scenarios to illustrate the overall concept and potential of our approach and then discuss the research issues involved.