Hyperlinking reality via camera phones

A novel user interface concept for camera phones, called “Hyperlinking Reality via Camera Phones”, that we present in this article, provides a solution to one of the main challenges facing mobile user interfaces, that is, the problem of selection and visualization of actions that are relevant to the user in her current context. Instead of typing keywords on a small and inconvenient keypad of a mobile device, a user of our system just snaps a photo of her surroundings and objects in the image become hyperlinks to information. Our method commences by matching a query image to reference panoramas depicting the same scene that were collected and annotated with information beforehand. Once the query image is related to the reference panoramas, we transfer the relevant information from the reference panoramas to the query image. By visualizing the information on the query image and displaying it on the camera phone’s (multi-)touch screen, the query image augmented with hyperlinks allows the user intuitive access to information.