Augmented reality camera tracking with homographies

To realistically integrate 3D graphics into an unprepared environment, camera position must be estimated by tracking natural image features. We apply our technique to cases where feature positions in adjacent frames of an image sequence are related by a homography, or projective transformation. We describe this transformation's computation and demonstrate several applications. First, we use an augmented notice board to explain how a homography, between two images of a planar scene, completely determines the relative camera positions. Second, we show that the homography can also recover pure camera rotations, and we use this to develop an outdoor AR tracking system. Third, we use the system to measure head rotation and form a simple low-cost virtual reality (VR) tracking solution.     

DietCam: Automatic dietary assessment with mobile camera phones

Obesity has become a severe health problem in developed countries, and a healthy food intake has been recognized as the key factor for obesity prevention. This paper presents a mobile phone based system, DietCam, to help assess food intakes with few human interventions. DietCam only requires users to take three images or a short video around the meal, then it will do the rest. The experiments of DietCam in real restaurants verify the possibility of food recognition with vision techniques.     

Stabilization and extraction of 2D barcodes for camera phones

With the ubiquity of cellular phones, mobile applications with 2D barcodes have drawn a lot of attentions in recent years. When a user takes a barcode image with the camera in a mobile device, the captured image tends to be blurred due to camera shaking when the user presses the shutter. In addition, the captured image includes part of the complex background of the page with the barcode. In this paper, we point out that the above two issues, which have not been identified in previous works, deteriorate the accuracy of barcode recognition in the mobile computing. We then propose an efficient and effective algorithm to restore and extract 2D barcode from a complex background in a camera-shaken image. Compared with previous approaches, our algorithm outperforms in not only smaller running time but also higher accuracy of the barcode recognition in the mobile computing.     

A reality check for radiometric camera response recovery algorithms

The radiometric response of a camera governs the relationship between the incident light on the camera sensor and the output pixel values that are produced. This relationship, which is typically unknown and nonlinear, needs to be estimated for applications that require accurate measurement of scene radiance. Until now, various camera response recovery algorithms have been proposed each with different merits and drawbacks. However, an evaluation study that compares these algorithms has not been presented. In this work, we aim to fill this gap by conducting a rigorous experiment that evaluates the selected algorithms with respect to three metrics: consistency, accuracy, and robustness. In particular, we seek the answer of the following four questions: (1) Which camera response recovery algorithm gives the most accurate results? (2) Which algorithm produces the camera response most consistently for different scenes? (3) Which algorithm performs better under varying degrees of noise? (4) Does the sRGB assumption hold in practice? Our findings indicate that Grossberg and Nayar's (GN) algorithm (2004 [1]) is the most accurate; Mitsunaga and Nayar's (MN) algorithm (1999 [2]) is the most consistent; and Debevec and Malik's (DM) algorithm (1997 [3]) is the most resistant to noise together with MN. We also find that the studied algorithms are not statistically better than each other in terms of accuracy although all of them statistically outperform the sRGB assumption. By answering these questions, we aim to help the researchers and practitioners in the high dynamic range (HDR) imaging community to make better choices when choosing an algorithm for camera response recovery.     

Binocular mobile augmented reality based on stereo camera tracking

Mobile augmented reality (AR) applications have become feasible with the evolution of mobile hardware. For example, the advent of the smartphone allowed implementing real-time mobile AR, which triggered the release of various applications. Recently, rapid development of display technology, especially for stereoscopic displays, has encouraged researches to implement more immersive and realistic AR. In this paper, we present a framework of binocular augmented reality based on stereo camera tracking. Our framework was implemented on a smartphone and supports autostereoscopic display and video see-through display in which a smartphone can be docked. We modified edge-based 3-D object tracking in order to estimate poses of left and right cameras jointly; this guarantees consistent registration across left and right views. Then, virtual contents were overlaid onto camera images using estimated poses, and the augmented stereo images were distorted to be shown through a video see-through display. The feasibility of the proposed framework is shown by experiments and demonstrations.     

How mobile phones perform in collaborative augmented reality (CAR) applications

This paper presents the experimental analysis of mobile phones for Augmented Reality marker tracking, a core task that any CAR application must include. The results show that the most time consuming stage is the marker detection stage, followed by the image acquisition stage. Moreover, the rendering stage is decoupled on some devices, depending on the operative system used. This decoupling process allows avoiding low refresh rates, facilitating the collaborative work. However, the use of multicore devices does not significantly improve the performance provided by CAR applications. Finally, the results show that unless a poor network bandwidth makes the network to become the system bottleneck, the performance of CAR applications based on mobile phones will be limited by the detection stage. These results can be used as the basis for an efficient design of CAR systems and applications based on mobile phones.     

A real-time low-cost marker-based multiple camera tracking solution for virtual reality applications

One of the reasons of how virtual reality applications penetrate industrial production chains slowly is because of significant investment costs to purchase adequate supporting hardware. As a consequence such applications are only available to major companies and fail to benefit the production processes of small and medium enterprises. In this article, we introduce PTrack, a real-time, low-cost, marker-based multiple camera tracking solution for virtual reality providing the accuracy and scalability usually found in much more expensive tracking systems. PTrack is composed of single camera tracking PTrack Units. Each unit is connected to a video camera equipped with infrared strobes and features a novel iterative geometric pose estimation algorithm which does marker-based single camera tracking and is therefore completely autonomous. Multiple PTrack units successively extend the tracking range of the system. For a smooth transition of tracked labels from one camera to another, camera range areas must overlap to form a contiguous tracking space. A PTrack Sensor Fusion Module then computes the pose of a certain label within tracking space and forwards it to interested applications. A universal test setup for optical tracking systems has been built allowing to measure translational and rotational accuracy of PTrack as well as of competing systems.     

Source camera identification via low dimensional PRNU features

Multimedia Tools and Applications - Identifying the source of digital images is the key task in the community of image forensics. Sensor pattern noise dominantly serves as an intrinsic fingerprint...     

Interactive theatre via mixed reality and Ambient Intelligence

Our system of Mixed Reality and 3D Live with Ambient Intelligence (AmI) is indented to bring performance art to the people while offering to the performance artists a creative tool to extend the grammar of the traditional theatre. Actors and dancers at different places are captured by multiple cameras and their images are rendered in 3D form in such a way that they can play and dance together on the same place in real-time. Our Quanticum Man is an allegory of the time of the General Relativity and the matter of the Quantum Mechanics. The new type of interactive theatre enables social networking by supporting simultaneous participants in human-to-human social manner.     

Parallax correction via disparity estimation in a multi-aperture camera

In this paper, an image fusion algorithm is proposed for a multi-aperture camera. Such camera is a feasible alternative to traditional Bayer filter camera in terms of image quality, camera size and camera features. The camera consists of several camera units, each having dedicated optics and color filter. The main challenge of a multi-aperture camera arises from the fact that each camera unit has a slightly different viewpoint. Our image fusion algorithm corrects the parallax error between the sub-images using a disparity map, which is estimated from the single-spectral images. We improve the disparity estimation by combining matching costs over multiple views using trifocal tensors. Images are matched using two alternative matching costs, mutual information and Census transform. We also compare two different disparity estimation methods, graph cuts and semi-global matching. The results show that the overall quality of the fused images is near the reference images.     

RGB-D point cloud registration via infrared and color camera

Multimedia Tools and Applications - The iterative closest point (ICP) algorithm is widely used for rigid registration for its simplicity and speed, but the registration is easy to fail when point...     

Real-time and accurate extrinsic camera parameter estimation using feature landmark database for augmented reality

In the field of augmented reality (AR), many kinds of vision-based extrinsic camera parameter estimation methods have been proposed to achieve geometric registration between real and virtual worlds. Previously, a feature landmark-based camera parameter estimation method was proposed. This is an effective method for implementing outdoor AR applications because a feature landmark database can be automatically constructed using the structure-from-motion (SfM) technique. However, the previous method cannot work in real time because it entails a high computational cost or matching landmarks in a database with image features in an input image. In addition, the accuracy of estimated camera parameters is insufficient for applications that need to overlay CG objects at a position close to the user's viewpoint. This is because it is difficult to compensate for visual pattern change of close landmarks when only the sparse depth information obtained by the SfM is available. In this paper, we achieve fast and accurate feature landmark-based camera parameter estimation by adopting the following approaches. First, the number of matching candidates is reduced to achieve fast camera parameter estimation by tentative camera parameter estimation and by assigning priorities to landmarks. Second, image templates of landmarks are adequately compensated for by considering the local 3-D structure of a landmark using the dense depth information obtained by a laser range sensor. To demonstrate the effectiveness of the proposed method, we developed some AR applications using the proposed method.     

Heads up and camera down: a vision-based tracking modality for mobile mixed reality

Anywhere Augmentation pursues the goal of lowering the initial investment of time and money necessary to participate in mixed reality work, bridging the gap between researchers in the field and regular computer users. Our paper contributes to this goal by introducing the GroundCam, a cheap tracking modality with no significant setup necessary. By itself, the GroundCam provides high frequency, high resolution relative position information similar to an inertial navigation system, but with significantly less drift. We present the design and implementation of the GroundCam, analyze the impact of several design and run-time factors on tracking accuracy, and consider the implications of extending our GroundCam to different hardware configurations. Motivated by the performance analysis, we developed a hybrid tracker that couples the GroundCam with a wide area tracking modality via a complementary Kalman filter, resulting in a powerful base for indoor and outdoor mobile mixed reality work. To conclude, the performance of the hybrid tracker and its utility within mixed reality applications is discussed.     

Virtual reality and mobile phones in the treatment of generalized anxiety disorders: a phase-2 clinical trial

Several studies have demonstrated that exposure therapy—in which the patient is exposed to specific feared situations or objects that trigger anxiety—is an effective way to treat anxiety disorders. However, to overcome a number of limitations inherent in this approach—lack of full control of the situation, costs and time required, etc.—some therapists have started to add virtual reality (VR) to the in vivo exposure-based therapy, providing in-office, controlled exposure therapy. Compared to the in vivo exposure, VR Exposure Therapy (VRET) is completely controlled: the quality, the intensity and the frequency of the exposure are decided by the therapist, and the therapy can be stopped at any time if the patient does not tolerate it. Moreover, the flexibility of a virtual experience allows the patient to experience situations that are often much worse and more exaggerated than those that are likely to be encountered in real life. However, a critical issue underlying the use of VRET in the treatment of anxiety-related disorders is the lack of a virtual reality system in the patient’s real-life context. In this paper, we present a clinical protocol for the treatment of Generalized Anxiety Disorders (GAD) based on the ubiquitous use of a biofeedback-enhanced VR system. The protocol includes the use of a mobile exposure system allowing patients to perform the virtual experience in an outpatient setting. A between-subjects study, involving 25 GAD patients, was carried out to verify the efficacy of the proposed approach. The clinical data in this pilot study seemed to support the efficacy of the ubiquitous approach.     

Gesture-based interaction via finger tracking for mobile augmented reality

The goal of this research is to explore new interaction metaphors for augmented reality on mobile phones, i.e. applications where users look at the live image of the device’s video camera and 3D virtual objects enrich the scene that they see. Common interaction concepts for such applications are often limited to pure 2D pointing and clicking on the device’s touch screen. Such an interaction with virtual objects is not only restrictive but also difficult, for example, due to the small form factor. In this article, we investigate the potential of finger tracking for gesture-based interaction. We present two experiments evaluating canonical operations such as translation, rotation, and scaling of virtual objects with respect to performance (time and accuracy) and engagement (subjective user feedback). Our results indicate a high entertainment value, but low accuracy if objects are manipulated in midair, suggesting great possibilities for leisure applications but limited usage for serious tasks.     

I spy with my little eye: Samsung is about to ban camera phones from its factories — why?

The other day I was having lunch with an investigator who received a call on his mobile telephone. As he took the call I noticed he was using the latest video and camera mobile phone technology. As an amateur photographer I was tempted to ridicule the abysmal quality of the picture image that these cameras usually produce. I have seen my own image captured unawares by friends and colleagues using these devices and the picture quality was never good — despite my movie star looks and natural photogenic charm. But as I gently ridiculed this technology, the investigator raised a salient point that I had failed to appreciate — that I had been photographed, yet again completely unawares, this time by him as he answered his call.     

Robust structure from motion with affine camera via low-rank matrix recovery

We present a novel approach to structure from motion that can deal with missing data and outliers with an affine camera. We model the corruptions as sparse error. Therefore the structure from motion problem is reduced to the problem of recovering a low-rank matrix from corrupted observations. We first decompose the matrix of trajectories of features into low-rank and sparse components by nuclear-norm and l1-norm minimization, and then obtain the motion and structure from the low-rank components by the classical factorization method. Unlike pervious methods, which have some drawbacks such as depending on the initial value selection and being sensitive to the large magnitude errors, our method uses a convex optimization technique that is guaranteed to recover the low-rank matrix from highly corrupted and incomplete observations. Experimental results demonstrate that the proposed approach is more efficient and robust to large-scale outliers.     

Training-free referenceless camera image blur assessment via hypercomplex singular value decomposition

Blur plays an important role in the perception of camera image quality. Generally, blur leads to attenuation of high frequency information and accordingly changes the image energy. Quaternion describes the color information as a whole. Recent researches in quaternion singular value decomposition show that the singular values and singular vectors of the quaternion can capture the distortion of color images, and thus we reasonably suppose that singular values can be utilized to evaluate the sharpness of camera images. Motivated by this, a novel training-free blind quality assessment method considering the integral color information and singular values of the distorted image is proposed to evaluate the sharpness of camera images. The blurred camera image is first converted to LAB color space and divided into blocks. Then pure quaternion is utilized to represent pixels of the blurred camera image and the energy of every block are obtained. Inspired by the human visual system appears to assess image sharpness based on the sharpest region of the image, the local sharpness normalized energy is defined as the sharpness score of the blurred camera image. Experimental results have demonstrated the effectiveness of the proposed metric compared with popular sharpness image quality metrics.