A systematic framework for on‐line calibration of a head‐mounted projection display for augmented‐reality systems

Abstract— Augmented reality (AR) is a technology in which computer‐generated virtual images are dynamically superimposed upon a real‐world scene to enhance a user's perceptions of the physical environment. A successful AR system requires that the overlaid digital information be aligned with the user's real‐world senses — a process known as registration. An accurate registration process requires the knowledge of both the intrinsic and extrinsic parameters of the viewing device and these parameters form the viewing and projection transformations for creating the simulations of virtual images. In our previous work, an easy off‐line calibration method in which an image‐based automatic matching method was used to establish the world‐to‐image correspondences was presented, and it is able to achieve subpixel accuracy. However, this off‐line method yields accurate registration only when a user's eye placements relative to the display device coincides with locations established during the offline calibration process. A likely deviation of eye placements, for instance, due to helmet slippage or user‐dependent factors such as interpupillary distance, will lead to misregistration. In this paper, a systematic on‐line calibration framework to refine the off‐line calibration results and to account for user‐dependent factors is presented. Specifically, based on an equivalent viewing projection model, a six‐parameter on‐line calibration method to refine the user‐dependent parameters in the viewing transformations is presented. Calibration procedures and results as well as evaluation experiments are described in detail. The evaluation experiments demonstrate the improvement of the registration accuracy.     

A multi‐plane optical see‐through head mounted display design for augmented reality applications

Augmented reality (AR) display technology greatly enhances users' perception of and interaction with the real world by superimposing a computer‐generated virtual scene on the real physical world. The main problem of the state‐of‐the‐art 3D AR head‐mounted displays (HMDs) is the accommodation‐vergence conflict because the 2D images displayed by flat panel devices are at a fixed distance from the eyes. In this paper, we present a design for an optical see‐through HMD utilizing multi‐plane display technology for AR applications. This approach manages to provide correct depth information and solves the accommodation‐vergence conflict problem. In our system, a projector projects slices of a 3D scene onto a stack of polymer‐stabilized liquid crystal scattering shutters in time sequence to reconstruct the 3D scene. The polymer‐stabilized liquid crystal shutters have sub‐millisecond switching time that enables sufficient number of shutters to achieve high depth resolution. A proof‐of‐concept two‐plane optical see‐through HMD prototype is demonstrated. Our design can be made lightweight, compact, with high resolution, and large depth range from near the eye to infinity and thus holds great potential for fatigue‐free AR HMDs.     

Optical characterization and ergonomical factors of near‐to‐eye displays

Abstract— Near‐to‐eye displays (NEDs) have unique optical properties requiring different characterization techniques compared to direct‐view display measurements. Here, a new version of a NED measurement system is introduced, and optical measurements of five commercially available consumer NED products are discussed. Luminance, focal distance, qualified viewing space, angular properties, and interocular differences are among the values. In addition, these results are compared to extensive subjective studies. The main intention is not to benchmark between the different products, but to show that display measurements are important for NEDs. According to the results, the determination of NED's characteristics helps to predict the subjective experiences, but the nature of the relation between subjective and objective findings is rather complex and depends on several NED‐, user‐, and task‐related features. Measured characteristics indicate that with a conventional biocular NED system approach of using two microdisplays and their enlarging optics, it is a design and a manufacturing challenge to build an ergonomically satisfactory NED device that fits everyone.     

An integral‐imaging‐based head‐mounted light field display using a tunable lens and aperture array

A head‐mounted light field display based on integral imaging is considered as one of the promising methods that can render correct or nearly correct focus cues and address the well‐known vergence‐accommodation conflict problem in head‐mounted displays. Despite its great potential, it still suffers some of the same limitations of conventional integral imaging‐based displays such as low spatial resolution and crosstalk. In this paper, we present a prototype design using tunable lens and aperture array to render 3D scenes over a large depth range while maintaining high image quality and minimizing crosstalk. Experimental results verify and show that the proposed design could significantly improve the viewing experience.     

High luminance and high see‐through head‐mounted displays with beam‐splitter‐array waveguides

We have developed head‐mounted displays with high transmittance and luminance, which could be utilized outside safely without dimming glasses. We first specified required optical performance specifications by considering user's safety and usability. In order to ensure that the user is able to recognize both surrounding environment and the image of the head‐mounted display, we set the target specification that the transmittance is higher than or equal to 85%, and the luminance contrast ratio is larger than or equal to 1.15 with display image of white solid pattern. We then developed the beam‐splitter‐array waveguide to achieve the requirements. It has advantages in high efficiency and high see‐through property. In order to determine configuration of the waveguide, we have performed optical ray trace simulation. We also established versatile waveguide measurement method applicable to different‐type waveguides. By utilizing the waveguide we have developed, we made a prototype of a head‐mounted display (HMD) with high transmittance 94% and high luminance 4.8 × 10³ cd/m² and thus luminance contrast ratio 1.25 under the sun. With these advantages, our HMD is suitable for usage outside including applications of work support systems where dimming effect is not preferred, and the HMD is used under the sun.     

High‐efficiency switchable optical elements for advanced head‐up displays

The Pancharatnam‐Berry optical elements (PBOEs) are utilized to enhance the performance of head‐up displays (HUDs). The Pancharatnam‐Berry lenses (PBLs) provide varifocal functionality and compensate chromatic aberrations, while the Pancharatnam‐Berry deflectors (PBDs) can function as optical combiners and waveguide couplers. Moreover, the Rigorous coupled wave analysis (RCWA) based on the scattering matrix is developed and applied in the structure optimization of PBOEs for HUD applications.     

Improved gray‐scale monochrome and color AMEL displays using analog pixel

Abstract— Active‐matrix electroluminescent (AMEL) microdisplays have been known for their numerous beneficial characteristics such as low weight, compactness, high brightness, and high contrast ratio. Beside these desirable characteristics, some of their drawbacks include difficulty in obtaining a high number of gray‐scale levels or a large number of colors, and interface‐electronics complexity. To address these drawbacks, AMEL displays using an analog addressing architecture have been developed. Utilizing this new driving scheme, 256 monochrome levels or 16 million colors are obtainable. Gray shade is proportional to an analog voltage stored on the hold node of each pixel. For color displays, each pixel is comprised of red, green, and blue subpixels arranged in vertical stripes, and can be sized independently to achieve the appropriate white balance. With the integration of control‐signal circuit blocks on the same substrate as the microdisplay, the number of input control signals is minimized and the display can be driven with very simple interface electronics. This results in low overall system cost, compact electronic packaging, and low power consumption. To accommodate most optical orientations, the display has built‐in modes to flip the image both vertically and horizontally. Additionally, the display supports multiple interlace addressing modes.     

Optical simulation for cross‐talk evaluation and improvement of autostereoscopic 3‐D displays with a projector array

Abstract— Multi‐view spatial‐multiplexed autostereoscopic 3‐D displays normally use a 2‐D image source and divide the pixels to generate perspective images. Due to the reduction in the resolution of each perspective image for a large view number, a super‐high‐resolution 2‐D image source is required to achieve 3‐D image quality close to the standard of natural vision. This paper proposes an approach by tiling multiple projection images with a low magnification ratio from a microdisplay to resolve the resolution issue. Placing a lenticular array in front of the tiled projection image can lead to an autostereoscopic display. Image distortion and cross‐talk issues resulting from the projection lens and pixel structure of the microdisplay have been addressed with proper selection of the active pixel and adequate pixel grouping and masking. Optical simulation has shown that a 37‐in. 12‐view autostereoscopic display with a full‐HD (1920 × 1080) resolution can be achieved with the proposed 3‐D architecture.     

MUTED: Multi‐user 3‐D display

Abstract— Research described in this paper encompasses the design and building of glasses‐free (autostereoscopic) displays that utilize a direct‐view liquid‐crystal display whose backlight is provided by a projector and novel steering optics. This is controlled by the output of a multi‐user head‐position tracker. As the displays employ spatial multiplexing on a liquid‐crystal‐display screen, they are inherently 2‐D/3‐D switchable with 2‐D being achieved by simply displaying the same image in the left and right channels. Two prototypes are described in this paper; one incorporating a holographic projector and the other a conventional LCOS projector. The LCOS projector version addresses the limitations of brightness, cross‐talk, banding in the images, and laser stability that occur in the holographic projector version. The future development is considered and a comparison between the prototypes and with other 3‐D displays is given.     

Depth‐enhanced integral‐imaging 3D display using different optical path lengths by polarization devices or mirror barrier array

Abstract— Depth‐enhanced integral three‐dimensional (3D) imaging using different optical path lengths by using a polarization selective mirror pair or mirror barrier array is proposed. In the proposed approach, the enhancement of image depth is achieved by repositioning two types of elemental image planes, thus effectively two central depth planes are obtained. One of the two implementation methods makes use of the two‐arm structure that has different optical path lengths and polarization‐selective mirrors. The other utilizes the mirror barrier array. The primary advantage of the method with polarization devices is that we can observe 3D images that maintain some level of viewing resolution with a large depth difference without any mechanical moving part. The mirror barrier array has the advantage of the compact thickness. We demonstrated and verified our proposals experimentally.     

Refractive‐reflective optical system for realizing an ultra‐thin rear projector

Abstract— This paper presents a new optical system used in an ultra‐thin rear projector with a 1500‐mm diagonal size and 260‐mm depth. A refractive‐reflective optical system was developed to achieve a large projection angle of 136° with a small optical distortion of 0.1%. The optics consists of a convex aspherical mirror and a refractive lens. In addition, a new Fresnel screen composed of hybrid blades of refractive‐TIR (total internal reflection) elements was developed to attain good uniformity of brightness and color within the image area.     

Vision‐tangible interactive display method for mixed and virtual reality: Toward the human‐centered editable reality

Building a human‐centered editable world can be fully realized in a virtual environment. Both mixed reality (MR) and virtual reality (VR) are feasible solutions to support the attribute of edition. Based on the current development of MR and VR, we present the vision‐tangible interactive display method and its implementation in both MR and VR. We address the issue of MR and VR together because they are similar regarding the proposed method. The editable mixed and virtual reality system is useful for studies, which exploit it as a platform. In this paper, we construct a virtual reality environment based on the Oculus Rift, and an MR system based on a binocular optical see‐through head‐mounted display. In the MR system about manipulating the Rubik's cube, and the VR system about deforming the virtual objects, the proposed vision‐tangible interactive display method is utilized to provide users with a more immersive environment. Experimental results indicate that the vision‐tangible interactive display method can improve the user experience and can be a promising way to make the virtual environment better.     

How color break‐up occurs in the human‐visual system: The mechanism of the color break‐up phenomenon

Abstract— In the present set of experiments, we examined the mechanisms underlying color break‐up (CBU), a phenomenon observed when images produced with a color‐sequential projector are viewed. The perceived position of CBU was measured during fast eye movement, saccade with static and briefly flashed stimuli. Results showed that CBU did not simply correspond to the locus of the stimulus on the retina during saccades, because the width of the CBU perception was narrower than the distance of the eye movements. This effect was thought to be related to visual stability, which allows objects to be perceived as stationary even when the eyes move and the retinal image changes position. Visual stability is assumed to operate by compensating for the change in retinal image position using eye‐position signals; however, this compensation is imperfect during saccades. Thus, incomplete compensation results in a CBU perception that is of a narrower width than the amplitude of the saccade. In conclusion, CBU cannot be simulated with moving video cameras because it results largely from the mechanisms of visual perception. Large inter‐individual differences in perception of CBU were also found. This observation also supports the idea that CBU depends on the mechanism of perception.     

The design and optimization of lens array for LED backlight in LCD imaging engine of helmet‐mounted display

The light‐emitting diode has become the mainstream lightsource for backlight in the liquid crystal display imaging engine of helmet‐mounted display. The paper proposes a secondary‐optics‐based design to increase the luminance and obtain a uniform illumination. Based on a point lightsource, a single double‐freeform‐surface lens is firstly designed. Then an optimization is performed according to the theory of edge‐ray to improve the uniformity for planar source. As a result, the uniformity reaches 83.4% in a circular illumination with a diameter of 8 mm. Then the lens is cut and four rectangular lenses are combined to form a lens array. But the combination leads to a non‐uniformity. So a method of optimizing the light energy distribution on the target surface is proposed. Finally, the designed lens array is manufactured. The practical measurement results show that the luminance increases by 96.4% compared with the traditional backlight and that the non‐uniformity slightly decreases by 0.86%. The lens array designed in this paper presents high practicability for applications in helmet‐mounted display.     

Efficient and compact green laser for micro‐projector applications

Abstract— Efficient and compact green lasers are keystone components for micro‐projector applications in mobile devices. An architecture that consists of an infrared‐producing DBR (distributed Bragg reflector) laser with a frequency‐doubling crystal is used to synthesize a green laser that has high electrical‐to‐optical conversion efficiency and can be modulated at speeds required for scanner‐based projectors. The design and performance of a green‐laser package that uses adaptive optics to overcome the challenge of maintaining alignment between the waveguides of the DBR laser and the frequency‐doubling crystal over temperature and lifetime is described. The adaptive optics technology that is employed uses the piezo‐based smooth impact drive mechanism (SIDM) actuators that offer a very small step size and a range of travel adequate for the alignment operation. The laser is shown to be compact (0.7 cm³ in volume) and capable of a wall‐plug efficiency approaching 10% (at 100‐mW green power). It was demonstrated that the adaptive optics enables operation over a wide temperature range (10–60°C) and provides the capability for low‐cost assembly of the device.     

Depth‐ and viewing‐angle‐enhanced 3‐D/2‐D switchable display system with high contrast ratio using multiple display devices and a lens array

Abstract— An improved 3‐D/2‐D switchable display system with enhanced depth and viewing angle by adding two LCD panels to an integral imaging system has been realized. The proposed system uses the see‐through property of an LCD panel and displays multiple sets of elemental images on the LCD panels to integrate them on multiple locations simultaneously. As a result, the depth of the 3‐D image can be enhanced. For wide viewing angles, the time‐multiplexing method was adopted by displaying mask patterns on the front LCD panel. In addition, another technique to increase the contrast ratio of the proposed system has also been developed. Some experimental results will be provided.     

Prototyping and optical evaluation of 9‐in. OCB time‐division‐multiplexing 18‐view 3‐D display

Abstract— An integral imaging time‐division‐multiplexing 18‐view 3‐D display based on the one‐dimensional integral‐imaging (1‐D‐II) technique using a 9‐in. OCB‐LCD, lenticular sheet, and active shutter has been developed. By simulating a lens shape and a shutter structure and analyzing the light‐beam profile of the increasing‐parallax‐number region to find the best conditions, depth range, and viewing angle were an enhanced and a brighter and flicker‐less 3‐D image with smooth motion parallax was obtained.     

Large‐area black/white bistable cholesteric liquid‐crystal display and the thermal‐addressing system

Abstract— A 3‐m‐length black/white bistable cholesteric liquid‐crystal display was made by a roll‐to‐roll process and the display area is 25 × 300 cm. The black/white performance was made by black nano‐pigment and blended ChLC droplets with different wavelengths. It was written by a thermal‐addressing system, realizing high resolution and low cost.     

In‐field assessment of display resolution and noise: Performance evaluation of a commercial measurement system

Abstract— Two key metrics of image quality for high‐fidelity displays, including medical displays, are resolution and noise. Until now, these properties have been primarily measured in laboratory settings. For the first time, a system consisting of a CCD camera and analysis software has been made commercially available for measuring the resolution and noise of medical displays in a clinical setting. This study aimed at evaluating this new product in terms of accuracy and precision. In particular, the project involved the measurement of the modulation transfer function (MTF) and the signal‐to‐noise ratio (SNR) of two medical imaging displays, one cathode‐ray tube (CRT) display and one liquid‐crystal display (LCD) using this camera/software system. To assess the system's precision, measurements were made multiple times at the same setting. To check for accuracy, the results were compared with published values of the MTF and noise for the same displays. The performance of the system was also ascertained as a function of the focus setting of the camera. The results indicated that for the LCD, when the camera is focused within ±0.6 mm of the optimum focus setting, the MTF values lie within approximately 14% of the best focus MTF at the Nyquist frequency and 11% of the optimum overall sharpness (∫ MTF² df). Similar results were obtained in the horizontal and vertical directions. For the CRT, this focus produced vertical and horizontal MTF values at the Nyquist frequency within 15.2% and 61.2% of the optimum focus MTF, respectively. The figures in terms of overall sharpness were 3.0% and 0.7%. The results for the noise measurements showed a repeatability of 3% for the LCD and 13% for the CRT and a relative (but not absolute) magnitude of the noise between the two displays reflective of prior measurements. Overall, the measurement system yielded reasonably precise resolution and noise results for both display devices. The accuracy was traceable to published results only for the MTF and for relative level of display noise with differences in the absolute magnitude of noise between current and prior measurements attributed to variations in the non‐standard techniques applied for display noise measurements.