Interacting with proactive public displays

In this paper we describe the design and architecture of an adaptive proactive environment in which information, which reflects the communal interests of current inhabitants, is proactively displayed on large-scale public displays. Adaptation is achieved through implicit communication between the environment and personal sensor devices worn by users. These devices, called Pendle, serve two purposes: they store and make available to the environment user preferences, and they allow users to override the environment's proactive behavior by means of simple gestures. The result is a smooth integration of environment-controlled interaction (experienced by the user as implicit interaction, triggered by their presence) and user-controlled explicit interaction. Initial results show that user-controlled adaptation leads to an engaging user experience that is unobtrusive and not distracting.     

Tangible displays for the masses: spatial interaction with handheld displays by using consumer depth cameras

Spatially aware handheld displays are a promising approach to interact with complex information spaces in a more natural way by extending the interaction space from the 2D surface to the 3D physical space around them. This is achieved by utilizing their spatial position and orientation for interaction purposes. Technical solutions for spatially tracked displays already exist in research laboratories, e.g., embedded in a tabletop environment. Along with a large stationary screen, such multi-display systems provide a rich design space with a variety of benefits to users, e.g., the explicit support of co-located parallel work and collaboration. As we see a great future in the underlying interaction principles, the question is how the technology can be made accessible to the public. With our work, we want to address this issue. In the long term, we envision a low-cost tangible display ecosystem that is suitable for everyday usage and supports both active displays (e.g., the iPad) and passive projection media (e.g., paper screens and everyday objects such as a mug). The two major contributions of this article are a presentation of an exciting design space and a requirement analysis regarding its technical realization with special focus on a broad adoption by the public. In addition, we present a proof of concept system that addresses one technical aspect of this ecosystem: the spatial tracking of tangible displays with a consumer depth camera (Kinect).     

Multicamera human detection and tracking supporting natural interaction with large-scale displays

This paper presents a computer vision system that supports non-instrumented, location-based interaction of multiple users with digital representations of large-scale artifacts. The proposed system is based on a camera network that observes multiple humans in front of a very large display. The acquired views are used to volumetrically reconstruct and track the humans robustly and in real time, even in crowded scenes and challenging human configurations. Given the frequent and accurate monitoring of humans in space and time, a dynamic and personalized textual/graphical annotation of the display can be achieved based on the location and the walk-through trajectory of each visitor. The proposed system has been successfully deployed in an archaeological museum, offering its visitors the capability to interact with and explore a digital representation of an ancient wall painting. This installation permits an extensive evaluation of the proposed system in terms of tracking robustness, computational performance and usability. Furthermore, it proves that computer vision technology can be effectively used to support non-instrumented interaction of humans with their environments in realistic settings.     

From human-machine interaction to human-machine cooperation

Since the 1960s, the rapid growth of information systems has led to the wide development of research on human-computer interaction (HCI) that aims at the designing of human-computer interfaces presenting ergonomic properties, such as friendliness, usability, transparency, etc. Various work situations have been covered--clerical work, computer programming, design, etc. However, they were mainly static in the sense that the user fully controls the computer. More recently, public and private organizations have engaged themselves in the enterprise of managing more and more complex and coupled systems by the means of automation. Modern machines not only process information, but also act on dynamic situations as humans have done in the past, managing stock exchange, industrial plants, aircraft, etc. These dynamic situations are not fully controlled and are affected by uncertain factors. Hence, degrees of freedom must be maintained to allow the humans and the machine to adapt to unforeseen contingencies. A human-machine cooperation (HMC) approach is necessary to address the new stakes introduced by this trend. This paper describes the possible improvement of HCI by HMC, the need for a new conception of function allocation between humans and machines, and the main problems encountered within the new forms of human-machine relationship. It proposes a conceptual framework to study HMC from a cognitive point of view in highly dynamic situations like aircraft piloting or air-traffic control, and concludes on the design of 'cooperative' machines.     

PDAs and shared public displays: Making personal information public, and public information personal

We are investigating how people move from individual to group work through the use of both personal digital assistants (PDAs) and a shared public display. Our scenario of this work covers the following activities. First, mobile individuals can create personal notes on their PDAs. Second, when individuals meet in real time, they can selectively publicise notes by moving them to a shared public display. Third, the group can manipulate personal and public items in real time through both PDAs and the shared public display, where the notes contained on both PDAs and public display are automatically synchronised. Finally, people leave a meeting with a common record of their activity. We describe our SharedNotes system that illustrates how people move through this scenario. We also highlight a variety of problematic design issues that result from having different devices and from having the system enforce a rigid distinction between personal and public information.     

Visual performance on CRT screens and hard-copy displays

Three experiments examined the effect of hard-copy print and CRT screens of different resolution/addressability ratios (RAR) on accommodation and visual search performance. Three different display modes were generated with an IBM personal computer and a Princeton Graphics Terminal: capital letters in high-RAR mode, capital letters in low-RAR mode, and capital letters in high-RAR mode programmed to simulate the low-RAR mode. Same-sized letters were also presented on hard-copy print. Experiment 1 demonstrated that accommodation to hardcopy print and high-RAR screens was more accurate than to low-RAR screens. In Experiment 2 the spatial frequency channels activated by each display were evaluated by measuring the effect of display adaptation on the contrast-sensitivity function. The results suggested that high-RAR screens and hard-copy print activated higher-frequency channels than did low-RAR screens. In Experiment 3 significantly better visual search performance was obtained for high-RAR screens and for hard-copy print than for low-RAR screens. It was concluded that screen RAR is an important variable to consider in the design of CRTs.     

The impact of drowsiness on in-vehicle human-machine interaction with head-up and head-down displays

Various studies show that drowsiness reduces driver alertness and can significantly affect driver performance. In this paper, we investigate the effect of drowsiness on the interaction with the in-vehicle infotainment system (IVIS) while driving. The motivation was to investigate whether a specific type of user interface can provide better performance and lower distraction when the driver is drowsy. The users were asked to navigate a vehicle in a driving simulator and simultaneously perform a set of tasks of varying complexity first when they were rested and alert and then when they were drowsy, after 7 h without sleep. A hierarchical, list-based menu was presented using a stereoscopic head-up display (HUD) and a head-down display (HDD). Based on the results, no general and statistically significant connection was found between drowsiness and driving performance. Surprisingly, the secondary task performance was even better when participants were drowsy, which was evident from the faster task completion times. This could be attributed to extra efforts invested in executing tasks as a result of the participants being aware of their drowsiness. However, when comparing the participants’ performance using HUD and HDD displays, the results showed that using HUD introduces less mental fatigue than using HDD but only in rested and alert condition. No significant difference between the displays was found in the drowsy state.     

Visualization task performance with 2D, 3D, and combination displays

We describe a series of experiments that compare 2D displays, 3D displays, and combined 2D/3D displays (orientation icon, ExoVis, and clip planes) for relative position estimation, orientation, and volume of interest tasks. Our results indicate that 3D displays can be very effective for approximate navigation and relative positioning when appropriate cues, such as shadows, are present. However, 3D displays are not effective for precise navigation and positioning except possibly in specific circumstances, for instance, when good viewing angles or measurement tools are available. For precise tasks in other situations, orientation icon and ExoVis displays were better than strict 2D or 3D displays (displays consisting exclusively of 2D or 3D views). The combined displays had as good or better performance, inspired higher confidence, and allowed natural, integrated navigation. Clip plane displays were not effective for 3D orientation because users could not easily view more than one 2D slice at a time and had to frequently change the visibility of individual slices. Major factors contributing to display preference and usability were task characteristics, orientation cues, occlusion, and spatial proximity of views that were used together.     

From the private into the public: privacy-respecting mobile interaction techniques for sharing data on surfaces

Interactive horizontal surfaces provide large semi-public or public displays for colocated collaboration. In many cases, users want to show, discuss, and copy personal information or media, which are typically stored on their mobile phones, on such a surface. This paper presents three novel direct interaction techniques (Select&Place2Share, Select&Touch2Share, and Shield&Share) that allow users to select in private which information they want to share on the surface. All techniques are based on physical contact between mobile phone and surface. Users touch the surface with their phone or place it on the surface to determine the location for information or media to be shared. We compared these three techniques with the most frequently reported approach that immediately shows all media files on the table after placing the phone on a shared surface. The results of our user study show that such privacy-preserving techniques are considered as crucial in this context and highlight in particular the advantages of Select&Place2Share and Select&Touch2Share in terms of user preferences, task load, and task completion time.     

The interaction of chromostereopsis and stereopsis in stereoscopic CRT displays

To give operators of high-speed, highly-manoeuvrable vehicles the ability to keep pace with their increasingly dynamic environment, stereoscopic 3-D displays may soon replace conventional 2-D displays. Important to the development of stereoscopic 3-D displays is the interaction of perceived depth created by hues (chromostereopsis) and perceived depth created by presenting different images of a single object to the left and right eye of the observer (stereopsis). The purpose of this research is to evaluate the interaction of chromostereopsis and stereopsis on a stereoscopic CRT by determining the level of accuracy with which subjects can properly interpret the relative depth differences of adjacent symbols containing six levels of hue and seven levels of stereoscopic disparity. This research demonstrated that hue, disparity, and the interaction of hue and disparity significantly influenced one's perception of depth on a stereoscopic monitor. The results suggest that caution should be exercised by the stereoscopic 3-D display format designer when choosing hues to represent images located in close proximity on a stereoscopic display. Due to the chromostereoscopic effect on the perception of depth, hues on extreme ends of the colour spectrum should not be used in situations where less than 3.39 arc minutes of disparity difference is being portrayed on a stereoscopic display, unless the hues are consistently being used to alter the depth presented by stereoscopic disparity, or the chromostereoscopic depth resulting from certain hues is consistently nullified by altering disparity levels accordingly.     

Interacting with large displays

Today's computing systems often rely on display technologies originally designed for the desktop, but wall-size screens and large table-top displays have qualitatively different requirements. As our research moves beyond basic input devices and interaction techniques, we aim to enable a complete large-screen user experience by rethinking existing applications as well as developing new ones that exploit these displays unique capabilities.     

Tactile feedback enhanced hand gesture interaction at large, high-resolution displays

Human beings perceive their surroundings based on sensory information from diverse channels. However, for human–computer interaction we mostly restrict the user on visual perception. In this paper, we contribute to the investigation of tactile feedback as an additional perception modality. Therefore, we will first discuss existing user studies and provide a classification scheme for tactile feedback techniques. We will then present and discuss a comparative evaluation study based on the ISO 9241-9 [Ergonomic requirements for office work with visual display terminals (VDTs) – Part 9: requirements for non-keyboard input devices, 2000]. The 20 participants performed horizontal and vertical one-directional tapping tasks with hand gesture input with and without tactile feedback in front of a large, high-resolution display. In contrast to previous research, we cannot confirm a benefit of tactile feedback on user performance. Our results show no significant effect in terms of throughput (effective index of performance (IPe)) and even a significant higher error rate for horizontal target alignment when using tactile feedback. Based on these results, we suggest that tactile feedback can interfere with other senses in a negative way, resulting in the observed higher error rate for horizontal targets. Therefore, more systematic research is needed to clarify the influencing factors on the usefulness of tactile feedback. Besides these results, we found a significant difference in favor of the horizontal target alignment compared with the vertical one in terms of the effective index of performance (IPe), confirming the work by Dennerlein et al. [Force feedback improves performance for steering and combined steering–targeting tasks, in: CHI ’00: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, New York, NY, USA, 2000, pp. 423–429].     

From lab to stage: practice-based research in performance

Performance animation is a new artform, which opens new possibilities in the performing arts. Experimental practice-based research in this area is necessary to integrate digital media, motion capture and tracking technology in a meaningful way. This article describes an experimental research project based on collaboration between artists and researchers developing artistic prototypes.     

From monolithic to component-based performance evaluation of software architectures

Model-based performance evaluation methods for software architectures can help architects to assess design alternatives and save costs for late life-cycle performance fixes. A recent trend is component-based performance modelling, which aims at creating reusable performance models; a number of such methods have been proposed during the last decade. Their accuracy and the needed effort for modelling are heavily influenced by human factors, which are so far hardly understood empirically. Do component-based methods allow to make performance predictions with a comparable accuracy while saving effort in a reuse scenario? We examined three monolithic methods (SPE, umlPSI, Capacity Planning (CP)) and one component-based performance evaluation method (PCM) with regard to their accuracy and effort from the viewpoint of method users. We conducted a series of three experiments (with different levels of control) involving 47 computer science students. In the first experiment, we compared the applicability of the monolithic methods in order to choose one of them for comparison. In the second experiment, we compared the accuracy and effort of this monolithic and the component-based method for the model creation case. In the third, we studied the effort reduction from reusing component-based models. Data were collected based on the resulting artefacts, questionnaires and screen recording. They were analysed using hypothesis testing, linear models, and analysis of variance. For the monolithic methods, we found that using SPE and CP resulted in accurate predictions, while umlPSI produced over-estimates. Comparing the component-based method PCM with SPE, we found that creating reusable models using PCM takes more (but not drastically more) time than using SPE and that participants can create accurate models with both techniques. Finally, we found that reusing PCM models can save time, because effort to reuse can be explained by a model that is independent of the inner complexity of a component. The tasks performed in our experiments reflect only a subset of the actual activities when applying model-based performance evaluation methods in a software development process. Our results indicate that sufficient prediction accuracy can be achieved with both monolithic and component-based methods, and that the higher effort for component-based performance modelling will indeed pay off when the component models incorporate and hide a sufficient amount of complexity.     

Stereoscopic 3D displays and human performance: A comprehensive review

To answer the question: “what is 3D good for?” we reviewed the body of literature concerning the performance implications of stereoscopic 3D (S3D) displays versus non-stereo (2D or monoscopic) displays. We summarized results of over 160 publications describing over 180 experiments spanning 51 years of research in various fields including human factors psychology/engineering, human–computer interaction, vision science, visualization, and medicine. Publications were included if they described at least one task with a performance-based experimental evaluation of an S3D display versus a non-stereo display under comparable viewing conditions. We classified each study according to the experimental task(s) of primary interest: (a) judgments of positions and/or distances; (b) finding, identifying, or classifying objects; (c) spatial manipulations of real or virtual objects; (d) navigation; (e) spatial understanding, memory, or recall and (f) learning, training, or planning. We found that S3D display viewing improved performance over traditional non-stereo (2D) displays in 60% of the reported experiments. In 15% of the experiments, S3D either showed a marginal benefit or the results were mixed or unclear. In 25% of experiments, S3D displays offered no benefit over non-stereo 2D viewing (and in some rare cases, harmed performance). From this review, stereoscopic 3D displays were found to be most useful for tasks involving the manipulation of objects and for finding/identifying/classifying objects or imagery. We examine instances where S3D did not support superior task performance. We discuss the implications of our findings with regard to various fields of research concerning stereoscopic displays within the context of the investigated tasks.