Interaction with large ubiquitous displays using camera-equipped mobile phones

In the ubiquitous computing environment, people will interact with everyday objects (or computers embedded in them) in ways different from the usual and familiar desktop user interface. One such typical situation is interacting with applications through large displays such as televisions, mirror displays, and public kiosks. With these applications, the use of the usual keyboard and mouse input is not usually viable (for practical reasons). In this setting, the mobile phone has emerged as an excellent device for novel interaction. This article introduces user interaction techniques using a camera-equipped hand-held device such as a mobile phone or a PDA for large shared displays. In particular, we consider two specific but typical situations (1) sharing the display from a distance and (2) interacting with a touch screen display at a close distance. Using two basic computer vision techniques, motion flow and marker recognition, we show how a camera-equipped hand-held device can effectively be used to replace a mouse and share, select, and manipulate 2D and 3D objects, and navigate within the environment presented through the large display.     

GeoWall: Stereoscopic Visualization for Geoscience Research and Education

The GeoWall lets people visualize the structure and dynamics of the Earth in stereo, aiding their understanding of spatial relationships. Making use of inexpensive, polarized 3D glasses, entire classrooms or conference audiences can share the 3D experience. Since the hardware is portable, it's easy to bring visualizations to the audience. In this article we describe the GeoWall hardware and software and discuss how the GeoWall Consortium was instrumental in creating a community of users in a variety of disciplines. Using case studies, we show how the GeoWall helped research and education. Finally, we describe our recent work in high-resolution tiled displays     

Nightingallery: theatrical framing and orchestration in participatory performance

The Nightingallery project encouraged participants to converse, sing, and perform with a musically responsive animatronic bird, playfully interacting with the character while members of the public could look on and observe. We used Nightingallery to frame an HCI investigation into how people would engage with one another when confronted with unfamiliar technologies in conspicuously public, social spaces. Structuring performances as improvisational street theatre, we styled our method of exhibiting the bird character. We cast ourselves in supporting roles as carnival barkers and minders of the bird, presenting him as if he were a fantastical creature in a fairground sideshow display, allowing him the agency to shape and maintain dialogues with participants, and positioning him as the focal character upon which the encounter was centred. We explored how the anthropomorphic nature of the bird itself, along with the cultural connotations associated with the carnival/sideshow tradition helped signpost and entice participants through the trajectory of their encounters with the exhibit. Situating ourselves as secondary characters within the narrative defining the performance/use context, our methods of mediation, observation, and evaluation were integrated into the performance frame. In this paper, we explore recent HCI theories in mixed reality performance to reflect upon how genre-based cultural connotations can be used to frame trajectories of experience, and how manipulation of roles and agency in participatory performance can facilitate HCI investigation of social encounters with playful technologies.     

The design of virtual audiences: Noticeable and recognizable behavioral styles

Expressive virtual audiences are used in scientific research, psychotherapy, and training. To create an expressive virtual audience, developers need to know how specific audience behaviors are associated with certain characteristics of an audience, such as attitude, and how well people can recognize these characteristics. To examine this, four studies were conducted on a virtual audience and its behavioral models: (I) a perception study of a virtual audience showed that people (n = 24) could perceive changes in some of the mood, personality, and attitude parameters of the virtual audience; (II) a design experiment whereby individuals (n = 24) constructed 23 different audience scenarios indicated that the understanding of audience styles was consistent across individuals, and the clustering of similar settings of the virtual audience parameters revealed five distinct generic audience styles; (III) a perception validation study of these five audience styles showed that people (n = 100) could differentiate between some of the styles, and the audience's attentiveness was the most dominating audience characteristic that people perceived; (IV) the examination of the behavioral model of the virtual audience identified several typical audience behaviors for each style. We anticipate that future developers can use these findings to create distinct virtual audiences with recognizable behaviors.     

A poetry performance in a collaborative virtual environment

The article presents the design of a poetry performance in a collaborative virtual environment (CVE) and discusses the experience of staging this performance to an audience of 200 members of the general public. The performance involved an extensive collaboration among computer scientists, poets, a graphic artist, and an event production team. The article explores a range of technical and production issues arising from this experience, including: developing rich and engaging forms of content for CVEs; exploring the tension between autonomy of action for audience members and the management of a performance to encourage appropriate behaviors at relevant times and places; considering spatial scoping techniques in enabling different modes of awareness and communication between participants as an event unfolds; developing appropriate embodiments for performers and audience; developing appropriate navigation interfaces for different participants who may or may not have experienced virtual reality (VR); exploring different ways of situating interfaces to a virtual environment within different physical environments based on notions of mixed reality; understanding the issues involved in designing and managing such an event and how these might influence construction tools for CVEs     

Strength matters: Self-presentation to the strongest audience rather than lowest common denominator when faced with multiple audiences in social network sites

On social network sites (e.g. Facebook), individuals self-present to multiple audiences simultaneously 24 h a day. Prior research has inferred this results in a lowest common denominator effect (LCDE) whereby people constrain their online presentation to the standards of their strictest audience. However, this existing work neglects to address differences in the ‘value’ (social/economic) of the audience. Through the lens of self-presentation theory, we argue that it is not the strictest audience that constrains behavior but the strongest (i.e. that which has the highest score for standards and value combined). We call this the strongest audience effect (SAE). The aim of this research is to examine and contrast the LCDE and SAE. A survey of young Facebook users (n = 379) provides support for the SAE when compared to LCDE, with the strength of the strongest audience predicting behavioral constraint and also social anxiety. Additional insights are generated into which audiences are perceived as the strongest. This study contributes a novel and more holistic lens to understand self-presentation in the presence of multiple audiences in social network sites.     

On designing interactivity awareness for ambient displays

In the not-so-distant world in which ambient displays will likely become prevalent, immediate awareness of their interaction affordances to passerbies will be decisive for their usability. However, how to address awareness at user perception level represents a challenge for which little progress has been made so far in contrast with considerable advances in designing interaction techniques for such displays. Even though many interactive ambient displays exist with properly designed interfaces, people may not always be aware of their interactivity. This work addresses the problem of interactivity awareness by focusing on two important questions: How can people tell whether a public display is interactive or not? and, assuming interactivity, How can people tell what the interface is? A study was conducted in order to investigate factors potentially related to perceived interactivity. Results show that people’s evaluations are correct in most cases but they also tend to form incorrect perceptions in many other situations. We found that location, installation, reachability, and displayed content are factors that can influence people’s perceptions. Our findings are complemented with a discussion of techniques that can be used in the practice of designing for interactivity awareness.     

Reflection-through-performance: personal implications of documenting health behaviors for the collective

Previous work has examined how technology can support health behavior monitoring in social contexts. These tools incentivize behavior documentation through the promise of virtual rewards, rich visualizations, and improved co-management of disease. Social influence is leveraged to motivate improved behaviors through friendly competition and the sharing of emotional and informational support. Prior work has described how by documenting and sharing behaviors in these tools, people engage in performances of the self. This performance happens as users selectively determine what information to share and hide, crafting a particular portrayal of their identity. Much of the prior work in this area has examined the implications of systems that encourage people to share their behaviors with friends, family, and geographically distributed strangers. In this paper, we report upon the performative nature of behavior sharing in a system created for a different social group: the local neighborhood. We designed Community Mosaic (CM), a system with a collectivistic focus: this tool asks users to document their behaviors using photographs and text, but not for their own benefit—for the benefit of others in their community. Through a 6-week deployment of CM, we evaluated the nature of behavior sharing in this system, including participants’ motivations for sharing, the way in which this sharing happened, and the reflexive impact of sharing. Our findings highlight the performative aspects of photograph staging and textual narration and how sharing this content led participants to become more aware and evaluative of their behaviors, and led them to try to eat more healthfully. We conclude with recommendations for behavior monitoring tools, specifically examining the implications of users’ perceived audience and automated behavioral tracking on opportunities for reflection-through-performance.     

The Audience Funnel: Observations of Gesture Based Interaction With Multiple Large Displays in a City Center

Data are presented from observations of Magical Mirrors, a set of four large public displays with gesture-based interaction installed in downtown Berlin, Germany. The displays show a mirror image of the environment in front of them and react with optical effects to the gestures of the audience. Observations of audience behavior revealed recurring behavioral patterns, like glancing at a first display while passing it, moving the arms to cause some effects, then directly approaching one of the following displays and positioning oneself in the center of the display. This was often followed by positioning oneself in the center of the other displays to explore the possibilities of the different effects, and sometimes by taking photographs or videos. From these observations a framework of interaction with gesture-based public display systems was deduced. It describes the phases of passing by a display, viewing & reacting, subtle interaction, direct interaction, multiple interactions, and follow-up actions. Quantitative data of these behavioral phases was collected by observing 660 passers-by on 2 weekend evenings. This article shows how many passers-by pass the thresholds between these phases. This “Audience Funnel” should provide a framework to encourage systematic investigation of public display systems and enable comparability between different studies.     

Social decisions and fairness change when people’s interests are represented by autonomous agents

There has been growing interest on agents that represent people’s interests or act on their behalf such as automated negotiators, self-driving cars, or drones. Even though people will interact often with others via these agent representatives, little is known about whether people’s behavior changes when acting through these agents, when compared to direct interaction with others. Here we show that people’s decisions will change in important ways because of these agents; specifically, we showed that interacting via agents is likely to lead people to behave more fairly, when compared to direct interaction with others. We argue this occurs because programming an agent leads people to adopt a broader perspective, consider the other side’s position, and rely on social norms—such as fairness—to guide their decision making. To support this argument, we present four experiments: in Experiment 1 we show that people made fairer offers in the ultimatum and impunity games when interacting via agent representatives, when compared to direct interaction; in Experiment 2, participants were less likely to accept unfair offers in these games when agent representatives were involved; in Experiment 3, we show that the act of thinking about the decisions ahead of time—i.e., under the so-called “strategy method”—can also lead to increased fairness, even when no agents are involved; and, finally, in Experiment 4 we show that participants were less likely to reach an agreement with unfair counterparts in a negotiation setting. We discuss theoretical implications for our understanding of the nature of people’s social behavior with agent representatives, as well as practical implications for the design of agents that have the potential to increase fairness in society.     

Effects of ambient illuminance and electronic displays on users' visual performance for young and elderly users

Abstract— This study investigated the effects of age (young participants aged 20–30 and elderly participants older than 60), ambient illuminance (50, 500, 1 500, 3000, 6000, and 9000 lx) on participants' visual performance when using various electronic displays (conventional transmissive LCD, Ch‐LC display, and E‐ink display). Overall, the results indicated that participants' visual performance showed sign if icant difference among various ambient il luminance cond itions, electronic displays, and between two groups of participants. Significantly different visual performance was observed between young and elderly participants. The interaction among ambient illuminance, type of electronic display, and age had a significant effect on participants' visual performance. When participants used the E‐ink display, both young and elderly participants had significantly worse visual performance under an ambient illuminance of 50 lx compared to other illuminance settings. When participants used the Ch‐LC display, young participants had significantly worse visual performance under an ambient illuminance of 50 lx compared to other illuminance settings, but elderly participants had significantly better visual performance under an ambient illuminance higher than 1500 lx. When young participants used a conventional transmissive LCD, the ambient illuminance had no significant effect on their visual performance. When elderly participants use a conventional transmissive LCD, an illuminance of 50 and 6000 lx seem to have a detrimental effect on their visual performance.     

Impact of information display on worker performance for wood frame wall assembly using AR HMD under different task conditions

Augmented Reality (AR) head-mounted displays (HMDs) provide users with an immersive virtual experience in the real world. The portability of this technology affords various information display options for construction workers that are not possible otherwise. However, the impact of these different information presentation options on human performance should be carefully evaluated before such technology is deployed in the jobsite. In this paper, we describe a research effort examining how different information displays presented via AR HMD influence task performance when assembling three sized wooden wall frame assembly tasks. We asked 18 construction engineering students with framing experience to finish three wood frame assembly tasks (large, medium, and small) using one of the three information displays (AR 3D conformal, AR 2D tag-along, and paper blueprints). The task performance was measured by time of completion and framing errors, which were analyzed and compared among each factor.     

Networked Press Freedom and Social Media: Tracing Historical and Contemporary Forces in Press‐Public Relations

This paper analyzes how mainstream, online news organizations understand press autonomy in their relationships to audiences. I situate the press in terms of neo‐institutional sociology, seeing its autonomy as a distributed, field‐level phenomenon involving “boundary work” among distributed actors. I then trace press‐audience relations through two historical examples (letters to the editor and ombudsmen), showing how the press has historically both separated itself from and relied upon audiences. Examining eight news organizations' social media policies, I analyze the “inside‐out” and “outside‐in” forces through which the press distinguishes itself from audiences, concluding with a discussion of how such guidelines structure the types of control that news organizations have, or might have, as they use social network sites in their news work.     

Effects of ambient illuminance on users' visual performance using various electronic displays

Abstract— This study investigated the effect of ambient illuminance (50, 500, 1500, 3000, 6000, and 9000 lx) on participants' visual performance using four electronic displays (conventional LCD under transmissive mode, conventional LCD under reflective mode, Ch‐LC display, and E‐ink display). Analysis results indicate that participants' visual performance shows significant difference under various ambient illuminance and electronic displays. The interaction between ambient illuminance and electronic display also has significant effect on participants' visual performance. When participants use the conventional LCD under transmissive mode, ambient luminance has no significant effect on participants' visual performance. However, participants' visual performance is significantly different under various ambient illuminances using the conventional LCD under reflective mode, Ch‐LC display, and E‐ink display. The conventional LCD under transmissive mode is the only choice at a lower illuminance of 50 lx. Higher illuminance (500 lx) for E‐ink displays may result in as good a performance as the conventional LCD under transmissive mode; nevertheless, much higher illuminance (1500 lx) for the conventional LCD under reflectance mode and Ch‐LC display may achieve better performance.     

Gestures that people can understand and use

Recent advances in computing devices push researchers to envision new interaction modalities that go beyond traditional mouse and keyboard input. Typical examples are large displays for which researchers hope to create more “natural” means of interaction by using human gestures and body movements as input. In this article, we reflect about this goal of designing gestures that people can easily understand and use and how designers of gestural interaction can capitalize on the experience of 30 years of research on visual languages to achieve it. Concretely, we argue that gestures can be regarded as “visual expressions to convey meaning” and thus are a visual language. Based on what we have learned from visual language research in the past, we then explain why the design of a generic gesture set or language that spans many applications and devices is likely to fail. We also discuss why we recommend using gestural manipulations that enable users to directly manipulate on-screen objects instead of issuing commands with symbolic gestures whose meaning varies among different users, contexts, and cultures.     

Motion–Display Gain: A New Control–Display Mapping Reflecting Natural Human Pointing Gesture to Enhance Interaction with Large Displays at a Distance

The use of large displays is becoming increasingly prevalent, but development of the usability of three-dimensional (3D) interaction with large displays is still in the early stage. One way to improve the usability of 3D interaction is to develop appropriate control–display (CD) gain function. Nevertheless, unlike in desktop environments, the effects of the relationship between control space and display space in 3D interaction have not been investigated. Moreover, 3D interaction with large displays is natural and intuitive similar to how we work in the physical world. Therefore, a CD gain function that considers human behavior might improve the usability of interaction with large displays. The first experiment was conducted to identify the characteristics of user’s natural hand motion and the user perception of target in distal pointing. Thirty people participated and the characteristics of users’ natural hand movements and the 3D coordinates of their pointing positions were derived. These characteristics were considered in development of motion–display (MD) gain which is a new position-to-position CD mapping. Then, MD gain was experimentally verified by comparing it with Laser pointing, which is currently the best existing CD mapping technique; 30 people participated. MD gain was superior to the existing pointing technique in terms of both performance and subjective satisfaction. MD gain can also be personalized for further improvement. This is an initial attempt to reflect natural human pointing gesture in distal pointing technique, and the developed technique (MD gain) was experimentally proved to be superior to the existing techniques. This achievement is worthy because even a marginal improvement in the performance of pointing task, which is a fundamental and frequent task, can have a large effect on users’ productivity. These results can be used as a resource to understand the characteristics of user’s natural hand movement, and MD gain can be directly applied to situations in which distal pointing is needed, such as interacting with smart TVs or with wall displays. Furthermore, the concept that maps natural human behavior in motor space and an object in visual space can be applied to any interactive system.     

Eliciting situated feedback: A comparison of paper,web forms and public displays

Interactive displays are becoming an increasingly popular civic engagement mechanism for collecting user feedback in urban settings. However, to date no study has investigated (i) how the situatedness of public displays affects the quantity and quality of collected feedback, and (ii) how public displays compare to traditional paper or web feedback mechanisms. We answer these research questions in a series of lab and field studies. We demonstrate that because people tend to approach this technology with no specific purpose in mind, the feedback collected with public displays is noisier than web and paper forms. However, we also show that public displays attract much more feedback than web and paper forms, and generate much more interest. Furthermore, we found that users appropriated our technology beyond its original purpose. Our analysis provides implications on the tradeoffs of using public displays as a feedback mechanism, and we discuss ways of improving the collected feedback using public displays.     

How social distance shapes human–robot interaction

This paper investigates how social distance can serve as a lens through which we can understand human–robot relationships and develop guidelines for robot design. In two studies, we examine the effects of distance based on physical proximity (proxemic distance), organizational status (power distance), and task structure (task distance) on people׳s experiences with and perceptions of a humanlike robot. In Study 1, participants (n=32) played a card-matching game with a humanlike robot. We manipulated the power distance (supervisor vs. subordinate) and proxemic distance (close vs. distant) between participants and the robot. Participants who interacted with the supervisor robot reported a more positive user experience when the robot was close than when the robot was distant, while interactions with the subordinate robot resulted in a more positive experience when the robot was distant than when the robot was close. In Study 2, participants (n=32) played the game in two different task distances (cooperation vs. competition) and proxemic distances (close vs. distant). Participants who cooperated with the robot reported a more positive experience when the robot was distant than when it was close. In contrast, competing with the robot resulted in a more positive experience when it was close than when the robot was distant. The findings from the two studies highlight the importance of consistency between the status and proxemic behaviors of the robot and of task interdependency in fostering cooperation between the robot and its users. This work also demonstrates how social distance may guide efforts toward a better understanding of human–robot interaction and the development of effective design guidelines.