Air pointing: Design and evaluation of spatial target acquisition with and without visual feedback

Sensing technologies such as inertia tracking and computer vision enable spatial interactions where users make selections by ‘air pointing’: moving a limb, finger, or device to a specific spatial region. In addition of expanding the vocabulary of possible interactions available, air pointing brings the potential benefit of enabling ‘eyes-free’ interactions, where users rely on proprioception and kinaesthesia rather than vision. This paper explores the design space for air pointing interactions, and presents tangible results in the form of a framework that helps designers understand input dimensions and resulting interaction qualities. The framework provides a set of fundamental concepts that aid in thinking about the air pointing domain, in characterizing and comparing existing solutions, and in evaluating novel techniques. We carry out an initial investigation to demonstrate the concepts of the framework by designing and comparing three air pointing techniques: one based on small angular ‘raycasting’ movements, one on large movements across a 2D plane, and one on movements in a 3D volume. Results show that large movements on the 2D plane are both rapid (selection times under 1 s) and accurate, even without visual feedback. Raycasting is rapid but inaccurate, and the 3D volume is expressive but slow, inaccurate, and effortful. Many other findings emerge, such as selection point ‘drift’ in the absence of feedback. These results and the organising framework provide a foundation for innovation and understanding of air pointing interaction.     

A gesture-free geometric approach for mid-air expression of design intent in 3D virtual pottery

The advent of depth cameras has enabled mid-air interactions for shape modeling with bare hands. Typically, these interactions employ a finite set of pre-defined hand gestures to allow users to specify modeling operations in virtual space. However, human interactions in real world shaping processes (such as pottery or sculpting) are complex, iterative, and continuous. In this paper, we show that the expression of user intent in shaping processes can be derived from the geometry of contact between the hand and the manipulated object. Specifically, we describe the design and evaluation of a geometric interaction technique for bare-hand mid-air virtual pottery. We model the shaping of a pot as a gradual and progressive convergence of the pot’s profile to the shape of the user’s hand represented as a point-cloud (PCL). Thus, a user does not need to learn, know, or remember any gestures to interact with our system. Our choice of pottery simplifies the geometric representation, allowing us to systematically study how users use their hands and fingers to express the intent of deformation during a shaping process. Our evaluations demonstrate that it is possible to enable users to express their intent for shape deformation without the need for a fixed set of gestures for clutching and deforming a shape.     

Empirical Research in Mid-Air Interaction: A Systematic Review

ABSTRACT

Mid-air interaction is a distinct style of natural HCI (Human-Computer Interaction). In mid-air interaction, users make use of their whole body—with a strong focus on hands—and apply gestures, postures, and movements to interact with digital content on distant displays or remote devices. Although the idea of exploiting body movements and gestures in HCI is not new, mid-air interaction is now possible due to the availability of reasonably priced depth cameras and sensors. This article presents a systematic review of empirical research in mid-air interaction, based on a corpus of 104 publications from 2011 to 2018. The review includes: (a) a retrospective on mid-air interaction, which outlines its historical development and clarifications on important concepts; (b) current application domains of mid-air interaction; (c) user requirements methods, focusing on gesture elicitation studies; (d) dimensions of prototyping and design; (e) empirical evaluation methods and issues; (f) a discussion on several trends and challenges for further research and development.     

Design and evaluation of freehand menu selection interfaces using tilt and pinch gestures

Freehand gestural interaction, in which the user’s hands move in mid-air to provide input, has been of interest to researchers, but freehand menu selection interfaces have been under-investigated so far. Freehand menu selection is inherently difficult, especially with increasing menu breadth (i.e., the number of items), largely because moving hands in free space cannot achieve precision as high as physical input devices such as mouse and stylus. We have designed a novel menu selection interface called the rapMenu (Ni et al., 2008), which is controlled by wrist tilt and multiple pinch gestures, and takes advantage of the multiple discrete gesture inputs to reduce the required precision of the user hand movements.In this article, we first review the visual design and behavior of the rapMenu technique, as well as related design issues and its potential advantages. In the second part, we present two studies of the rapMenu in order to further investigate the strengths and limitations of the design principle. In the first study, we compared the rapMenu to the extensively studied tilt menu technique (Rahman et al., 2009). Our results revealed that the rapMenu outperforms the tilt menu as menu breadth increases. In the second study, we investigated how the rapMenu affords the opportunity of eyes-free selection and users’ transition from novice to expert. We found that within 10 min of practice, eyes-free selection with rapMenu has competitive speed and accuracy with the visual rapMenu and the tilt menu. Finally, we discuss design variations that use other axes of wrist movement and adopt alternative auditory feedback.     

General support vector representation machine for one-class classification of non-stationary classes

Novelty detection, also referred to as one-class classification, is the process of detecting ‘abnormal’ behavior in a system by learning the ‘normal’ behavior. Novelty detection has been of particular interest to researchers in domains where it is difficult or expensive to find examples of abnormal behavior (such as in medical/equipment diagnosis and IT network surveillance). Effective representation of normal data is of primary interest in pursuing one-class classification. While the literature offers several methods for one-class classification, very few methods can support representation of non-stationary classes without making stringent assumptions about the class distribution. This paper proposes a one-class classification method for non-stationary classes using a modified support vector machine and an efficient online version for reducing computational time. The presented method is applied to several simulated datasets and actual data from a drilling machine. In addition, we present comparison results with other methods that demonstrate its superior performance.     

Effects of User’s Hand Orientation and Spatial Movements on Free Hand Interactions with Large Displays

In motion-sensing interaction with large displays through bare hands, we can observe that users alternate their hands and move their bodies frequently. What cause such actions and how these actions affect free hand interaction results are less systematically investigated. To address this gap in knowledge, we conducted studies on Pointer-Acceleration (PA)-based free hand interactions of target selection and found that (1) users made more frequent hand alternations when selecting small targets with large movement amplitudes, as in such cases users were not only affected by observable arm fatigue, but were also motivated to switch hands for higher selection accuracy and convenience; (2) hand alternation led to the hand orientation effects: target selection on display areas at the operating hand’s side was more efficient and accurate than that at the opposite side; (3) large movement amplitudes on the user interface increased users’ physical movements in front of the large display, which harmed selection efficiency; (4) selection of small targets led to a closer interaction distance, while large movement amplitudes led to a larger interaction distance; and (5) selection results were affected by interaction distances, as users gained high efficiency, but low accuracy at a large distance and low efficiency, but high accuracy at a close distance. Given these results, this article discusses practical implications for applying PA-based free hand interaction techniques and designing related user interfaces on large displays.     

CUDA-based real-time hand gesture interaction and visualization for CT volume dataset using leap motion

Touchless interaction has received considerable attention in recent years with benefit of removing barriers of physical contact. Several approaches are available to achieve mid-air interactions. However, most of these techniques cause discomfort when the interaction method is not direct manipulation. In this paper, gestures based on unimanual and bimanual interactions with different tools for exploring CT volume dataset are designed to perform the similar tasks in realistic applications. Focus + context approach based on GPU volume ray casting by trapezoid-shaped transfer function is used for visualization and the level-of-detail technique is adopted for accelerating interactive rendering. Comparing the effectiveness and intuitiveness of interaction approach with others by experiments, ours has a better performance and superiority with less completion time. Moreover, the bimanual interaction with more advantages is timesaving when performing continuous exploration task.     

SpinRadar: a spontaneous service provision middleware for place-aware social interactions

With the advancements of mobile phones and the integration of multiple communication interfaces, online social interaction between users is no longer restricted to a specific place with connectivity to the Internet but can happen anywhere and at any time. This has promoted the development of mobile social applications to enable opportunistic interactions with co-located users. One of the challenging problems in such interactions is to discover interaction opportunities with nearby users. Existing works focus on properties related to mobile users in order to find similar users in the surrounding area; these works depend on predefined logic such as conditional statements to recommend spontaneous social interaction opportunities. However, the social implications of the place in which the interaction is taking place are an important factor for recommendations, as those implications provide hints about the most plausible types of interactions among co-located users. In this work, we present a middleware called SpinRadar which is designed to support spontaneous interactions between co-located users by taking into account the semantics of a place, which we call ‘placeness.’ Our evaluation shows that the proposed scheme satisfies users much more than existing schemes.     

Real-time construction worker posture analysis for ergonomics training

Construction activities performed by workers are usually repetitive and physically demanding. Execution of such tasks in awkward postures can strain their body parts and can result in fatigue, injuries or in severe cases permanent disabilities. In view of this, it is essential to train workers, before the commencement of any construction activity. Furthermore, traditional worker monitoring methods are tedious, inefficient and are carried out manually whereas, an automated approach, apart from monitoring, can yield valuable information concerning work-related behavior of worker that can be beneficial for worker training in a virtual reality world. Our research work focuses on developing an automated approach for posture estimation and classification using a range camera for posture analysis and categorizing it as ergonomic or non-ergonomic. Using a range camera, first we classify worker’s pose to determine whether a worker is ‘standing’, ‘bending’, ‘sitting’, or ‘crawling’ and then estimate the posture of the worker using OpenNI middleware to get the body joint angles and spatial locations. A predefined set of rules is then formulated to use this body posture information to categorize tasks as ergonomic or non-ergonomic.     

A study of mutual information based feature selection for case based reasoning in software cost estimation

Software cost estimation is one of the most crucial activities in software development process. In the past decades, many methods have been proposed for cost estimation. Case based reasoning (CBR) is one of these techniques. Feature selection is an important preprocessing stage of case based reasoning. Most existing feature selection methods of case based reasoning are ‘wrappers’ which can usually yield high fitting accuracy at the cost of high computational complexity and low explanation of the selected features. In our study, the mutual information based feature selection (MICBR) is proposed. This approach hybrids both ‘wrapper’ and ‘filter’ mechanism which is another kind of feature selector with much lower complexity than wrappers, and the features selected by filters are likely to be generalized to other conditions. The MICBR is then compared with popular feature selectors and the published works. The results show that the MICBR is an effective feature selector for case based reasoning by overcoming some of the limitations and computational complexities of other feature selection techniques in the field.     

基于驾驶员面部时序数据的疲劳驾驶检测算法

针对传统疲劳驾驶检测方法识别准确率低、泛化能力差的问题,提出了一种基于CNNs和LSTM的端到端可训练网络,检测驾驶员的疲劳状态。根据驾驶员面部特征点提取ROI,将在其他计算机视觉任务上表现较好的深度网络迁移到疲劳检测任务中,并结合LSTM处理时序数据的能力,提出一种新的疲劳检测网络,该网络能够读入视频流中的时序数据并检测出驾驶员的疲劳状态。实验证明所提方法和模型在公开数据集中具有较高的识别准确率,并且在不同的数据集间具有很好的泛化能力,对于减少路面车祸、保障人身安全具有很重要的意义。     

Communication Spaces

Concepts of space are fundamental to our understanding of human action and interaction. The common sense concept of uniform, metric, physical space is inadequate for design. It fails to capture features of social norms and practices that can be critical to the success of a technology. The concept of ‘place’ addresses these limitations by taking account of the different ways a space may be understood and used. This paper argues for the importance of a third concept: communication space. Motivated by Heidegger’s discussion of ‘being-with’ this concept addresses differences in interpersonal ‘closeness’ or mutual-involvement that are a constitutive feature of human interaction. We apply the concepts of space, place and communication space to the analysis of a corpus of interactions from an online community, ‘Walford’, which has a rich communicative ecology. A novel measure of sequential integration of conversational turns is proposed as an index of mutal-involvement. We demonstrate systematic differences in mutual-involvement that cannot be accounted for in terms of space or place and conclude that a concept of communication space is needed to address the organisation of human encounters in this community.     

Assisting Visually Impaired People to Acquire Targets on a Large Wall-Mounted Display

Large displays have become ubiquitous in our everyday lives, but these displays are designed for sighted people.This paper addresses the need for visually impaired people to access targets on large wall-mounted displays. We developed an assistive interface which exploits mid-air gesture input and haptic feedback, and examined its potential for pointing and steering tasks in human computer interaction（HCI）. In two experiments, blind and blindfolded users performed target acquisition tasks using mid-air gestures and two different kinds of feedback（i.e., haptic feedback and audio feedback）. Our results show that participants perform faster in Fitts＇ law pointing tasks using the haptic feedback interface rather than the audio feedback interface. Furthermore, a regression analysis between movement time（MT） and the index of difficulty（ID）demonstrates that the Fitts＇ law model and the steering law model are both effective for the evaluation of assistive interfaces for the blind. Our work and findings will serve as an initial step to assist visually impaired people to easily access required information on large public displays using haptic interfaces.     

Study on interaction-induced symptoms with respect to virtual grasping and manipulation

Owing to the popularity of various hand tracking interfaces, there have been numerous applications developed to provide intuitive hand interaction with the virtual world. As users start with great anticipation, they end up with dissatisfaction due to difficulties of manipulation or physical tiredness coming very short. Although the task itself is rather trivial in a real life situation, it requires much effort in the virtual environment. We address this awkwardness as ‘VR interaction-induced fatigue symptom’ and hypothesize its causes based on our observations. We argue that the source of the fatigue comes from the restricted sensory information of the VR interfaces, and that users try to accommodate the missing sensory feedback by excessive motion leading to wrong posture or bad timing. We demonstrate our hypothesis by conducting experiments of two types of virtual interaction scenarios: object transport and 3D selection. Furthermore, by analyzing the behaviors of users' action collected from our experiment, we derive essential factors to be considered in designing VR applications, and propose a conceptual interaction model for orchestrating virtual grasping.     

Commitments and interaction norms in organisations

In an organisational setting such as an online marketplace, an entity called the ‘organisation’ or ‘institution’ defines interaction protocols, monitors agent interaction, and intervenes to enforce the interaction protocols. The organisation might be a software system that thus regulates the marketplace, for example. In this article we abstract over application-specific protocols and consider commitment lifecycles as generic interaction protocols. We model interaction protocols by explicitly-represented norms, such that we can operationalise the enforcement of protocols by means of norm enforcement, and we can analyse the protocols by a logical analysis of the norms. We adopt insights and methods from commitment-based approaches to agent interaction as well as from norm-based approaches to agent behaviour governance. First, we show how to use explicitly-represented norms to model commitment dynamics (lifecycles). Second, we introduce an operational semantics to operationalise norm enforcement. Third, we show how to logically analyse interaction protocols by means of commitment dynamics and norm enforcement. The model, semantics, and analysis are illustrated by a running example from a vehicle insurance domain.     

基于Android手机的健康调理手环设计

针对人体日常行为习惯做出健康调理建议,设计了基于Android智能手机的健康调理手环系统,采用包含人体测量手环和Android智能手机应用软件的系统结构;对人体测量手环的蓝牙模块、加速度传感器、心率传感器、体表温度传感器等硬件电路和软件流程算法进行设计,重点研究并提出了人体卡路里数综合消耗估计方法,并对Android操作系统的应用软件健康调理建议APP进行设计与实现。实验结果显示,系统能够有效实时测量人体数据,估算人体卡路里消耗,给出人体健康调理食谱和饮食建议,实现设计效果。     

基于动机模型的自主性虚拟人行为选择研究

自主性虚拟人的研究是人工生命和计算机动画交叉融合的新领域.但是由于人的心理活动是一个整体的过程,动机、感知等这些参数都是互有联系和影响的,目前的研究仍然只是局部和有限的.借鉴马斯洛理论,基于动机模型框架提出一种简化的抑制和疲劳模型控制的虚拟人自主行为选择机制,实验结果表明,该方法较好地解决了在资源有限的动态虚拟环境中,虚拟人如何在多个相互抑制的行为之间对行为进行仲裁和选择.经实验证明,该研究可以应用于智能交互领域.     

Innovation as Improvisation ‘In The Shadow’

In this article, we explore processes of innovation – which are inherently uncertain – from a complexity perspective, in which they are understood as new patterns of experiences as they emerge in human conversational interaction. We reflect on local interactions between people involved in emerging processes of innovation, with a particular emphasis on the improvisational nature of interaction. Through an abductive approach, by iterating actual experiences and our understanding of them, we show that such processes are collective efforts that take place as informal, highly improvised conversations — happening ‘below the radar’ — which may unpredictably offer windows of opportunity to enable change. We show that innovation often emerges as ‘shadow themes’, experienced as subversive by those involved in the moments of interaction. While these themes are embedded in informal conversations and processes, they can be induced by invitations – conscious or unconscious moves that encourage those involved to make spontaneous moves together in a mutually improvised context. Our experience shows that the emergence of shadow themes can have a long‐term impact on the organization and the people involved, and that managers may be ‘in charge but not in control’ of such innovation processes.     

Decentring the Creative Self: How Others Make Creativity Possible in Creative Professional Fields

Since its inception, the psychology of creativity has been concerned primarily with the study of individual creators. In contrast, this research is dedicated to an exploration of (a) who has a significant impact on a creative professional's activity and (b) what the contribution is that others make to creative outcomes. The research included interviews with 60 professionals working in science and creative industries in France. The following categories of others emerged: family and friends, peers and students, clients and funders, critics and gatekeepers, and the general public – and they were related to themes depicting the interaction between these different others and the creator. Findings reveal both similarities and differences across the five domains in terms of the specific contribution of others to the creative process. Social interactions play a key formative, regulatory, motivational and informational role in relation to creative work. From ‘internalized’ to ‘distant’, other people are an integral part of the equation of creativity calling for a de‐centring of the creative self and its re‐centring in a social space of actions and interactions.