The history and future of direct manipulation

The earliest interactive computer systems were based on a conversational mode of interaction in which user and computer communicated through the exchange of linguistic utterances. Since the advent of 'direct manipulation' technology there has been a tendency to develop and promote an alternative mode of interaction, based on the user's manipulation of computer-displayed objects. This paper reviews recent developments in the implementation and understanding of direct manipulation interfaces. These point to various limitations in manipulative interaction which might be overcome through the selective reintroduction of conversational interaction. A new philosophy of graceful interaction is suggested to accommodate these developments in which directness is said to be a property of both action and language based systems. A number of practical guidelines are offered to reduce the incidence of clumsy manipulation.     

Combining natural language with direct manipulation: the conceptual framework for a hybrid human-computer interface

Abstract

This paper describes the development of a conceptual model of an interaction system for future human-computer interaction. It is suggested that an integrated system is the most promising solution for diverse users and tasks. A ‘ hybrid interaction system’ is considered, which tries to efficiently direct manipulation, menu selection, and natural language. The mechanism of the natural language interface module is described in more detail. Finally, the allocation of functionalities for generalized tasks on the conceptual model is considered from the taxonomic approach.     

Gaze-based prediction of pen-based virtual interaction tasks

In typical human–computer interaction, users convey their intentions through traditional input devices (e.g. keyboards, mice, joysticks) coupled with standard graphical user interface elements. Recently, pen-based interaction has emerged as a more intuitive alternative to these traditional means. However, existing pen-based systems are limited by the fact that they rely heavily on auxiliary mode switching mechanisms during interaction (e.g. hard or soft modifier keys, buttons, menus). In this paper, we describe how eye gaze movements that naturally occur during pen-based interaction can be used to reduce dependency on explicit mode selection mechanisms in pen-based systems. In particular, we show that a range of virtual manipulation commands, that would otherwise require auxiliary mode switching elements, can be issued with an 88% success rate with the aid of users׳ natural eye gaze behavior during pen-only interaction.     

Direct 3D painting with a metaball-based paintbrush

This paper presents a direct 3D painting algorithm for polygonal models in 3D object-space with a metaball-based paintbrush in virtual environment.The user is allowed to directly manipulate the parameters used to shade the surface of the 3D shape by applying the pigment to its surface with direct 3D manipulation through a 3D flying mouse.     

2维至3维图像/视频转换的深度图提取方法综述

目的 深度图提取是计算机视觉领域的研究热点。随着3D显示设备的普及,2D-3D图像/视频转换的深度图提取研究受到越来越多国内外学者的关注。为此回顾深度图提取研究历程,并对已有成果进行分类、概括和评述。方法 由于深度图提取方法的实现主要依赖于深度线索,不同方法存在人机交互程度上的差异。采用基于深度线索和基于人机交互程度的两种分类方法进行归纳评述。结果 根据深度线索的不同,将深度图提取方法分为基于单目线索的方法、基于双目线索的方法和基于混合线索的3类方法。然后从人机交互的角度,将深度图提取方法分为人工法、半自动法和全自动法。介绍了这些方法的基本思想,比较归纳不同方法的优点与不足。最后,阐述了近年来热门的机器学习方法在深度图提取的应用。结论 对深度图提取研究进行简要的总结和展望。指出深度图提取研究具有从研究热点中挖掘创新思路、引入新的深度线索等发展趋势。     

Exploring educational immersive videogames: an empirical study with a 3D multimodal interaction prototype

ABSTRACT

Gestural interaction devices emerged and originated various studies on multimodal human–computer interaction to improve user experience (UX). However, there is a knowledge gap regarding the use of these devices to enhance learning. We present an exploratory study which analysed the UX with a multimodal immersive videogame prototype, based on a Portuguese historical/cultural episode. Evaluation tests took place in high school environments and public videogaming events. Two users would be present simultaneously in the same virtual reality (VR) environment: one as the helmsman aboard Vasco da Gama’s fifteenth-century Portuguese ship and the other as the mythical Adamastor stone giant at the Cape of Good Hope. The helmsman player wore a VR headset to explore the environment, whereas the giant player used body motion to control the giant, and observed results on a screen, with no headset. This allowed a preliminary characterisation of UX, identifying challenges and potential use of these devices in multi-user virtual learning contexts. We also discuss the combined use of such devices, towards future development of similar systems, and its implications on learning improvement through multimodal human–computer interaction.     

多任务CAD系统的执行中心模型

针对直接操纵的多任务CAD系统的实时反馈、并发对话等需求,提出CAD 系统的基于事件驱动的层次化执行中心模型EDHM。该模型遵循对话独立性原则,将执行中心划分为3个层次5大部件,支持连续语义反馈和异步并发对话,并以统一的框架支持顺序和异步交织的交互风格。此模型已被应用到多任务CAD支撑系统OpenDesign的执行中心构造中。实践表明,基于EDHM模型建立的执行中心满足直接操纵和多进程的需求,具有结构上的独立性,易于移植和封装复用。     

Semantics for consistent activation in context-oriented systems

ContextContext-oriented programming languages provide dedicated programming abstractions to define behavioral adaptations and means to combine those adaptations dynamically according to sensed context changes. Some of these languages feature programming abstractions to explicitly define interaction dependencies among contexts. However, the semantics of context activation and the meaning of dependency relations have been described only informally, which in some cases has led to incorrect specifications, faulty implementations and inconsistent system behavior.ObjectiveWith the aim of avoiding faulty implementations and inconsistencies during system execution, this paper proposes both a formal and run-time model of contexts, context activation and context interaction.MethodAs a formal and computational basis, we introduce context Petri nets, a model based on Petri nets, which we found to match closely the structure of contexts in context-oriented systems. The operational semantics of Petri nets permits the modeling of run-time context activations. Existing Petri net analyses allow us to reason about system properties. As validation, we carried out small and medium-sized case studies.ResultsIn the cases explored, context Petri nets served effectively as underlying run-time model to ensure that declared context interaction constraints remain consistent during context manipulation. Moreover, context Petri nets enabled us to analyze certain properties regarding the activation state of particular contexts.ConclusionContext Petri nets thus proved to be appropriate to encode and manage the semantics of context activation, both formally and computationally, so as to preserve the consistency of context-oriented systems.     

Simplified data structure for analysing mechanisms using character handling techniques

This paper introduces the program SIMPLA¹ for the kinematic analysis of planar linkages. It uses character manipulation subroutines to allow the program to be written with maximum emphasis placed on the user's finding the input data structure easy to understand. Very complex planar linkages may be described using an extremely simple and familiar form of input data. This avoids the usual extended period of acclimatization to the peculiarities of most other linkage programs. The program, which is written in Fortran, includes extensive data checking routines and a self-explanatory conversational feature which assists the user to produce a self-consistent input data set. It has been used successfully on an IBM/370 computer. The methods for writing programs using these character manipulation subroutines are described.     

A theory of personalisation of appearance: quantitative evaluation of qualitatively derived data

Abstract

This paper suggests three motivations for the strong interest in human factors' aspects of user interfaces and reviews five design issues: command language versus menu selection, response time and display rates, wording of system messages, on-line tutorials, explanations and help messages and hardware devices. Five methods and tools for system development are considered: participatory design, specification methods, software implementation tools, pilot studies and acceptance tests and evolutionary refinement based on user feedback.

The final portion of the paper presents direct manipulation, an approach which promises to become widely used in interactive systems. Direct manipulation involves representation of the object of interest, rapid incremental reversible actions and physical action instead of complex syntax.     

Progress in systems ergonomics: a selective review

Abstract

The 1967 conference on the Human Operator in Complex Systems identified several developments required in systems ergonomics. Progress in four areas requiring development is reviewed, based on information from workshops and surveys of applications of systems ergonomics in military systems. It is concluded that there has been less progress than might have been hoped in the use of systems relevant criteria in systems ergonomics. Few applications have involved trade-offs between selection, training and equipment design. Task analysis techniques have not advanced, although their use may be changing. Although there appears to be general agreement on the major activities associated with systems ergonomics, the emphasis placed on those different activities may be changing as a result of technological developments. However, these technological developments are now providing opportunities for systems ergonomists to improve their tools and techniques.     

Clifford and Graßmann Hopf algebras via the BIGEBRA package for Maple

Hopf algebraic structures will replace groups and group representations as the leading paradigm in forthcoming times. K-theory, co-homology, entanglement, statistics, representation categories, quantized or twisted structures as well as more geometric topics of invariant theory, e.g., the Graßmann-Cayley bracket algebra, are all covered by the Hopf algebraic framework. The new branch of experimental mathematics allows one to easily enter these fields through direct calculations using symbolic manipulation and computer algebra system (CAS). We discuss problems which were solved when building the BIGEBRA package for Maple and CLIFFORD to handle tensor products, Graßmann and Clifford algebras, coalgebras and Hopf algebras. Recent results showing the usefulness of CAS for investigating new and involved mathematics provide us with examples. An outlook on further developments is given.     

Semiotics of virtual reality as a communication process

ABSTRACT

The experience of immersive virtual reality (VR) can be considered as a communication process between human beings, mediated by computer systems, which uses visualisation and other sensory stimulation. In this paper, we analyse how VR characteristics can be explored using semiotic theory and, with methods of generative semiotics, we explore aspects of narrative and interaction in VR. We propose a semiotic analysis of VR communication focusing on syntax, semantics and pragmatics and considering also some principles of generative semiotics. The syntactic level is analysed as determined by the characteristics of the visual communication adopted. The semantic of VR is related to the functional model chosen to realise the virtual system. The pragmatic of VR is based on the human–computer interaction that changes the user's role. We explore how these aspects can be characterised in the context of VR communication design and what principles can be adopted for a VR application, and we present an analysis and a classification of the iconic signs that are being used in VR. Moreover, we present a framework that can be used to classify and describe different kinds of virtual reality systems and to better understand communication in VR, and we use it to classify eight popular systems for e-learning and collaboration.     

Two approaches to on-line graphics systems

This article describes two types of software systems which provide an interface to users who wish to apply the power of interactive graphics to their specific problem. The systems are based on an 8K IMLAC PDS-1 minicomputer CRT display which provides sufficient processor power to handle display generation, management, and dynamic manipulation. The minicomputer software was first designed as an independent operating system which not only provides the user with high-speed timesharing access to the host computer, but also the resources of the local processor and its associated input/output devices. In this configuration, host compiled FORTRAN IV programs are loaded into the minicomputer through the timesharing link and executed independently from the host processor. A higher degree of processing is provided by the master-slave relationship between a local DEC PDP $\text{11}\text{40}$ minicomputer and the IMLAC. The user's FORTRAN IV program, executing in the PDP $\text{11}\text{40}$, directs an IMLAC graphics subsystem through subroutine calls. These systems provide the programmer with a foundation for a broad range of high level applications involving computer graphics.     

ON THE STRUCTURE OF DIRECTED GRAPHS WITH APPLICATIONS: A Rapprochement with Systems Theory Part II

Abstract

In Part I of this paper¹, a rapprochement with system theory has been established for the study of digraph structure.

In this Part II we first extend the basic results of Part I to the structure of composite digraphs- Then the decomposition of a digraph according to its internal composition/interaction will be explored using the concept of subsystem interaction. An algorithm is given for closed/irreducible closed subgraph identification and decomposition. Applications to computer systems, Markov chains, and to control and dynamic systems are discussed. A new canonical form for large-scale decentralized control/estimation is proposed.     

Gestures and multimodal input

For users with motion impairments, the standard keyboard and mouse arrangement for computer access often presents problems. Other approaches have to be adopted to overcome this. In this paper, we will describe the development of a prototype multimodal input system based on two gestural input channels. Results from extensive user trials of this system are presented. These trials showed that the physical and cognitive loads on the user can quickly become excessive and detrimental to the interaction. Designers of multimodal input systems need to be aware of this and perform regular user trials to minimize the problem.     

Systematic Review: Trust-Building Factors and Implications for Conversational Agent Design

ABSTRACT

Off-the-shelf conversational agents are permeating people’s everyday lives. In these artificial intelligence devices, trust plays a key role in users’ initial adoption and successful utilization. Factors enhancing trust toward conversational agents include appearances, voice features, and communication styles. Synthesizing such work will be useful in designing evidence-based, trustworthy conversational agents appropriate for various contexts. We conducted a systematic review of the experimental studies that investigated the effect of conversational agents’ and users’ characteristics on trust. From a full-text review of 29 articles, we identified five agent design-themes affecting trust toward conversational agents: social intelligence of the agent, voice characteristics and communication style, look of the agent, non-verbal communication, and performance quality. We also found that participants’ demographic, personality, or use context moderate the effect of these themes. We discuss implications for designing trustworthy conversational agents and responsibilities around on stereotypes and social norm building through agent design.     

Conversation with and through computers

People design what they say specifically for their conversational partners, and they adapt to their partners over the course of a conversation. A comparison of keyboard conversations involving a simulated computer partner (as in a natural language interface) with those involving a human partner (as in teleconferencing) yielded striking differences and some equally striking similarities. For instance, there were significantly fewer acknowledgments in human/computer dialogue than in human/human. However, regardless of the conversational partner, people expected connectedness across conversational turns. In addition, the style of a partner's response shaped what people subsequently typed. These results suggest some issues that need to be addressed before a natural language computer interface will be able to hold up its end of a conversation.