The history and future of direct manipulation

Abstract

The earliest interactive computer systems were based on a conversational mode of interaction in which user and computer communicated through the exchange of linguistic utterances. Since the advent of ‘direct manipulation’ technology there has been a tendency to develop and promote an alternative mode of interaction, based on the user's manipulation of computer-displayed objects. This paper reviews recent developments in the implementation and understanding of direct manipulation interfaces. These point to various limitations in manipulative interaction which might be overcome through the selective reintroduction of conversational interaction. A new philosophy of graceful interaction is suggested to accommodate these developments in which directness is said to be a property of both action and language based systems. A number of practical guidelines are offered to reduce the incidence of clumsy manipulation.     

The history and future of direct manipulation

The earliest interactive computer systems were based on a conversational mode of interaction in which user and computer communicated through the exchange of linguistic utterances. Since the advent of 'direct manipulation' technology there has been a tendency to develop and promote an alternative mode of interaction, based on the user's manipulation of computer-displayed objects. This paper reviews recent developments in the implementation and understanding of direct manipulation interfaces. These point to various limitations in manipulative interaction which might be overcome through the selective reintroduction of conversational interaction. A new philosophy of graceful interaction is suggested to accommodate these developments in which directness is said to be a property of both action and language based systems. A number of practical guidelines are offered to reduce the incidence of clumsy manipulation.     

Put: language-based interactive manipulation of objects

Our approach to scene generation capitalizes the expressive power of natural language by separating its aptness in specifying spatial relations from the difficulties of understanding text. We are implementing an object-placement system called Put that uses a combination of linguistic commands and direct manipulation. The system is language-based, meaning that its design and structure are guided by natural language. Our approach (inspired by research in cognitive linguistics) is to analyze the natural use of spatial relations, define a well-understood class of fundamental relationships, and gradually build a coherent and natural spatial-manipulation system. Just a few simple spatial relationships, such as in, on, and at, parameterized by a limited number of environmental variables can provide comfortable object manipulation. These natural commands can be used to quickly prototype a complex scene and constrain object placement. We believe that we have an extensible, predictable, and computationally feasible environment for object manipulation. We have focused first on spatial relationships because they are fundamental to many conceptual domains beyond object placement, including motion and time. These particular domains are very important to areas of computer graphics such as animation. Uses of spatial relationships in these areas can be quite complex. We briefly introduce the complexities of understanding spatial relations and summarize related work. Then we describe the core of the Put placement system, followed by its linguistic, procedural, and interactive interfaces. We conclude by discussing future enhancements to the system     

The transformation-driven architecture for interactive systems

We propose new software architecture for interactive systems (systems with interacting components). This architecture, called the Transformation-Driven Architecture, TDA, uses certain ideas of the Model-Driven Architecture, MDA. However, unlike MDA, which uses models and model transformations at software development time, TDA uses them at runtime. To describe the dynamics of the system as well as interaction between the components, TDA uses special objects called events and commands.     

Virtual and interactive environments for work of the future

Virtual reality (VR) systems and the virtual environments (VEs) experienced within them have presented challenges to human computer interaction over many years. The sheer range of different interfaces which might be experienced and of different behaviours which might be exhibited have caused difficulties in general understanding of participants’ performance within VR/VE and in providing coherent guidance for designers. We have recently completed a European Information Society Technologies (IST) project, Virtual and Interactive Environments for Workplaces of the Future (VIEW of the Future), which has made great strides in developing improved VR systems and interaction concepts and devices, based upon good understanding of participation in VEs. Particular emphasis in the VIEW of the Future project has been upon mobility and multiple active collaboration in use of VR/VE. This paper introduces a special issue devoted to this project and overviews the project as a whole. In doing so it also reviews some of the human factors issues defined for VR/VE over the years and the contribution of VIEW of the Future to addressing these.     

The role of excursions in interactive systems

It is recommended that man—machine dialogue systems be designed so as to make them more self-explanatory, and hence more user-friendly, by the inclusion of a dialogue form called an excursion tour. This is defined as an information-gathering sequence of operations that enables the user to learn the commands which change his data sets, and which are called direct way commands. The distinction between the excursion and the direct way forms of dialogue is justified in terms of cognitive psychological theory, and corresponds to the distinction made there between planning and performance, or between knowing and doing. The operation of the two dialogue forms and their interaction is represented by an extension of the TOTE model of human task performance. The term interactive deadlock, or unsolvable incongruence, is introduced to denote the situation in which either no command exists that will further an essential subgoal, or if it does exist the user has no way of discovering it within the system. In terms of the TOTE model, it is shown how such deadlocks or incongruences that are unsolvable on one level may be overcome by extending the user's knowledge with the help of excursions.     

The future of systems research

After 20 years in academia and the Silicon Valley, the new Provost of Stanford University calls for a shift in focus for systems research. Performance-long the centerpiece-needs to share the spotlight with availability, maintainability, and other qualities. Although performance increases over the past 15 years have been truly amazing, it will be hard to continue these trends by sticking to the basically evolutionary path that the research community is currently on. The author advocates a more evolutionary approach to systems problems and thinks the approach needs to be more integrated. Researchers need to think about hardware and software as a continuum, not as separate parts. He sees society on the threshold of a “post PC” era, in which computing is ubiquitous, and everyone will use information services and everyday utilities. When everyone starts using these systems, guess what? They expect them to work and to be easy to use. So this era will drive system design toward greater availability, maintainability, and scalability, which will require a real refocusing of current research directions     

The influence of design decisions on the usability of direct manipulation user interfaces

It is assumed that the usability of direct manipulation user interfaces is influenced by a number of design aspects. In this experimental study, the order of command specification and the type of function activation were manipulated in a 2 × 2 factorial design, in order to test hypotheses H1, that object-function specification contributes more to usability than function-object specification; and H2. that the type of function activation (clicking vs. dragging) will influence the usability of direct manipulation user interfaces. Sixty-four subjects, balanced by sex, without computer experience, were assigned randomly to the four experimental conditions. The dependent variables include performance data such as time, efficiency and error rates (logfile-recording), and subjective user rating of the user interface (questionnaire). Whereas HI had to be rejected in this general form, a more elaborated analysis showed significant differences between the factor levels in terms of performance time and syntactically correct actions. Furthermore, the results of the study demonstrated evidence for H2.     

Cost-benefit analysis of interactive systems

This paper assess the state-of-the-art in cost-benefit analyses of interactive systems and suggests an approach for developing improved methodology. Cost-benefit analyses are distinguished from analyses of system performance in that the latter are directed at optimizing system performance at a given level of investment, while the former are directed at justifying the investment itself.Methods of analyzing the performance and costs of computer systems in general and interactive systems in particular are discussed. With this information it is shown how cost-effectiveness analyses may be performed. The next crucial step is to conduct benefit analysis, an ill-defined art. The results of benefit analysis must be combined with cost-effectiveness analysis in order to perform the desired cost-benefit analysis.An experimental methodology is suggested for better performing benefit analyses of interactive systems. A more rigorous formulation of the cost-benefit procedure is then outlined. Two examples of the application this concept are reported.     

The role of user models in cooperative interactive systems

For interactive systems to communicate in a cooperative manner, they must have knowledge about their users. This article explores the role of user models in such systems, with the goal of identifying when and how user models may be useful in a cooperative interactive system. User models are classified by the types of knowledge they contain, several user modelling characteristics that serve as dimension for an additional classification of user models are presented, and user model representations are discussed. These topics help to characterize the space of user modelling in cooperative interactive systems-addressing how they can be used-but do not fully address when it is appropriate to include a user model in an interactive system. Thus, a set of design considerations for user models is presented, while a final example illustrates how these topics influence the user model for a hypothetical investment consulting system.     

The semiotics of configurations for the immanent design of interactive computational systems

In this paper the authors propose a novel semiotic approach to the design of interactive systems and computational systems, grounded in the most recent contributions within the debate around semiotic theory and analysis. This approach, that is here called Semiotics of Configurations (SoC), is proposed for its analytic power in describing material artifacts and settings with a purposely a-conceptualistic stance. The resulting analysis informs a kind of design that is aimed at reproducing and supporting the programs of action detected in the use of artifacts, as this use is “abducted” from the physical and material form of the artifacts themselves and from the observation of how content is transformed within and across them. This approach to design, called immanent design, has inspired a platform for the user-driven development and use of electronic documents and forms in cooperative and organizational domains. The framework is illustrated with a case drawn from a study performed in the domain of hospital work.     

Preferred mental models for direct manipulation and command-based interfaces

One of the main methods used to compare direct manipulation and command-based interfaces is to examine user preferences. User preferences are extended in our research to examine which mental model, direct manipulation or command-based, subjects prefer to use in transfer situations. One group of subjects was provided with both a command-line and direct manipulation interface during a training phase. After training, several transfer experiments were conducted to determine the preferred mental model. The developed mental models were investigated by "running" the models and extending the models to new situations. They were evaluated by determining the operators specified by the subjects. By comparing the class of operators to subjects trained on only one of the interfaces, direct manipulation or command-based, the preferred model could be determined. The preferred mental model for "running" the model was the direct manipulation. For extensions that had a concrete or graphical basis in the interface, the direct manipulation was preferred. For extensions that were abstract, one model was not preferred over the other. Some ability to use multiple mental models, based upon the task, was also observed.     

The complexity of the max word problem and the power of one-way interactive proof systems

We study the complexity of the max word problem for matrices, a variation of the well-known word problem for matrices. We show that the problem is NP-complete, and cannot be approximated within any constant factor, unless P=NP. We describe applications of this result to probabilistic finite state automata, rational series andk-regular sequences. Our proof is novel in that it employs the theory of interactive proof systems, rather than a standard reduction argument. As another consequence of our results, we characterize NP exactly in terms ofone-way interactive proof systems.