Evaluation of user interfaces: EVADIS II— a comprehensive evaluation approach

Abstract

As a result of the importance of the usability approach in system development and the EC's ‘Directive concerning the minimum safety and health requirements for VDT workers’ (EWG 1990), there is an accepted need for practical evaluation methods for user interfaces. The usability approach and the EC Directive are not restricted to user interface design, as they include the design of appropriate hardware and software, as well as organization, job, and task design. Therefore system designers are faced with many, often conflicting, requirements and need to address the question, ‘How can usability requirements comprehensively be considered and evaluated in system development?’ Customers buying hardware and software and introducing them into their organization ask, (How can I select easy-to-use hardware and software?’ Both designers and customers need an evaluation procedure that covers all the organizational, user, hard- and software requirements. The evaluation method, EVADIS.II, we present in this paper overcomes characteristic deficiencies of previous evaluation methods. In particular, it takes the tasks, the user, and the organizational context into consideration during the evaluation process, and provides computer support for the use of the evaluation procedure.     

Systematic evaluation of design choices for software development tools

Most design and evaluation of software tools is based on the intuition and experience of the designers. Software tool designers consider themselves typical users of the tools that they build and tend to subjectively evaluate their products rather than objectively evaluate them using established usability methods. This subjective approach is inadequate if the quality of software tools is to improve and the use of more systematic methods is advocated. This paper summarises a sequence of studies that show how user interface design choices for software development tools can be evaluated using established usability engineering techniques. The techniques used included guideline review, predictive modelling and experimental studies with users.     

Training software developers and designers to conduct usability evaluations

Many efforts to improve the interplay between usability evaluation and software development rely either on better methods for conducting usability evaluations or on better formats for presenting evaluation results in ways that are useful for software designers and developers. Both of these approaches depend on a complete division of work between developers and evaluators. This article takes a different approach by exploring whether software developers and designers can be trained to conduct their own usability evaluations. The article is based on an empirical study where 36 teams with a total of 234 first-year university students on software development and design educations were trained through an introductory course in user-based website usability testing that was taught in 40 h. They used the techniques from this course for planning, conducting, and interpreting the results of a usability evaluation of an interactive website. They gained good competence in conducting the evaluation, defining user tasks and producing a usability report, while they were less successful in acquiring skills for identifying and describing usability problems.     

The scenario‐based usability checklist development for home appliance design: A case study

In the past decade, home appliances have been rapidly developed to satisfy the various requirements of users. Thus, there are increases in requirements to explore appropriate evaluation methods that reflect the usability of home appliances quickly and comprehensively. This study aims to develop a scenario‐based usability checklist for product designers early in the design process. In this study, the scenario‐based usability checklist consists of two parts: 1) a heuristic evaluation checklist, and 2) a scenario evaluation checklist. In heuristic evaluation checklist development, usability factors of home appliances are extracted and then coupled to user interface (UI) elements. In scenario evaluation checklist development, scenarios are developed through brainstorming by focus group interview (FGI), and then evaluation elements are extracted from previous scenarios analysis. The proposed scenario‐based usability checklists can enable designers to evaluate product design quickly and comprehensively early in the development process with users' viewpoints. © 2010 Wiley Periodicals, Inc.     

Using task analysis to improve usability of fatigue modelling software

The design of any interactive computer system requires consideration of both humans and machines. Software usability is one aspect of human–computer interaction that can benefit from knowledge of the user and their tasks. One set of methods for determining whether an application enables users to achieve their predetermined goals effectively and efficiently is task analysis. In the present study, a task analysis was applied to the graphical user interface of fatigue modelling software used in industry. The task analysis procedure allowed areas of usability improvement to be identified and then addressed with alternate interface prototypes. The present method of task analysis illustrates a practical and efficient way for software designers to improve software usability, user effectiveness and satisfaction, by involving users in the design process.     

Model-Driven Interactive System Design for Therapy Robots

Physiotherapy using intelligent robots is emerging as a new approach to recovery for many stroke patients. Although therapy robots have a strong potential in dealing with therapeutic and other medical applications, they have not been fully utilized in everyday therapy activities due to concerns over safety and the lack of friendly robot user/patient interaction models. From the viewpoint of software engineering, a user-centred design based on UML (Unified Modelling Language) has been known to be one of the best solutions to satisfy usability since the design process relies heavily on the analysis of users and their tasks to reach their goals. Therefore, a model-driven approach to interactive system design via UML for therapy robots is needed to make them usable in the real world. This paper proposes such approach and introduces a new graphical notation that describes user interface elements and the methods of connection with hardware/software objects. With the proposed abstract interaction models, prototyping interactive systems can be made faster and allows for their evaluation by users and system developers before implementation in order to improve usability from the perspectives of users and system developers.     

Rethinking the path to usability: how to design what users really want

As the software market grows more competitive, companies with difficult-to-use products face higher customer support costs as they attempt to rework user interfaces to fix usability problems. It is time that software designers tried reversing the direction - get user requirements, design the user interaction with the product's features, and then code the features - to produce software from the user's perspective and logic.     

Exploring the benefits of the combination of a software architecture analysis and a usability evaluation of a mobile application

Designing easy to use mobile applications is a difficult task. In order to optimize the development of a usable mobile application, it is necessary to consider the mobile usage context for the design and the evaluation of the user-system interaction of a mobile application. In our research we designed a method that aligns the inspection method “Software ArchitecTure analysis of Usability Requirements realizatioN” SATURN and a mobile usability evaluation in the form of a user test. We propose to use mobile context factors and thus requirements as a common basis for both inspection and user test. After conducting both analysis and user test, the results described as usability problems are mapped and discussed. The mobile context factors identified define and describe the usage context of a mobile application. We exemplify and apply our approach in a case study. This allows us to show how our method can be used to identify more usability problems than with each method separately. Additionally, we could confirm the validity and identified the severity of usability problems found by both methods. Our work presents how a combination of both methods allows to address usability issues in a more holistic way. We argue that the increased quantity and quality of results can lead to a reduction of the number of iterations required in early stages of an iterative software development process.     

A structured methodology for comparative evaluation of user interface designs using usability criteria and measures

Previous research in software ergonomics has indicated the importance of evaluating the usability of computer user interfaces. This paper presents a quantitative basis for selecting from among multiple alternative interfaces relative to multiple criteria of usability concern. The proposed model consists of two main phases: the prescreening phase and the evaluation phase. The prescreening phase involves expert judgment-based assessment with qualitative criteria. The first phase uses absolute measurement analytic hierarchy process (AHP) to filter possible alternative interfaces to a reasonable subset. The evaluation phase involves user-based assessment such as user testing, with quantitative criteria. The objective of the second phase is to evaluate a subset of alternatives using objective measures and to select the best alternative. A set of criteria and measures for evaluating the usability of computer software designs is presented. The proposed methodology is demonstrated in the interface design of a database system used to analyze trip cases information of nuclear power plant.Relevance to industry

There is currently a focus on the usability of interactive computer software. Software developers, interface designers or human factors engineers often confront the task of comparative evaluation among systems, versions or interface designs. The proposed methodology provides practitioners with a structured approach to select the best interface based on usability criteria and measures.     

Integration of usability evaluation and model-based software development

Model-based software development is carried out as a well defined process. Depending on the applied approach, different phases can be distinguished, e.g. requirements specification, design, prototyping, implementation and usability evaluation. During this iterative process manifold artifacts are developed and modified, including, e.g. models, source code and usability evaluation data. CASE tools support the development stages well, but lack a seamless integration of usability evaluation methods. We aim at bridging the gap between development and usability evaluation, through enabling the cooperative use of artifacts with the particular tools. As a result of integration usability experts save time to prepare an evaluation and evaluation results can be easier incorporated back into the development process. We show exemplary our work on enhancing the Eclipse framework to support usability evaluation for task model-based software development.     

JSD and the design of user interface software

Systems analysis and design methods (SADMs), such as Jackson system development (JSD), have been proposed as a means of improving software quality. This is contrasted with the role of improving software usability proposed for human factors (HF). The paper argues the need for an explicit user interface specification stage in system development to ensure the usability of proposed systems. It is suggested that JSD supplemented by the specialist knowledge of HF can support such a specification stage. For HF and JSD to be successfully integrated, contributions from both need to be appropriately timed and structured. A model of the system development process that incorporates HF contributions is proposed. A means by which the HF contributions, in the form of a user interface specification, can be integrated with the JSD specification is developed and illustrated through examples.     

Computing reliability: On the differences between software testing and software fault injection techniques

System reliability has become a main concern during the computer-based system design process. It is one of the most important characteristics of the system quality. The continuous increase of the system complexity makes the reliability evaluation extremely costly. Therefore, there is need to develop new methods with less cost and effort. Furthermore, the system is vulnerable to both software and hardware faults. While the software faults are usually introduced by the programmer either at the design or the implementation stage of the software, the hardware faults are caused by physical phenomena affecting the hardware components, such as environmental perturbations, manufacturing defects, and aging-related phenomena. The software faults can only impact the software components. However, the hardware faults can propagate through the different system layers, and affect both the hardware and the software. This paper discusses the differences between the software testing and the software fault injections techniques used for reliability evaluation. We describe the mutation analysis as a method mainly used in software testing. Then, we detail the fault injection as a technique to evaluate the system reliability. Finally, we discuss how to use software mutation analysis in order to evaluate, at software level, the system reliability against hardware faults. The main advantage of this technique is its usability at early design stage of the system, when the instruction set architecture is not available. Experimental results run to evaluate faults occurring the memory show that the proposed approach significantly reduces the complexity of the system reliability evaluation in terms of time and cost.     

一种基于监测的嵌入式系统设计技术

提出一种嵌入式系统软硬件协同设计方法，它以数据流图为系统模型对嵌入式系统的功能和性能需求进行描述，并通过一种特定的实现结构，使得设计者可以借助快速样机平台和事件驱动式监测技术来精确测定目标系统对系统模型的实现状况，从而使得软硬件协同设计过程特别是系统优化和性能验证能在精确、可靠的测试数据基础上进行．同目前通常使用的以软硬件部件性能估算为基础的软硬件协同设计方法相比，这种以测试为基础的设计技术更能确保设计结果的合理．     

User interface evaluation and empirically-based evolution of a prototype experience management tool

Experience management refers to the capture, structuring, analysis, synthesis, and reuse of an organization's experience in the form of documents, plans, templates, processes, data, etc. The problem of managing experience effectively is not unique to software development, but the field of software engineering has had a high-level approach to this problem for some time. The Experience Factory is an organizational infrastructure whose goal is to produce, store, and reuse experiences gained in a software development organization. This paper describes The Q-Labs Experience Management System (Q-Labs EMS), which is based on the Experience Factory concept and was developed for use in a multinational software engineering consultancy. A critical aspect of the Q-Labs EMS project is its emphasis on empirical evaluation as a major driver of its development and evolution. The initial prototype requirements were grounded in the organizational needs and vision of Q-Labs, as were the goals and evaluation criteria later used to evaluate the prototype. However, the Q-Labs EMS architecture, data model, and user interface were designed to evolve, based on evolving user needs. This paper describes this approach, including the evaluation that was conducted of the initial prototype and its implications for the further development of systems to support software experience management.     

Refinement for user interface designs

Formal approaches to software development require that we correctly describe (or specify) systems in order to prove properties about our proposed solution prior to building it. We must then follow a rigorous process to transform our specification into an implementation to ensure that the properties we have proved are retained. Different transformation, or refinement, methods exist for different formal methods, but they all seek to ensure that we can guide the transformation in a way which preserves the desired properties of the system. Refinement methods also allow us to subsequently compare two systems to see if a refinement relation exists between the two. When we design and build the user interfaces of our systems we are similarly keen to ensure that they have certain properties before we build them. For example, do they satisfy the requirements of the user? Are they designed with known good design principles and usability considerations in mind? Are they correct in terms of the overall system specification? However, when we come to implement our interface designs we do not have a defined process to follow which ensures that we maintain these properties as we transform the design into code. Instead, we rely on our judgement and belief that we are doing the right thing and subsequent user testing to ensure that our final solution remains useable and satisfactory. We suggest an alternative approach, which is to define a refinement process for user interfaces which will allow us to maintain the same rigorous standards we apply to the rest of the system when we implement our user interface designs.     

An integrated component design approach to the development of a design information system for customer-oriented product design

In product design, the designer must draw upon many different types of information including both product-user requirements and design developments in related fields. As each designer has his/her own subjective opinion, an appropriate link between the user requirements and design features must be carefully developed to generate the most suitable design recommendation.The objective of this research is to develop a component design approach for the generation and evaluation of feasible design alternatives to help designers make the most effective design decisions. The development of the approach is based on the concept of component design and interaction links between user requirements and design output parameter values so that the design alternatives can be generated. A confidence interval for order statistics is then used to determine if enough unique feasible design alternatives have been generated. The research is presented in conjunction with a ballpoint pen design to demonstrate the feasibility and advantages of this approach.The results should provide designers with a useful method for the development of a design information system that will efficiently generate and evaluate a set of feasible design alternatives in product design process.     

To twist or poke? A method for identifying usability issues with the rotary controller and touch screen for control of in-vehicle information systems

In-vehicle information systems (IVIS) can be controlled by the user via direct or indirect input devices. In order to develop the next generation of usable IVIS, designers need to be able to evaluate and understand the usability issues associated with these two input types. The aim of this study was to investigate the effectiveness of a set of empirical usability evaluation methods for identifying important usability issues and distinguishing between the IVIS input devices. A number of usability issues were identified and their causal factors have been explored. These were related to the input type, the structure of the menu/tasks and hardware issues. In particular, the translation between inputs and on-screen actions and a lack of visual feedback for menu navigation resulted in lower levels of usability for the indirect device. This information will be useful in informing the design of new IVIS, with improved usability.

Statement of Relevance: This paper examines the use of empirical methods for distinguishing between direct and indirect IVIS input devices and identifying usability issues. Results have shown that the characteristics of indirect input devices produce more serious usability issues, compared with direct devices and can have a negative effect on the driver–vehicle interaction.     

A method to elicit architecturally sensitive usability requirements: its integration into a software development process

In the Human-Computer Interaction (HCI) community, software usability has primarily been concerned with the presentation of information, more precisely with the user interface. However, some usability problems can prove costly to fix if the changes require modifications that reach beyond the presentation layer, namely those that cannot be easily accommodated by the software architecture. Taking into account some usability requirements earlier in the software development cycle, specifically prior to the architectural design phase, can reduce the cost of these modifications. There is a scarcity of methods and guidelines with the scope to direct users in eliciting the usability requirements that can impact the software architecture. This paper proposes a usability-driven adaptation of the quality attribute workshop (QAW) to assist software development organizations in discovering and documenting usability requirements. It shows how this method can be integrated into a software development process, by discussing how the existing software framework workflows can be adjusted to take this new activity into consideration. A preliminary exercise was conducted to help discern the utility and the limits of the proposed method. Participants with different levels of knowledge of usability and comprehension of the system being developed found the method constructive, as it guided them in identifying the architecturally relevant usability requirements. It also helped determine the usability aspects that would not necessarily have been defined if this technique had not been employed.     

Modeling collaboration for large-scale software system integration by formal communication models

In the development of large software systems, several synchronization points have to be passed successfully in order to achieve high-end user acceptance of the final system. In particular, the integration of conceptual design entities has turned out to be the most crucial step in software development. At this point, an explicit representation of the organizational knowledge, namely how the integration can be performed, is required. The overall organization of the integration of design entities is a set of knowledge, action (including communication), and time. The knowledge part comprises the integration procedure itself as well as the design entities. Actions are performed by the members of the project team (project leaders, developers). Each integration step has to be performed synchronized, so that each member of the development team knows concurrently the status of the integration procedure. All other individual design activities are usually performed asynchronously. According to the need for an explicit representation of the organizational knowledge for large software system design, this paper attempts tounderstand the nature of large software system integration and formallydescribes the coordination of a dispersed set of software developers. The organizational knowledge is represented by processes that occur in an asynchronous development environment. The design knowledge is assigned to logical formulas that are processed as exchanged messages among developers. Hence, the activities for integration are based on the transmission of messages. Time-critical events, such as the integration of a particular design entity into the final conceptual design of the software system, have to become common knowledge concurrently for all developers. This organizational constraint has been formalized in the communication model. Not only can the proposed framework be adapted to organizational changes in a flexible way, it can also be applied to any specific development strategy for the integration of conceptual design entities. Thus, such kind of formal communication models may provide a sound basis for cooperative CASE (computer-aided software engineering) tools.This work has been sponsored by the Max Kade Foundation, New York.