Evaluation of user interfaces: EVADIS II— a comprehensive evaluation approach

As a result of the importance of the usability approach in system development and the EC's 'Directive concerning the minimum safety and health requirements for VDT workers' (EWG 1990), there is an accepted need for practical evaluation methods for user interfaces. The usability approach and the EC Directive are not restricted to user interface design, as they include the design of appropriate hardware and software, as well as organization, job, and task design. Therefore system designers are faced with many, often conflicting, requirements and need to address the question, 'How can usability requirements comprehensively be considered and evaluated in system development?' Customers buying hardware and software and introducing them into their organization ask, (How can I select easy-to-use hardware and software?' Both designers and customers need an evaluation procedure that covers all the organizational, user, hard- and software requirements. The evaluation method, EVADIS.II, we present in this paper overcomes characteristic deficiencies of previous evaluation methods. In particular, it takes the tasks, the user, and the organizational context into consideration during the evaluation process, and provides computer support for the use of the evaluation procedure.     

Evaluation of user interfaces: EVADIS II— a comprehensive evaluation approach

Abstract

As a result of the importance of the usability approach in system development and the EC's ‘Directive concerning the minimum safety and health requirements for VDT workers’ (EWG 1990), there is an accepted need for practical evaluation methods for user interfaces. The usability approach and the EC Directive are not restricted to user interface design, as they include the design of appropriate hardware and software, as well as organization, job, and task design. Therefore system designers are faced with many, often conflicting, requirements and need to address the question, ‘How can usability requirements comprehensively be considered and evaluated in system development?’ Customers buying hardware and software and introducing them into their organization ask, (How can I select easy-to-use hardware and software?’ Both designers and customers need an evaluation procedure that covers all the organizational, user, hard- and software requirements. The evaluation method, EVADIS.II, we present in this paper overcomes characteristic deficiencies of previous evaluation methods. In particular, it takes the tasks, the user, and the organizational context into consideration during the evaluation process, and provides computer support for the use of the evaluation procedure.     

Towards automatic evaluation of multimodal user interfaces

The evaluation of the usability and the learnability of a computer system may be performed with predictive models during the design phase. It may be done on the executable code as well as by observing the user in action. In the latter case, data collected in vivo must be processed. The goal is to provide software supports for performing this difficult and time consuming task.

The paper presents an early analysis of, and experience relating to, the automatic evaluation of multimodal user interfaces. With this end in view, a generic Wizard of Oz platform has been designed to allow the observation and automatic recording of subjects' behavior while they interact with a multimodal interface. It is then shown how recorded data can be analyzed to detect behavioral patterns, and how deviations of such patterns from a data-flow-oriented task model can be exploited by a software usability critic.     

Exploring the benefits of the combination of a software architecture analysis and a usability evaluation of a mobile application

Designing easy to use mobile applications is a difficult task. In order to optimize the development of a usable mobile application, it is necessary to consider the mobile usage context for the design and the evaluation of the user-system interaction of a mobile application. In our research we designed a method that aligns the inspection method “Software ArchitecTure analysis of Usability Requirements realizatioN” SATURN and a mobile usability evaluation in the form of a user test. We propose to use mobile context factors and thus requirements as a common basis for both inspection and user test. After conducting both analysis and user test, the results described as usability problems are mapped and discussed. The mobile context factors identified define and describe the usage context of a mobile application. We exemplify and apply our approach in a case study. This allows us to show how our method can be used to identify more usability problems than with each method separately. Additionally, we could confirm the validity and identified the severity of usability problems found by both methods. Our work presents how a combination of both methods allows to address usability issues in a more holistic way. We argue that the increased quantity and quality of results can lead to a reduction of the number of iterations required in early stages of an iterative software development process.     

Effectiveness of user testing and heuristic evaluation as a function of performance classification

For different levels of user performance, different types of information are processed and users will make different types of errors. Based on the error's immediate cause and the information being processed, usability problems can be classified into three categories. They are usability problems associated with skill-based, rule-based and knowledge-based levels of performance. In this paper, a user interface for a Web-based software program was evaluated with two usability evaluation methods, user testing and heuristic evaluation. The experiment discovered that the heuristic evaluation with human factor experts is more effective than user testing in identifying usability problems associated with skill-based and rule-based levels of performance. User testing is more effective than heuristic evaluation in finding usability problems associated with the knowledge-based level of performance. The practical application of this research is also discussed in the paper.     

Can icons enhance requirements engineering work?

When novel artifacts are proposed, their usability must be taken into account. We argue that effective visual notation depends critically on its usability, which assists members in more easily communicating with each other while reducing misunderstanding. In this paper, we extend prior research by elaborating already-refined artifacts (requirements engineering modeling, icon-based modeling and user preference modeling), and by carrying out summative (confirmatory) evaluation. The summative evaluation assists in determining artifacts׳ utility. Employing usability measurements, we examine perceptive, ergonomic, and hedonic qualities. The tests were performed in three series of iterations by two groups of practitioners. The first iteration was assigned to a few selected experts in computer and software companies to execute pre-testing. The second iteration was tested by the rest of the experts in the computer and software companies. The third iteration was examined by PhD students in the Faculty of Information Technology at the University of Jyväskylä. The findings demonstrate the usability of an icon-based approach, integrating altogether three artifacts, that serves users in a novel yet fairly effective manner. We suggest that an icon-based approach could serve as a positive basis on which to construct a requirements engineering visual notation that is easily understood and learned.     

A structured methodology for comparative evaluation of user interface designs using usability criteria and measures

Previous research in software ergonomics has indicated the importance of evaluating the usability of computer user interfaces. This paper presents a quantitative basis for selecting from among multiple alternative interfaces relative to multiple criteria of usability concern. The proposed model consists of two main phases: the prescreening phase and the evaluation phase. The prescreening phase involves expert judgment-based assessment with qualitative criteria. The first phase uses absolute measurement analytic hierarchy process (AHP) to filter possible alternative interfaces to a reasonable subset. The evaluation phase involves user-based assessment such as user testing, with quantitative criteria. The objective of the second phase is to evaluate a subset of alternatives using objective measures and to select the best alternative. A set of criteria and measures for evaluating the usability of computer software designs is presented. The proposed methodology is demonstrated in the interface design of a database system used to analyze trip cases information of nuclear power plant.Relevance to industry

There is currently a focus on the usability of interactive computer software. Software developers, interface designers or human factors engineers often confront the task of comparative evaluation among systems, versions or interface designs. The proposed methodology provides practitioners with a structured approach to select the best interface based on usability criteria and measures.     

基于投影的软件易用性评价方法

随着IT行业的不断发展,软件的易用性越来越受到用户和专家的关注与重视。如何科学地量化软件产品的易用性,进而对产品质量进行综合地测评,决定了软件产品的竞争力和用户的满意度。针对软件的易用性,提出了一种基于投影和直觉模糊理论的评价方法。首先,依据用户调查得到的信息,给出评价软件易用性的直觉模糊评价矩阵,并在此基础上建立了评价矩阵的理想决策。其次,借助群决策的框架,给出评价矩阵到理想决策上的投影。然后,基于投影给出测评软件的优劣序。最后,用一个测评实例展示出本方法的有效性和实用性。结果表明,本文提出的模型是一种软件质量综合评价的有效方法。     

An approach to usability evaluation of e-learning applications

Despite recent advances of electronic technologies in e-learning, a consolidated evaluation methodology for e-learning applications is not available. The evaluation of educational software must consider its usability and more in general its accessibility, as well as its didactic effectiveness. This work is a first step towards the definition of a methodology for evaluating e-learning applications. Specific usability attributes capturing the peculiar features of these applications are identified. A preliminary user study involving a group of e-students, observed during their interaction with an e-learning application in a real situation, is reported. Then, the proposal is put forward to adapt to the e-learning domain a methodology for systematic usability evaluation, called SUE. Specifically, evaluation patterns are proposed that are able to drive the evaluators in the analysis of an e-learning application.     

Modeling input modality choice in mobile graphical and speech interfaces

In this paper, we review three experiments with a mobile application that integrates graphical input with a touch-screen and a speech interface and develop a model for input modality choice in multimodal interaction. The model aims to enable simulation of multimodal human–computer interaction for automatic usability evaluation. The experimental results indicate that modality efficiency and input performance are important moderators of modality choice. Accordingly, we establish a utility-driven model that provides probability estimations of modality usage, based on the parameters of modality efficiency and input performance. Four variants of the model that differ in training data are fitted by means of Sequential Least Squares Programming. The analysis reveals a considerable fit regarding averaged modality usage. When applied to individual modality usage profiles, the accuracy decreases significantly. In an application example it is shown how the modality choice mechanism can be deployed for simulating interaction in the field of automatic usability evaluation. Results and possible limitations are discussed.     

Innovative User Interfaces in Automation Engineering by Application of Usability Engineering Methods Shown by the Example of a Three-Dimensional Plant Representation

This article is the first of 3 that focus on user-centered design and usability in the field of industrial process control. This article starts with an introduction to this special domain and sets out the need for user-centered design of innovative user interfaces. The MediaPlant research project is then introduced. This project aims to generate innovative user interface solutions for process control. The usability engineering methods presented are well known in relation to office application software and were applied in the course of the project to (a) evaluate the potential and application fields of innovative multimedia technology and 3-dimensional information presentation (described at the end of this article), (b) define innovative user interface building blocks for process control that can be applied across sectors (described in the second article, Komischke & Burmester [this issue]) and (c) develop human-centered user interfaces for controlling paper recycling plants (described in the third article, Epstein & Beu [this issue]).     

Training software developers and designers to conduct usability evaluations

Many efforts to improve the interplay between usability evaluation and software development rely either on better methods for conducting usability evaluations or on better formats for presenting evaluation results in ways that are useful for software designers and developers. Both of these approaches depend on a complete division of work between developers and evaluators. This article takes a different approach by exploring whether software developers and designers can be trained to conduct their own usability evaluations. The article is based on an empirical study where 36 teams with a total of 234 first-year university students on software development and design educations were trained through an introductory course in user-based website usability testing that was taught in 40 h. They used the techniques from this course for planning, conducting, and interpreting the results of a usability evaluation of an interactive website. They gained good competence in conducting the evaluation, defining user tasks and producing a usability report, while they were less successful in acquiring skills for identifying and describing usability problems.     

一种新的笔式交互范式的可用性评估

笔式用户界面是Post—WIMP界面的一种重要形态,具有自然、和谐的交互特性。尽管笔式用户界面广泛应用于无处不在计算环境中,但很少专门对它的可用性进行评估。针对笔式用户界面的特征,提出了一种新的笔式用户界面范式,基于此范式设计并实现了一个面向教学的笔式交互系统(PenOffice),利用此系统对范式从易学性、效率、出错率和用户满意度这几个方面进行了可用性评估,从中分析了笔式用户界面以及这种新的界面范式的优点和不足。     

Advance human–machine interface automatic evaluation

The need for accessibility evaluation tools is motivated by several endogenous and exogenous reasons coming from the end user (the designer and the developer) and companies releasing information systems. Existing evaluation tools mainly concentrate on examining the code of Web pages: Web pages more and more frequently contain non-HTML parts that entirely escape from being treated by existing techniques. This is the case of the advanced human–machine interface (AHMI), a piece of software programmed in C/C++, used for controlling the advanced flight management system in the aircraft cockpit. Studying this new user interface (UI) requires a structured approach to evaluate and validate AHMI designs. The goal in this work is to develop an evaluation tool to automate the process of evaluating the AHMI. The method addresses: support of multiple bases of guidelines (accessibility or usability or both) on-demand (partial or total evaluation), with different levels of details (a presentation for developers and for those responsible for certifying accessibility). The method goes a step toward the automatic evaluation of UI containing non-HTML parts.     

Evaluating a domain-specialist-oriented knowledge management system

We discuss the evaluation of software tools whose purpose is to assist humans to perform complex creative tasks. We call these creative task assistants (CTAs) and use as a case study CODE4, a CTA designed to allow domain specialists to manage their own knowledge base. We present an integrated process involving evaluation of usability, attractiveness and feature contribution, the latter two being the focus. To illustrate attractiveness evaluation, we assess whether CODE4 has met its objective of having users not trained in logical formalisms choose the tool to represent and manipulate knowledge in a computer. By studying use of the tool by its intended users, we conclude that it has met this objective. To illustrate feature contribution evaluation, we assess what aspects of CODE4 have in fact led to its success. To do this, we study what tasks are performed by users, and what features of both knowledge representation and user interface are exercised. We find that features for manipulating the inheritance hierarchy and naming concepts are considered the most valuable. Our overall conclusion is that those developing or researching CTAs would benefit from using the three types of evaluation in order to make effective decisions about the evolution of their products.     

Evaluation of e-commerce websites accessibility and usability: an e-commerce platform analysis with the inclusion of blind users

Considering the importance associated with e-commerce website accessibility and usability, a study on one of the most relevant Portuguese e-commerce websites has been performed using both automatic and manual assessment procedures. In an initial stage, we evaluated the chosen website with a Web accessibility and usability automatic tool called SortSite; after that, we performed a manual evaluation to verify each previously detected error and present possible solutions to overcome those faults. In a third phase, three usability specialists have been used to perform a heuristic evaluation of the chosen website. Finally, user tests with blind people were carried out in order to fully assess the compliance with accessibility and usability guidelines and standards. The results showed that the platform had a good score regarding the automatic evaluation; however, when the heuristic and manual evaluations were performed, some accessibility and usability problems were discovered. Moreover, the user test results showed bad marks regarding efficiency, effectiveness, and satisfaction by the group of participants. As a conclusion, we highlight user interaction problems and propose seven recommendations focused on enhancing accessibility and usability of not only the evaluated e-commerce website, but also of other similar ones.     

Using task analysis to improve usability of fatigue modelling software

The design of any interactive computer system requires consideration of both humans and machines. Software usability is one aspect of human–computer interaction that can benefit from knowledge of the user and their tasks. One set of methods for determining whether an application enables users to achieve their predetermined goals effectively and efficiently is task analysis. In the present study, a task analysis was applied to the graphical user interface of fatigue modelling software used in industry. The task analysis procedure allowed areas of usability improvement to be identified and then addressed with alternate interface prototypes. The present method of task analysis illustrates a practical and efficient way for software designers to improve software usability, user effectiveness and satisfaction, by involving users in the design process.     

Interpretation of a cross-cultural usability evaluation: A case study based on a hypermedia system for rare species management in Namibia

We present the results of a usability evaluation of a locally developed hypermedia information system aiming at conservation biologists and wildlife managers in Namibia. Developer and end user come from different ethnic backgrounds, as is common to software development in Namibia and many developing countries. To overcome both the cultural and the authoritarian gap between usability evaluator and user, the evaluation was held as a workshop with usability evaluators who shared the target users’ ethnic and social backgrounds. Different data collection methods were used and results as well as specific incidences recorded. Results suggest that it is difficult for Namibian computer users to evaluate functionality independently from content. Users displayed evidence of a passive search strategy and an expectation that structure is provided rather than self generated. The comparison of data collection methods suggests that questionnaires are inappropriate in Namibia because they do not elicit a truthful response from participants who tend to provide answers they think are “expected”. The paper concludes that usability goals and methods have to be determined and defined within the target users’ cultural context.     

JSD and the design of user interface software

Systems analysis and design methods (SADMs), such as Jackson system development (JSD), have been proposed as a means of improving software quality. This is contrasted with the role of improving software usability proposed for human factors (HF). The paper argues the need for an explicit user interface specification stage in system development to ensure the usability of proposed systems. It is suggested that JSD supplemented by the specialist knowledge of HF can support such a specification stage. For HF and JSD to be successfully integrated, contributions from both need to be appropriately timed and structured. A model of the system development process that incorporates HF contributions is proposed. A means by which the HF contributions, in the form of a user interface specification, can be integrated with the JSD specification is developed and illustrated through examples.