Usability testing in 2000 and beyond

Usability testing is a widely used technique to evaluate user performance and acceptance of products and systems. It was introduced in the late 1980s and rose to popularity in the past decade. This paper provides a view of the current status of usability testing as a method and describes how it will be used in the 21st century. Although usability testing may not be the most efficient technique for discovery of usability problems, it is a reliable way to estimate quantitatively users' performance and subjective satisfaction with products. Four major trends in usability testing include: common reporting formats and methods for industry; Internet application and website testing; testing of mobile, handheld devices; and testing in more naturalistic environments such as simulated homes and classrooms. In the 21st century, 'quick and clean' usability testing methods are needed to provide valid and reliable data on how well people use products and systems, and how they like using them.     

The effects of prototype medium on usability testing

Inconsistencies among testing methods and results in previous research prompted this study that builds upon a systematic usability testing research framework to better understand how interface medium influences users' abilities to detect usability flaws in applications. Interface medium was tested to identify its effects on users' perceptions of usability and abilities to detect usability problems and severe usability problems. Results indicated that medium has no effect on users' abilities to detect usability problems or perceptions of usability. However, results did indicate an interaction between the medium and the tested application in which users were able to identify significantly more usability problems on a higher fidelity medium using a particular application. Results also indicated that as users' perceptions of an application's usability increases, the users are less able to detect usability problems in that application. Usability testing should begin early in the design process, even if low fidelity mediums will be used.     

Usability Testing and Instruction Librarians

ABSTRACT

This study examines how librarians are experiencing usability testing and how their observations are influencing library instruction. A survey of instruction librarians illustrates how usability testing and library instruction are connected. Survey results prove that instruction librarians are involved in usability testing. Furthermore, their participation in usability studies has led instruction librarians to alter their instructional methods. An overwhelming majority changed one or more instructional tools as a result of usability testing, and many reported creating new instructional resources. The authors add their own insights as both instruction librarians and participants in usability testing.     

A methodology for evaluating the usability of audiovisual consumer electronic products

Usability evaluation is now considered an essential procedure in consumer product development. Many studies have been conducted to develop various techniques and methods of usability evaluation hoping to help the evaluators choose appropriate methods. However, planning and conducting usability evaluation requires considerations of a number of factors surrounding the evaluation process including the product, user, activity, and environmental characteristics. In this perspective, this study suggested a new methodology of usability evaluation through a simple, structured framework. The framework was outlined by three major components: the interface features of a product as design variables, the evaluation context consisting of user, product, activity, and environment as context variables, and the usability measures as dependent variables. Based on this framework, this study established methods to specify the product interface features, to define evaluation context, and to measure usability. The effectiveness of this methodology was demonstrated through case studies in which the usability of audiovisual products was evaluated by using the methods developed in this study. This study is expected to help the usability practitioners in consumer electronics industry in various ways. Most directly, it supports the evaluators' plan and conduct usability evaluation sessions in a systematic and structured manner. In addition, it can be applied to other categories of consumer products (such as appliances, automobiles, communication devices, etc.) with minor modifications as necessary.     

Usability testing in a competive market: lessons learned

Abstract

Usability testing is a relatively new and rapidly developing field. Newcomers to usability testing typically enter the profession with a knowledge of social science research methods and the belief that usability testing is conducted in a lab using thinking aloud techniques to identify usability defects. Our practice of usability testing at Mead Data Central has shown that these core beliefs represent an approach of limited utility in designing products like the LEX1S®NEXIS® research systems. In this paper, we describe our experiences in conducting traditional usability testing and how we used the results of those efforts to develop more effective methods of testing for Mead Data Central's products and customers.     

Metacognition and system usability: Incorporating metacognitive research paradigm into usability testing

There is an agreement that perceived usability is important beyond actual effectiveness of software systems. Perceived usability is often obtained by self-reports provided after system use. Aiming to improve summative usability testing, we propose a methodology to enhance in-depth testing of users' performance and perceived usability at the task level. The metacognitive research approach allows detailed analysis of cognitive processes. Adapting its methodologies, we propose the Metacognitive Usability Profile (MUP) which includes a comprehensive set of measures based on collecting confidence in the success of each particular task and triangulating it with objective measures. We demonstrate using the MUP by comparing two versions of a project management system. Based on a task analysis we allocated tasks that differ between the versions and let participants (N = 100) use both versions. Although no difference was found between the versions in system-level perceived usability, the detailed task-level analysis exposed many differences. In particular, overconfidence was associated with low performance, which suggests that user interfaces better avoid illusions of knowing. Overall, the study demonstrates how the MUP exposes challenges users face. This, in turn, allows choosing the better task implementation among the examined options and to focus attempts for usability improvement.     

The role of non-visual aesthetics in consumer product evaluation

The present article addresses the impact of non-visual aesthetics of consumer products on the outcomes of usability tests such as perceived usability, user performance and user affect. Building on the research surrounding the impact of visual aesthetics in usability testing, the present work aimed to determine whether the same pattern of effects repeatedly found for visual aesthetics will also be found for non-visual aesthetics. A series of three experiments was carried out, all examining sound and touch as two prominent dimensions of non-visual aesthetics. To increase the robustness of the results, the experiments were conducted with three different consumer products, which were a smartphone (N=60), vacuum cleaner (N=60), and video racing game (N=60). Although manipulation checks confirmed that users generally experienced both sound and tactile product properties as intended by the experimental manipulation, in none of the studies the results showed an effect on perceived usability. Other outcome variables such as affect and performance showed a few selected effects. Overall, this suggests that findings from the field of visual aesthetics cannot be easily transferred to the domain of non-visual aesthetics. The findings of the present studies are discussed within the framework of Schifferstein’s work on sensory dominance.     

Information system professionals’ knowledge and application gaps toward Web design guidelines

Web design guidelines are adopted by many usability evaluation methods as one of the criteria for success, while usability is proven to significantly impact Website performance. Since Web design guidelines cover a broad range of system and interface design solutions, knowledge of them can be considered as a prominent indicator of Web design skills for information systems (IS) professionals. This study empirically assessed how much IS professionals know and apply Web design guidelines via a survey to 500 randomly selected companies from Taiwan’s Fortune 2000 corporations. As expected, the knowledge–application gaps of IS professionals were statistically significant in all Web design guideline categories. Meanwhile, certain guideline categories were proven to be more difficult to acquire or apply than others. Finally, degree, gender, experience, training hours, and courses taken were also proven to be determining factors for Web design guideline skills. Implications for developing Web design guideline skills are also discussed.     

Work-domain knowledge in usability evaluation: Experiences with Cooperative Usability Testing

Usability evaluation helps to determine whether interactive systems support users in their work tasks. However, knowledge about those tasks and, more generally, about the work-domain is difficult to bring to bear on the processes and outcome of usability evaluation. One way to include such work-domain knowledge might be Cooperative Usability Testing, an evaluation method that consists of (a) interaction phases, similar to classic usability testing, and (b) interpretation phases, where the test participant and the moderator discuss incidents and experiences from the interaction phases. We have studied whether such interpretation phases improve the relevance of usability evaluations in the development of work-domain specific systems. The study included two development cases. We conclude that the interpretation phases generate additional insight and redesign suggestions related to observed usability problems. Also, the interpretation phases generate a substantial proportion of new usability issues, thereby providing a richer evaluation output. Feedback from the developers of the evaluated systems indicates that the usability issues that are generated in the interpretation phases have substantial impact on the software development process. The benefits of the interpretation phases may be explained by the access these provide both to the test participants’ work-domain knowledge and to their experiences as users.     

A method to elicit architecturally sensitive usability requirements: its integration into a software development process

In the Human-Computer Interaction (HCI) community, software usability has primarily been concerned with the presentation of information, more precisely with the user interface. However, some usability problems can prove costly to fix if the changes require modifications that reach beyond the presentation layer, namely those that cannot be easily accommodated by the software architecture. Taking into account some usability requirements earlier in the software development cycle, specifically prior to the architectural design phase, can reduce the cost of these modifications. There is a scarcity of methods and guidelines with the scope to direct users in eliciting the usability requirements that can impact the software architecture. This paper proposes a usability-driven adaptation of the quality attribute workshop (QAW) to assist software development organizations in discovering and documenting usability requirements. It shows how this method can be integrated into a software development process, by discussing how the existing software framework workflows can be adjusted to take this new activity into consideration. A preliminary exercise was conducted to help discern the utility and the limits of the proposed method. Participants with different levels of knowledge of usability and comprehension of the system being developed found the method constructive, as it guided them in identifying the architecturally relevant usability requirements. It also helped determine the usability aspects that would not necessarily have been defined if this technique had not been employed.     

A systematic mapping study of API usability evaluation methods

An Application Programming Interface (API) provides a programmatic interface to a software component that is often offered publicly and may be used by programmers who are not the API’s original designers. APIs play a key role in software reuse. By reusing high quality components and services, developers can increase their productivity and avoid costly defects. The usability of an API is a qualitative characteristic that evaluates how easy it is to use an API. Recent years have seen a considerable increase in research efforts aiming at evaluating the usability of APIs. An API usability evaluation can identify problem areas and provide recommendations for improving the API. In this systematic mapping study, we focus on 47 primary studies to identify the aim and the method of the API usability studies. We investigate which API usability factors are evaluated, at which phases of API development is the usability of API evaluated and what are the current limitations and open issues in API usability evaluation. We believe that the results of this literature review would be useful for both researchers and industry practitioners interested in investigating the usability of API and new API usability evaluation methods.     

Stakeholder logistics of an interactive system

Although it seems that software metrics have moved beyond mere performance measurement, it is not too clear how machine effectiveness, efficiency, and effort pertain to human requirements on such matters. In industry as well as academia, the ISO 9241-11 norm provides the dominant view on usability, stating that usability is a function of effectiveness, efficiency, and satisfaction. Although intuitively, usability requirements should be part of a software's design in an early stage, conceptually and empirically, it seems more likely that performance requirements (i.e., the absence of errors) should be the center of concern. This paper offers an elaborated view on usability, satisfaction, and performance. Certain theoretical conceptions are tested with data gathered from professional users of banking and hospital systems by means of a 4-year single-item survey and a structured questionnaire, respectively. Results suggested that performance factors (i.e., efficiency) are more important than usability in understanding why stakeholders are satisfied with a system or not. Moreover, it neither is dissatisfaction with a system nor that a system is less usable that predicate requirements change. Instead, avoiding machine inaccuracy best predicted the variability in agreement to “must have” requirements, while achieving human accuracy predicted the variability in agreement to the “won’t have” requirements. The present contribution provides a consistent research framework that can bring more focus to design (i.e., prioritization), clarify discussions about design trade-offs, makes concepts measurable, and eventually may lead to better-informed designs.     

Exploring the benefits of the combination of a software architecture analysis and a usability evaluation of a mobile application

Designing easy to use mobile applications is a difficult task. In order to optimize the development of a usable mobile application, it is necessary to consider the mobile usage context for the design and the evaluation of the user-system interaction of a mobile application. In our research we designed a method that aligns the inspection method “Software ArchitecTure analysis of Usability Requirements realizatioN” SATURN and a mobile usability evaluation in the form of a user test. We propose to use mobile context factors and thus requirements as a common basis for both inspection and user test. After conducting both analysis and user test, the results described as usability problems are mapped and discussed. The mobile context factors identified define and describe the usage context of a mobile application. We exemplify and apply our approach in a case study. This allows us to show how our method can be used to identify more usability problems than with each method separately. Additionally, we could confirm the validity and identified the severity of usability problems found by both methods. Our work presents how a combination of both methods allows to address usability issues in a more holistic way. We argue that the increased quantity and quality of results can lead to a reduction of the number of iterations required in early stages of an iterative software development process.     

Usability guideline for banking software design

As the diversity of services in the financial market increases, it is critical to design usable banking software in order to overcome the complex structure of the system. The current study presents a usability guideline based on heuristics and their corresponding criteria that could be used during the early stages of banking software design process. In the design of a usability guideline, the heuristics and their criteria are categorized in terms of their effectiveness in solving usability problems grouped and ranging from usability catastrophe to cosmetic problems. The current study comprises of three main steps: First, actual usability problems from three banking software development projects are categorized according to their severity level. Secondly, usability criteria are rated for how well they explain the usability problems encountered. Finally, usability heuristics are categorized according to the severity level of usability problems through two analytical models; corresponding and cluster analyses. As the result, designers and project managers may give more importance to the heuristics related with the following usability problem categories: Usability catastrophe and then major usability problems. Furthermore, the proposed guideline can be used to understand which usability criteria would be helpful in explaining usability problems as well as preventing banking system catastrophes, by highlighting the critical parts in system design of banking software.     

A proposal for modelling usability in a holistic MDD method

Holistic methods for Model-Driven Development (MDD) aim to model all the system features in a conceptual model. This conceptual model is the input for a model compiler that can generate software systems by means of automatic transformations. However, in general, MDD methods focus on modelling the structure and functionality of systems, relegating the interaction and usability features to manual implementations at the last steps of the software development process. Some usability features are strongly related to the functionality of the system and their inclusion is not so easy. In order to facilitate the inclusion of functional usability features from the first steps of the development process and bring closer MDD methods to the holistic perspective, we propose a Usability Model. The Usability Model gathers conceptual primitives that represent functional usability features in a sufficiently abstract way so that the model can be used with different holistic MDD methods. This paper defines all the primitives that can be used to represent functional usability features. Moreover, we have defined a process to include the Usability Model in any MDD method without affecting its existing conceptual model. The proposal is based on model-to-model and model-to-code transformations. As proof of concept, we have applied our proposal to an existing MDD method called the OO-method and we have measured its efficiency.     

多媒体产品的可用性设计指南

多媒体产品的设计应在重视产品功能性的同时重视其可用性。可用性体现了从用户角度感受到的产品质量,是决定产品成败的关键因素之一。为了提高多媒体产品的可用性质量,必须在产品的设计开发过程中运用可用性工程方法,其中包括使用多媒体的可用性设计指南,该文将综合介绍目前提出的多媒体产品的可用性设计指南。     

Software development methods and usability: Perspectives from a survey in the software industry in Norway

This paper investigates the relationship between software development methodologies and usability. The point of departure is the assumption that two important disciplines in software development, one of software development methods (SDMs) and one of usability work, are not integrated in industrial software projects.

Building on previous research we investigate two questions; (1) Will software companies generally acknowledge the importance of usability, but not prioritise it in industrial projects? and (2) To what degree are software development methods and usability perceived by practitioners as being integrated? To this end a survey in the Norwegian IT industry was conducted. From a sample of 259 companies we received responses from 78 companies.

In response to our first research question, our findings show that although there is a positive bias towards usability, the importance of usability testing is perceived to be much less than that of usability requirements. Given the strong time and cost pressures associated with the software industry, we believe that these results highlight that there is a gap between intention and reality. Regarding our second research question our survey revealed that companies perceive usability and software development methods to be integrated. This is in contrast to earlier research, which, somewhat pessimistically, has argued for the existence of two different cultures, one of software development and one of usability. The findings give hope for the future, in particular because the general use of system development methods are pragmatic and adaptable.     

Multi‐Criteria Usability Evaluation of Electronic Devices in a Fuzzy Environment

The user‐oriented design is the most important competition issue in today's market. In recent years, user‐oriented design (i.e., usability) has become expected, particularly in electronic devices. Usability considers high user satisfaction in conjunction with user performance. Usability of electronic products is essential for high customer satisfaction as well as product life cycles in companies. Therefore, MP3 players were selected among the electronic products. Designing usable MP3 players is extremely important for users who have close interaction with them. In this study, MP3 players of different types and sizes were selected and their usability evaluated. The usability criteria used in the MP3 players' evaluation were divided into two major categories: performance and emotional expectations. The alternatives were ranked with fuzzy multi‐criteria decision‐making methods, namely, Fuzzy TOPSIS (FTOPSIS), Fuzzy Analytic Hierarchy Process (FAHP), and Fuzzy Axiomatic Design Theory (FADT), and the results were compared. Although the same decision matrices and data obtained from semantic differential experiment were used for all fuzzy multi‐criteria decision‐making methods, it is shown that different rankings were obtained. © 2012 Wiley Periodicals, Inc.     

A novel model usability evaluation framework (MUsE) for explainable artificial intelligence

When it comes to complex machine learning models, commonly referred to as black boxes, understanding the underlying decision making process is crucial for domains such as healthcare and financial services, as well as when they are used in connection with safety critical systems such as autonomous vehicles. As a result, interest in explainable artificial intelligence (xAI) tools and techniques has increased in recent years. However, the user experience (UX) effectiveness of existing xAI frameworks, especially concerning algorithms that work with data as opposed to images, is still an open research question. In order to address this gap, we examine the UX effectiveness of the Local Interpretable Model-Agnostic Explanations (LIME) xAI framework, one of the most popular model agnostic frameworks found in the literature, with a specific focus on its performance in terms of making tabular models more interpretable. In particular, we apply several state of the art machine learning algorithms on a tabular dataset, and demonstrate how LIME can be used to supplement conventional performance assessment methods. Based on this experience, we evaluate the understandability of the output produced by LIME both via a usability study, involving participants who are not familiar with LIME, and its overall usability via a custom made assessment framework, called Model Usability Evaluation (MUsE), which is derived from the International Organisation for Standardisation 9241-11:2018 standard.     

Evaluating teamwork support in tabletop groupware applications using collaboration usability analysis

Tabletop groupware systems have natural advantages for collaboration, but they present a challenge for application designers because shared work and interaction progress in different ways than in desktop systems. As a result, tabletop systems still have problems with usability. We have developed a usability evaluation technique, T-CUA, that focuses attention on teamwork issues and that can help designers determine whether prototypes provide adequate support for the basic actions and interactions that are fundamental to table-based collaboration. We compared T-CUA with expert review in a user study where 12 evaluators assessed an early tabletop prototype using one of the two evaluation methods. The group using T-CUA found more teamwork problems and found problems in more areas than those using expert review; in addition, participants found T-CUA to be effective and easy to use. The success of T-CUA shows the benefits of using a set of activity primitives as the basis for discount usability techniques.