Determining the effectiveness of the usability problem inspector: a theory-based model and tool for finding usability problems

Despite the increased focus on usability and on the processes and methods used to increase usability, a substantial amount of software is unusable and poorly designed. Much of this is attributable to the lack of cost-effective usability evaluation tools that provide an interaction-based framework for identifying problems. We developed the user action framework and a corresponding evaluation tool, the usability problem inspector (UPI), to help organize usability concepts and issues into a knowledge base. We conducted a comprehensive comparison study to determine if our theory-based framework and tool could be effectively used to find important usability problems in an interface design, relative to two other established inspection methods (heuristic evaluation and cognitive walkthrough). Results showed that the UPI scored higher than heuristic evaluation in terms of thoroughness, validity, and effectiveness and was consistent with cognitive walkthrough for these same measures. We also discuss other potential advantages of the UPI over heuristic evaluation and cognitive walkthrough when applied in practice. Potential applications of this work include a cost-effective alternative or supplement to lab-based formative usability evaluation during any stage of development.     

A heuristic checklist for an accessible smartphone interface design

Smartphone technology has evolved into a multi-functional device with advanced capabilities, but this mobile technology remains inaccessible to many individuals with visual impairments or upper extremity disabilities. This paper provides a heuristic checklist for accessible smartphone interface design, developed through reviewing existing design standards and guidelines and validating these guidelines with user involvement. Specifically, a set of preliminary user requirements (59 items) was extracted from existing standards, guidelines, and user requirements regarding mobile handheld device accessibility. Subsequently, the requirement set was filtered using a participatory method and then integrated to create an operational version of design guidelines. These guidelines were then used in a heuristic evaluation and usability testing on high-fidelity prototypes produced by a commercial manufacturer. A heuristic checklist for designing accessible smartphones was formed, which may also be applicable to other touchscreen handheld devices (e.g., printer screen) in terms of accessibility features. The initial set of 59 user requirements was re-organized into 44 statements in six general categories: mechanical controls, display, speech and general operation controls, audio feedback controls, touch-operated controls, and others. Using results from both qualitative and quantitative methods provides support, though with some limitations, for this accessibility checklist. This checklist is intended as a practical design support tool for use in early design phases of handheld products. A number of challenges and limitations are discussed as well.     

A Usability Checklist for the Usability Evaluation of Mobile Phone User Interface

In the last decade, the research of the usability of mobile phones has been a newly evolving area with few established methodologies and realistic practices that ensure capturing usability in evaluation. Thus, there exists growing demand to explore appropriate evaluation methodologies that evaluate the usability of mobile phones quickly as well as comprehensively. This study aims to develop a task-based usability checklist based on heuristic evaluations in views of mobile phone user interface (UI) practitioners. A hierarchical structure of UI design elements and usability principles related to mobile phones were developed and then utilized to develop the checklist. To demonstrate the practical effectiveness of the proposed checklist, comparative experiments were conducted on the usability checklist and usability testing. The majority of usability problems found by usability testing and additional problems were discovered by the proposed checklist. It is expected that the usability checklist proposed in this study could be used quickly and efficiently by usability practitioners to evaluate the mobile phone UI in the middle of the mobile phone development process.     

Developing SMASH: A set of SMArtphone's uSability Heuristics

The smartphone market is nowadays highly competitive. When buying a new device, users focus on visual esthetics, ergonomics, performance, and user experience, among others. Assessing usability issues allows improving these aspects. One popular method for detecting usability problems is heuristic evaluation, in which evaluators employ a set of usability heuristics as guide. Using proper heuristics is highly relevant. In this paper we present SMASH, a set of 12 usability heuristics for smartphones and mobile applications, developed iteratively. SMASH (previously named TMD: Usability heuristics for Touchscreen-based Mobile Devices) was experimentally validated. The results support its utility and effectiveness.     

Usability principles and best practices for the user interface design of complex 3D architectural design and engineering tools

This study proposes usability principles for the user interfaces (UI) design of complex 3D parametric architectural design and engineering tools. Numerous usability principles have been developed for generic desktop or web applications. The authors tried to apply existing usability principles as guidelines for evaluating complex 3D design and engineering applications. However, the principles were too generic and high-level to be useful as design or evaluation guidelines. The authors, all with more than 10 or 30 years of experience with various CAD systems, selected and reviewed 10 state-of-the-art 3D parametric design and engineering applications and captured what they thought were best practices, as screenshots and videos. The collected best practices were reviewed through a series of discussion sessions. During the discussion sessions, UI design principles underlying the collected best practices were characterized in the line of existing UI principles. Based on the best practices and the derived common UI principles, a new set of refined and detailed UI principles were proposed for improving and evaluating 3D parametric engineering design tools in the future.     

Understanding usability issues addressed by three user-system interface evaluation techniques

Three structured judgment evaluation techniques were applied to a system with a graphical direct manipulation style interface, to understand the types of usability problems they address. These evaluation techniques were cognitive walkthrough, heuristic evaluation, and the Smith and Mosier (1986) guidelines. The authors wanted to learn whether the techniques identify problems: across all stages of user activity, which noticeably affect users' performance with the system, and which are important to the usability of direct manipulation-style systems. Results showed that the cognitive walkthrough method identifies issues almost exclusively within the action specification stage, while guidelines covered more stages. The walkthrough was best, however, and the guidelines worst at predicting problems that cause users noticeable difficulty (as observed during a usability study). All the techniques could be improved in assessing semantic distance and addressing all stages on the evaluation side of the HCI activity cycle. To evaluate the directness of engagements, improved or new techniques are needed.     

The scenario‐based usability checklist development for home appliance design: A case study

In the past decade, home appliances have been rapidly developed to satisfy the various requirements of users. Thus, there are increases in requirements to explore appropriate evaluation methods that reflect the usability of home appliances quickly and comprehensively. This study aims to develop a scenario‐based usability checklist for product designers early in the design process. In this study, the scenario‐based usability checklist consists of two parts: 1) a heuristic evaluation checklist, and 2) a scenario evaluation checklist. In heuristic evaluation checklist development, usability factors of home appliances are extracted and then coupled to user interface (UI) elements. In scenario evaluation checklist development, scenarios are developed through brainstorming by focus group interview (FGI), and then evaluation elements are extracted from previous scenarios analysis. The proposed scenario‐based usability checklists can enable designers to evaluate product design quickly and comprehensively early in the development process with users' viewpoints. © 2010 Wiley Periodicals, Inc.     

What users really think about the usability of smartphone applications: diversity based empirical investigation

In recent years, smartphone devices are becoming progressively popular across a diverse range of users. However, user diversity creates challenges in smartphone application (app) development. The diversity of users is often ignored by designers and developers due to the absence of requirements. Owing to this, many smartphone users face usability issues. Despite that, no dedicated platform found that guide smartphone app designers and developers regarding human universality. The aim of this research is to explore the requirements of diverse users in smartphone apps and provide usability guidelines. The objectives of this research are achieved by following two scientific approaches. The human diversity requirements are located by conducting usability tests that investigated the requirements in the form of usability issues. The systematic literature review (SLR) process is followed in order to resolve the discovered usability issues. Both approaches resulted in a list of usability issues and guidelines. The usability tests returned 27 problems while the SLR came with a comprehensive set of universal usability guidelines that were grouped into eleven categories. The study concluded with some major outcomes. The results show evidence of critical usability problems that must be addressed during the design and development of smartphone apps. Moreover, the study also revealed that people with disabilities were three times severely affected by usability problems in such apps than people of different ages and their needs must be considered a top priority in the development of smartphone apps.

     

A comparison of usability evaluation methods for evaluating e-commerce websites

The importance of evaluating the usability of e-commerce websites is well recognised. User testing and heuristic evaluation methods are commonly used to evaluate the usability of such sites, but just how effective are these for identifying specific problems? This article describes an evaluation of these methods by comparing the number, severity and type of usability problems identified by each one. The cost of employing these methods is also considered. The findings highlight the number and severity level of 44 specific usability problem areas which were uniquely identified by either user testing or heuristic evaluation methods, common problems that were identified by both methods, and problems that were missed by each method. The results show that user testing uniquely identified major problems related to four specific areas and minor problems related to one area. Conversely, the heuristic evaluation uniquely identified minor problems in eight specific areas and major problems in three areas.     

Effectiveness of user testing and heuristic evaluation as a function of performance classification

For different levels of user performance, different types of information are processed and users will make different types of errors. Based on the error's immediate cause and the information being processed, usability problems can be classified into three categories. They are usability problems associated with skill-based, rule-based and knowledge-based levels of performance. In this paper, a user interface for a Web-based software program was evaluated with two usability evaluation methods, user testing and heuristic evaluation. The experiment discovered that the heuristic evaluation with human factor experts is more effective than user testing in identifying usability problems associated with skill-based and rule-based levels of performance. User testing is more effective than heuristic evaluation in finding usability problems associated with the knowledge-based level of performance. The practical application of this research is also discussed in the paper.     

Expert vs novice collaborative heuristic evaluation (CHE) of a smartphone app for cultural heritage sites

This study was conducted to compare CHE between Human-Computer Interaction (HCI) experts and novices in evaluating the Smartphone app for the cultural heritage site. It uses the Smartphone Mobile Application heuRisTics (SMART), focusing on smartphone applications and traditional Nielsen heuristics, focusing on a wider range of interactive systems. Six experts and six novices used the severity rating scale to categorise the severity of the usability issues. These issues were mapped to both heuristics. The study found that experts and novice evaluators identified 19 and 14 usability issues, respectively, with ten as the same usability issues. However, these same usability issues have been rated differently. Although the t-test indicates no significant differences between experts and novices in their ratings for usability issues, these results nevertheless indicate the need for both evaluators in CHE to provide a more comprehensive perspective on the severity of the usability issues. Furthermore, the mapping of the usability issues for Nielsen and SMART heuristics concluded that more issues with the smartphone app could be addressed through smartphone-specific heuristics than general heuristics, indicating a better tool for heuristic evaluation of the smartphone app. This study also provides new insight into the required number of evaluators needed for CHE.

     

Extending the Usability of Heuristics for Design and Evaluation: Lead,Follow, and Get Out of the Way

User-centered design practitioners have often relied on discount usability engineering methods using heuristics. Top 10 lists of design and evaluation heuristics have proliferated during the 1990s, leading to a plethora of heterogeneous heuristic guidelines for a multitude of user interfaces. A simple, unified set of heuristics that might be applicable across different technologies, understood between disciplines, and equivalent to metrics used to measure ease of use can be beneficial.

Lead, follow, and get out of the way (LF&G) theorizes that the optimal human-computer interaction (HCI) experience is analogous to a facilitative learning relation-Like a good teacher, mentor, or coach, the usable user interface leads the user to successful completion of tasks and goals; follows the user's progress and provides appropriate feedback and information when needed; and gets out of the way of the user to allow efficient and effective completion of tasks as the user attains mastery of the system, its concepts, and operations. A set of 18 heuristics grouped under the 3 general principles of the title are provided for use across the tasks of design guidance, development refinement, and end-user evaluation of computer systems.

This article provides background into the development of LF&G, case studies on its use in real-world product design and development, and directions for further research needed to develop this approach into an empirically based method for defining, describing, designing, and predicting the ease of use of interactive human-computer systems.     

User Interface Design Principles for Interactive Television Applications

Interactive Television (ITV) applications gratify entertainment needs and leisure activities in a relaxed domestic context. Then, the mentality of efficiency and task completion implied by many user interface heuristics may not be suitable for design and expert evaluation of ITV. In line with the contemporary movement towards the design of the user experience of computing systems, a set of design principles for ITV applications has been developed. Instead of information technology, the field of communication science was explored, which has accumulated an extensive theory of TV usage and viewer behavior. For the empirical validation, the principles were employed in the design of an interactive music TV application. The results suggest that a small set of generic principles could support the systematic design of ITV applications and produce a design rationale that is traceable and comparable to alternative solutions, during the process of early product development. In addition, the principles could be refined into specific guidelines and further employed to support expert evaluation techniques.     

Usability evaluation of anti-phishing toolbars

Phishing is considered as one of the most serious threats for the Internet and e-commerce. Phishing attacks abuse trust with the help of deceptive e-mails, fraudulent web sites and malware. In order to prevent phishing attacks some organizations have implemented Internet browser toolbars for identifying deceptive activities. However, the levels of usability and user interfaces are varying. Some of the toolbars have obvious usability problems, which can affect the performance of these toolbars ultimately. For the sake of future improvement, usability evaluation is indispensable. We will discuss usability of five typical anti-phishing toolbars: built-in phishing prevention in the Internet Explorer 7.0, Google toolbar, Netcraft Anti-phishing toolbar and SpoofGuard. In addition, we included Internet Explorer plug-in we have developed, Anti-phishing IEPlug. Our hypothesis was that usability of anti-phishing toolbars, and as a consequence also security of the toolbars, could be improved. Indeed, according to the heuristic usability evaluation, a number of usability issues were found. In this article, we will describe the anti-phishing toolbars, we will discuss anti-phishing toolbar usability evaluation approach and we will present our findings. Finally, we will propose advices for improving usability of anti-phishing toolbars, including three key components of anti-phishing client side applications (main user interface, critical warnings and the help system). For example, we found that in the main user interface it is important to keep the user informed and organize settings accordingly to a proper usability design. In addition, all the critical warnings an anti-phishing toolbar shows should be well designed. Furthermore, we found that the help system should be built to assist users to learn about phishing prevention as well as how to identify fraud attempts by themselves. One result of our research is also a classification of anti-phishing toolbar applications. Linfeng Li is a student at the University of Tampere, Finland. Marko Helenius is Assistant Professor at the Department of Computer Sciences, University of Tampere, Finland.     

Tracking the effectiveness of usability evaluation methods

. We present a case study that tracks usability problems predicted with six usability evaluation methods (claims analysis, cognitive walkthrough, GOMS, heuristic evaluation, user action notation, and simply reading the specification) through a development process. We assess the method's predictive power by comparing the predictions to the results of user tests. We assess the method's persuasive power by seeing how many problems led to changes in the implemented code. We assess design-change effectiveness by user testing the resulting new versions of the system. We concludethatpredictivemethodsarenot as effective as the HCI field would like and discuss directions for future research.     

Interface-usability evaluation: science of trade-offs

The effectiveness of four techniques in evaluating the usability of a graphical user interface is presented. The techniques are heuristic evaluation, usability testing, guidelines, and cognitive walkthrough. The techniques are compared as to the number, type, and severity of the problems each could identify for a specific product.<>     

A usability study of multimodal input in an augmented reality environment

In this paper, we describe a user study evaluating the usability of an augmented reality (AR) multimodal interface (MMI). We have developed an AR MMI that combines free-hand gesture and speech input in a natural way using a multimodal fusion architecture. We describe the system architecture and present a study exploring the usability of the AR MMI compared with speech-only and 3D-hand-gesture-only interaction conditions. The interface was used in an AR application for selecting 3D virtual objects and changing their shape and color. For each interface condition, we measured task completion time, the number of user and system errors, and user satisfactions. We found that the MMI was more usable than the gesture-only interface conditions, and users felt that the MMI was more satisfying to use than the speech-only interface conditions; however, it was neither more effective nor more efficient than the speech-only interface. We discuss the implications of this research for designing AR MMI and outline directions for future work. The findings could also be used to help develop MMIs for a wider range of AR applications, for example, in AR navigation tasks, mobile AR interfaces, or AR game applications.     

The influence of a knowledge-based system on designers' cognitive activities: a study involving professional web designers

Over the last few decades, several usability knowledge-based systems have been developed to provide user interface designers with usability knowledge (e.g. heuristics, usability guidelines, standards). Such systems are intended to assist designers during the design process, and to improve the usability of the user interface being designed. However, the assumption that such systems actually improve the usability of the resulting user interface remains to be demonstrated: virtually no systems have been empirically tested by designers who create products. In order to confirm this assumption, we conducted an experimental study in which professional web designers had to create webpages, either using a knowledge-based system – MetroWeb – or without it. This study was intended to determine the influence of MetroWeb on the professional web designers' cognitive activity, and to find out whether MetroWeb actually assists them to develop a user-centred design. The results show that the web designers did not very often use MetroWeb while designing webpages; however, rather surprisingly, the designers who did use MetroWeb more often exhibited a user-centred activity than those working without MetroWeb. We conclude this paper by discussing these findings, and suggesting future possible ways of research intended to assist designers to adopt a user-centred approach to their activity.     

一种可行的可用性测试过程

文章综述用户模型法、用户调查法、专家评审法和观察法4类可用性测试方法,根据可用性测试的经验,总结出一套合理实用的可用性测试过程,使得可用性测试的效率高、代价低,尽可能发现更多的可用性问题;提出在程序设计类课程的教学中,应适当向学生介绍可用性测试的方法和过程,以培养学生对自己开发的软件系统进行可用性测试的能力。