Developing usability heuristics with PROMETHEUS: A case study in virtual learning environments

Heuristic evaluation is one of the most widely-used methods for evaluating the usability of a software product. Proposed in 1990 by Nielsen and Molich, it consists in having a small group of evaluators performing a systematic revision of a system under a set of guiding principles known as usability heuristics. Although Nielsen’s 10 usability heuristics are used as the de facto standard in the process of heuristic evaluation, recent research has provided evidence not only for the need of custom domain specific heuristics, but also for the development of methodological processes to create such sets of heuristics. In this work we apply the PROMETHEUS methodology, recently proposed by the authors, to develop the VLEs heuristics: a novel set of usability heuristics for the domain of virtual learning environments. In addition to the development of these heuristics, our research serves as further empirical validation of PROMETHEUS. To validate our results we performed an heuristic evaluation using both VLEs and Nielsen’s heuristics. Our design explicitly controls the effect of evaluator variability by using a large number of evaluators. Indeed, for both sets of heuristics the evaluation was performed independently by 7 groups of 5 evaluators each. That is, there were 70 evaluators in total, 35 using VLEs and 35 using Nielsen’s heuristics. In addition, we perform rigorous statistical analyses to establish the validity of the novel VLEs heuristics. The results show that VLEs perform better than Nielsen’s heuristics, finding more problems, which are also more relevant to the domain, as well as satisfying other quantitative and qualitative criteria. Finally, in contrast to evaluators using Nielsen’s heuristics, evaluators using VLEs heuristics reported greater satisfaction regarding utility, clarity, ease of use, and need of additional elements.     

Information system professionals’ knowledge and application gaps toward Web design guidelines

Web design guidelines are adopted by many usability evaluation methods as one of the criteria for success, while usability is proven to significantly impact Website performance. Since Web design guidelines cover a broad range of system and interface design solutions, knowledge of them can be considered as a prominent indicator of Web design skills for information systems (IS) professionals. This study empirically assessed how much IS professionals know and apply Web design guidelines via a survey to 500 randomly selected companies from Taiwan’s Fortune 2000 corporations. As expected, the knowledge–application gaps of IS professionals were statistically significant in all Web design guideline categories. Meanwhile, certain guideline categories were proven to be more difficult to acquire or apply than others. Finally, degree, gender, experience, training hours, and courses taken were also proven to be determining factors for Web design guideline skills. Implications for developing Web design guideline skills are also discussed.     

Expert vs novice collaborative heuristic evaluation (CHE) of a smartphone app for cultural heritage sites

This study was conducted to compare CHE between Human-Computer Interaction (HCI) experts and novices in evaluating the Smartphone app for the cultural heritage site. It uses the Smartphone Mobile Application heuRisTics (SMART), focusing on smartphone applications and traditional Nielsen heuristics, focusing on a wider range of interactive systems. Six experts and six novices used the severity rating scale to categorise the severity of the usability issues. These issues were mapped to both heuristics. The study found that experts and novice evaluators identified 19 and 14 usability issues, respectively, with ten as the same usability issues. However, these same usability issues have been rated differently. Although the t-test indicates no significant differences between experts and novices in their ratings for usability issues, these results nevertheless indicate the need for both evaluators in CHE to provide a more comprehensive perspective on the severity of the usability issues. Furthermore, the mapping of the usability issues for Nielsen and SMART heuristics concluded that more issues with the smartphone app could be addressed through smartphone-specific heuristics than general heuristics, indicating a better tool for heuristic evaluation of the smartphone app. This study also provides new insight into the required number of evaluators needed for CHE.

     

A framework for evaluating the usability of mobile phones based on multi-level, hierarchical model of usability factors

As a mobile phone has various advanced functionalities or features, usability issues are increasingly challenging. Due to the particular characteristics of a mobile phone, typical usability evaluation methods and heuristics, most of which are relevant to a software system, might not effectively be applied to a mobile phone. Another point to consider is that usability evaluation activities should help designers find usability problems easily and produce better design solutions. To support usability practitioners of the mobile phone industry, we propose a framework for evaluating the usability of a mobile phone, based on a multi-level, hierarchical model of usability factors, in an analytic way. The model was developed on the basis of a set of collected usability problems and our previous study on a conceptual framework for identifying usability impact factors. It has multi-abstraction levels, each of which considers the usability of a mobile phone from a particular perspective. As there are goal-means relationships between adjacent levels, a range of usability issues can be interpreted in a holistic as well as diagnostic way. Another advantage is that it supports two different types of evaluation approaches: task-based and interface-based. To support both evaluation approaches, we developed four sets of checklists, each of which is concerned, respectively, with task-based evaluation and three different interface types: Logical User Interface (LUI), Physical User Interface (PUI) and Graphical User Interface (GUI). The proposed framework specifies an approach to quantifying usability so that several usability aspects are collectively measured to give a single score with the use of the checklists. A small case study was conducted in order to examine the applicability of the framework and to identify the aspects of the framework to be improved. It showed that it could be a useful tool for evaluating the usability of a mobile phone. Based on the case study, we improved the framework in order that usability practitioners can use it more easily and consistently.     

Better discount evaluation: illustrating how critical parameters support heuristic creation

This paper describes a heuristic creation process based on the notion of critical parameters, and a comparison experiment that demonstrates the utility of heuristics created for a specific system class. We focus on two examples of using the newly created heuristics to illustrate the utility of the usability evaluation method, as well as to provide support for the creation process, and we report on successes and frustrations of two classes of users, novice evaluators and domain experts, who identified usability problems with the new heuristics. We argue that establishing critical parameters for other domains will support efforts in creating tailored evaluation tools.     

What you get is what you see: revisiting the evaluator effect in usability tests

Usability evaluation is essential to user-centred design; yet, evaluators who analyse the same usability test sessions have been found to identify substantially different sets of usability problems. We revisit this evaluator effect by having 19 experienced usability professionals analyse video-recorded test sessions with five users. Nine participants analysed moderated sessions; 10 participants analysed unmoderated sessions. For the moderated sessions, participants reported an average of 33% of the problems reported by all nine of these participants and 50% of the subset of problems reported as critical or serious by at least one participant. For the unmoderated sessions, the percentages were 32% and 40%. Thus, the evaluator effect was similar for moderated and unmoderated sessions, and it was substantial for the full set of problems and still present for the most severe problems. In addition, participants disagreed in their severity ratings. As much as 24% (moderated) and 30% (unmoderated) of the problems reported by multiple participants were rated as critical by one participant and minor by another. The majority of the participants perceived an evaluator effect when merging their individual findings into group evaluations. We discuss reasons for the evaluator effect and recommend ways of managing it.     

A Usable Security Heuristic Evaluation for the Online Health Social Networking Paradigm

The security, privacy, and usability of health social networking websites remain key concerns for their successful adoption by patients. As far as it is known, no studies have developed a usable security heuristic evaluation to test these websites for the usability of their security and privacy features. This article describes the development of such a usable security heuristic evaluation in the context of the online health social networking paradigm. The tailored method approach was applied to transform themes identified from relevant literature and convert them into high-level heuristics for usable security. Checklist items were created for each high-level heuristic. The result is 13 high-level heuristics with individualized checklist items that help examine usable security. The results can identify which security and privacy features are lacking from a usability perspective and would, therefore, require design improvement. The improvements would enhance the users' experiences, which in turn can assist in the adoption of health social networking websites.     

Systematic evaluation of design choices for software development tools

Most design and evaluation of software tools is based on the intuition and experience of the designers. Software tool designers consider themselves typical users of the tools that they build and tend to subjectively evaluate their products rather than objectively evaluate them using established usability methods. This subjective approach is inadequate if the quality of software tools is to improve and the use of more systematic methods is advocated. This paper summarises a sequence of studies that show how user interface design choices for software development tools can be evaluated using established usability engineering techniques. The techniques used included guideline review, predictive modelling and experimental studies with users.     

Problem Prioritization in Usability Evaluation: From Severity Assessments Toward Impact on Design

Severity assessments enable prioritization of problems encountered during usability evaluations, and thereby provide a device for guiding the utilization of design resources. However, designers' response to usability evaluations is also influenced by other factors, which may overshadow severity. With the purpose of enhancing the impact of severity assessments, this study combines a field study of factors that influence the impact of evaluations with an experimental study of severity assessments made during usability inspections. The results show that even in a project receptive to input from evaluations, their impact was highly dependent on conducting evaluations early. This accorded with an informal method that blended elements of usability evaluation and participatory design and could be extended with user-made severity assessments. The major cost associated with the evaluations was not finding but fixing problems, emphasizing that, to be effective, severity assessments must be reliable, valid, and sufficiently persuasive to justify the cost of fixing problems. For the usability inspections, evaluators' ratings of problem impact and persistence were weakly correlated with the number of evaluators reporting a problem, indicating that different evaluators represent different subgroups of users or alternatively that evaluator-made severity assessments are of questionable reliability. To call designers' attention to the severe problems, the halving of the severity sum is proposed as a means of visualizing the large payoff of fixing a high-severity problem and, conversely, the modest potential of spending resources on low-severity problems.     

Evaluation of user interfaces: EVADIS II— a comprehensive evaluation approach

As a result of the importance of the usability approach in system development and the EC's 'Directive concerning the minimum safety and health requirements for VDT workers' (EWG 1990), there is an accepted need for practical evaluation methods for user interfaces. The usability approach and the EC Directive are not restricted to user interface design, as they include the design of appropriate hardware and software, as well as organization, job, and task design. Therefore system designers are faced with many, often conflicting, requirements and need to address the question, 'How can usability requirements comprehensively be considered and evaluated in system development?' Customers buying hardware and software and introducing them into their organization ask, (How can I select easy-to-use hardware and software?' Both designers and customers need an evaluation procedure that covers all the organizational, user, hard- and software requirements. The evaluation method, EVADIS.II, we present in this paper overcomes characteristic deficiencies of previous evaluation methods. In particular, it takes the tasks, the user, and the organizational context into consideration during the evaluation process, and provides computer support for the use of the evaluation procedure.     

Evidence for an elders' advantage in the naive product usability judgments of older and younger adults

OBJECTIVE: To determine whose naive judgments of consumer product usability are more accurate--those of younger or older adults. Accuracy is here defined as judgments compatible with results from performance-based usability tests. BACKGROUND: Older adults may be better able to predict usability problems than younger adults, making them particularly good participants in studies contributing to the user-centered design of products. This advantage, if present, may stem from older adults' motivation for more usable products or from their experience adapting their own environments to meet their changing physical, cognitive, and sensory needs. METHOD: Sixty older participants (ages 65-75 years) and 60 younger ones (ages 18-22 years) evaluated illustrations of consumer products on specific criteria (e.g., readability, learnability, or error rates). They either rated a single design for each product or ranked six alternative designs. They also explained their choices, indicated which features were most critical for usability, and selected usability-enhancing modifications. RESULTS: Although there was no reliable age difference in the amount of usability information provided in the open-ended explanations, older adults were more accurate at ranking alternative designs, selecting the most usability-critical features, and selecting usability-enhancing modifications (all ps < .05). CONCLUSION: The usability judgments of older adults are more accurate than those of younger adults when these judgments are solicited in a fixed-alternative, but not open-ended, format. APPLICATION: Because older adults are more discerning about potential product usability problems, they may be particularly valuable as research participants in early-stage design research (prior to the availability of working prototypes).     

The most important design guideline? [user interfaces]

The activity of "design" includes many things, but certainly one of the most important aspects is interface specification. Interfaces determine which aspects of a component are accessible and to whom; they thus determine encapsulation. Interfaces specify what functionality (data, properties, methods, and so forth) is available to clients. Interfaces reflect how a system is broken down into its constituent components. Regardless of our role in the development of a software system, it almost certainly involves some interface design, so it's helpful to have some heuristics that indicate when we're doing it well-or poorly. We've come to the conclusion that the most important general design guideline is interface should be easy to use correctly and hard to use incorrectly. This guideline leads to a conclusion that some developers find unsettling.     

Usability in agile software development: A tertiary study

In the last years the interest in developing research on integration of usability and agile software development has been increasing. The number of systematic literature reviews, systematic mapping studies and non-systematic reviews, related to this thematic has also increased. Nevertheless, there is no analysis on the quality of these published secondary studies, nor is there a consolidated research that brings the answer of how to integrate these two areas. The goal of this paper is to categorize secondary studies related to the integration of usability and agile software development and present a critical analysis on the quality of the selected studies. To accomplish this goal a tertiary study was performed to categorize the related studies selected. Initially 3,065 papers were identified and further narrowed to 14 by applying exclusion criteria and analysis. We classified the selected studies as systematic literature reviews, systematic mapping studies and non-systematic literature reviews to report the data analysis. As a result of this study different forms to integrate usability and agile software development were detected as well as the various challenges that must be overcome for the integration success. Six main categories were identified to represent ways of integrating usability into agile development: processes, techniques, practices, recommendations, principles and different approaches. Regarding to the challenges for the integration seven main categories were also identified: issues related to tests, time, work balance, modularization, feedback, prioritization, and documentation. Although the interest in researching the integration of usability and agile software development has increased in the last years, mostly of the analyzed studies neglected the quality criteria and presented difficulties to use methods to synthetize the research results. Despite this, it has been realized that the integration of usability with agile software development is possible and is strongly aligned with user-centered design. The initial studies indicated a separation of activities and roles into specific tracks with parallel work to treat usability in agile software development, but the trend is no longer to manage and control these activities in separate ways, so new challenges are becoming to appear. Although we have identified several points of tension, the integration does not become unfeasible.     

Menu — what menu?

In recent years information system interface design has become increasingly dominated by the use of menus, with the majority of systems relying on static menus as their main dialogue structure. Alternatives to this interface style are explored, and through the discussion of an application developed in banking a number of alternative styles which can be used are detailed. The usability and utility of this application is evaluated with positive results. It is suggested that while menus are a viable and useful interface design design technique, the use of other techniques should also be encouraged, and that small-scale innovations in interface design will improve the variety, usability and acceptability of information systems.     

中国农业银行网银表单的可用性问题研究

认知走查作为一种可用性研究方法,是专家将自己"扮演"成用户,通过一定任务对界面进行可用性评估。网上银行最基本的组成元素就是表单,其所有功能都通过用户与表单的交互来完成。本文就以认知走查为基本研究方法,设定一定的任务,查找现有中国农业银行网上银行(个人版)表单存在的可用性问题。最终将走查到的可用性问题汇总并提炼出具有一般指导意义的网银表单设计准则。     

Why usability gets lost or usability in in-house software development

This study tries to shed some light on what happens to usability and occupational health issues in a bespoke software development project. Usability is an essential quality in software, in particular in a work context where poor usability and other risk factors related to the software and computers may cause health problems. We have interviewed a number of software developers, usability people and users about their attitudes to and practices for integrating usability and users' health concerns in software development. The interviews were conducted in two Swedish organisations with in-house development of bespoke software. Our main conclusion is that several factors combine to push usability and occupational health matters aside, some of which are attitudes to usability and users' health issues, unclear responsibilities, poor support for user-centeredness and usability in software development models, ineffective user participation and usability and users' health being ignored or forgotten in decisions about the software, its use and its design.     

Criteria for the complementary allocation of functions in automated work systems and their use in simultaneous engineering projects

A crucial step in any system design concerns the allocation of functions between human and machine. In simultaneous engineering, function allocation is potentially an issue both in product design — if the product is a technical system itself — and in process design. It is argued that well-founded function allocation decisions in relation to the production process are of particular importance in simultaneous engineering as less and less time is allowed for compensation for inadequate or missing decisions which will impede production effectiveness. The focus of recent methods for the allocation of functions is to operationalize the concept of complementarity, meaning to provide guidelines for complementing humans by technical systems instead of replacing them gradually as technical systems become more and more sophisticated. As systems design becomes more complex, involving more and faster changes and placing higher demands on engineers as well as system operators, the idea of complementary gains importance because it stresses the need to allocate functions in a way that supports human control over the production process and the development and maintenance of the necessary skills. Looking at existing methods and instruments for the complementary allocation of functions, one is at first sight confronted with a multitude of criteria for complementarity, but to date neither a fixed set of criteria, comparable to software usability criteria, nor widely accepted methods for their measurement exist. As part of a research project concerned with the development of guidelines whose objective is to assist engineers in designing work systems according to the complementarity principle, four central criteria for complementary function allocation were identified: dynamic coupling, process transparency, human decision authority, and flexibility. These criteria and their application in a design process are illustrated by means of a case study and discussed in terms of other approaches to complementary design as well as their use for simultaneous engineering projects.

Relevance to industry

Adequate allocation of functions between human and machine is crucial for the effectiveness of any production system. With more complex and faster design processes in simultaneous engineering projects, the need for methods supporting prospective analysis and design of human-machine systems has increased even more. The article presents such a method based on the principle of complementarity between human operator and technical system and illustrates its use by means of a case study.     

User‐interface design heuristics for developing usable industrial hypermedia applications

Shop floor operators urgently need help to cope with the pressures of the present manufacturing context. One proposed solution is industrial hypermedia applications (IHA). A review of published papers on IHAs has shown that while the potential benefits of such applications, if accepted and used by operators, are well documented, user‐interface design aspects are not. In other words, the rationale used for the user‐interface design is not explained. This article fills the gap by drawing on human–computer interaction and hypermedia literature to present user‐interface design guidelines or heuristics. The latter were incorporated in 2 IHAs developed for this study. Shop floor operators empirically assessed the IHAs through usability trials held in the workplace and provided data on their perceptions of the IHA's ease of use. The findings show that the 7 design heuristics contributed to the usability of the IHAs. However, developers should not apply the design heuristics in isolation but should also elicit user characteristics and task domain. © 2002 Wiley Periodicals, Inc.     

Developing SMASH: A set of SMArtphone's uSability Heuristics

The smartphone market is nowadays highly competitive. When buying a new device, users focus on visual esthetics, ergonomics, performance, and user experience, among others. Assessing usability issues allows improving these aspects. One popular method for detecting usability problems is heuristic evaluation, in which evaluators employ a set of usability heuristics as guide. Using proper heuristics is highly relevant. In this paper we present SMASH, a set of 12 usability heuristics for smartphones and mobile applications, developed iteratively. SMASH (previously named TMD: Usability heuristics for Touchscreen-based Mobile Devices) was experimentally validated. The results support its utility and effectiveness.     

Evaluation of user interfaces: EVADIS II— a comprehensive evaluation approach

Abstract

As a result of the importance of the usability approach in system development and the EC's ‘Directive concerning the minimum safety and health requirements for VDT workers’ (EWG 1990), there is an accepted need for practical evaluation methods for user interfaces. The usability approach and the EC Directive are not restricted to user interface design, as they include the design of appropriate hardware and software, as well as organization, job, and task design. Therefore system designers are faced with many, often conflicting, requirements and need to address the question, ‘How can usability requirements comprehensively be considered and evaluated in system development?’ Customers buying hardware and software and introducing them into their organization ask, (How can I select easy-to-use hardware and software?’ Both designers and customers need an evaluation procedure that covers all the organizational, user, hard- and software requirements. The evaluation method, EVADIS.II, we present in this paper overcomes characteristic deficiencies of previous evaluation methods. In particular, it takes the tasks, the user, and the organizational context into consideration during the evaluation process, and provides computer support for the use of the evaluation procedure.