Developing usability heuristics with PROMETHEUS: A case study in virtual learning environments

Heuristic evaluation is one of the most widely-used methods for evaluating the usability of a software product. Proposed in 1990 by Nielsen and Molich, it consists in having a small group of evaluators performing a systematic revision of a system under a set of guiding principles known as usability heuristics. Although Nielsen’s 10 usability heuristics are used as the de facto standard in the process of heuristic evaluation, recent research has provided evidence not only for the need of custom domain specific heuristics, but also for the development of methodological processes to create such sets of heuristics. In this work we apply the PROMETHEUS methodology, recently proposed by the authors, to develop the VLEs heuristics: a novel set of usability heuristics for the domain of virtual learning environments. In addition to the development of these heuristics, our research serves as further empirical validation of PROMETHEUS. To validate our results we performed an heuristic evaluation using both VLEs and Nielsen’s heuristics. Our design explicitly controls the effect of evaluator variability by using a large number of evaluators. Indeed, for both sets of heuristics the evaluation was performed independently by 7 groups of 5 evaluators each. That is, there were 70 evaluators in total, 35 using VLEs and 35 using Nielsen’s heuristics. In addition, we perform rigorous statistical analyses to establish the validity of the novel VLEs heuristics. The results show that VLEs perform better than Nielsen’s heuristics, finding more problems, which are also more relevant to the domain, as well as satisfying other quantitative and qualitative criteria. Finally, in contrast to evaluators using Nielsen’s heuristics, evaluators using VLEs heuristics reported greater satisfaction regarding utility, clarity, ease of use, and need of additional elements.     

Developing SMASH: A set of SMArtphone's uSability Heuristics

The smartphone market is nowadays highly competitive. When buying a new device, users focus on visual esthetics, ergonomics, performance, and user experience, among others. Assessing usability issues allows improving these aspects. One popular method for detecting usability problems is heuristic evaluation, in which evaluators employ a set of usability heuristics as guide. Using proper heuristics is highly relevant. In this paper we present SMASH, a set of 12 usability heuristics for smartphones and mobile applications, developed iteratively. SMASH (previously named TMD: Usability heuristics for Touchscreen-based Mobile Devices) was experimentally validated. The results support its utility and effectiveness.     

What users really think about the usability of smartphone applications: diversity based empirical investigation

In recent years, smartphone devices are becoming progressively popular across a diverse range of users. However, user diversity creates challenges in smartphone application (app) development. The diversity of users is often ignored by designers and developers due to the absence of requirements. Owing to this, many smartphone users face usability issues. Despite that, no dedicated platform found that guide smartphone app designers and developers regarding human universality. The aim of this research is to explore the requirements of diverse users in smartphone apps and provide usability guidelines. The objectives of this research are achieved by following two scientific approaches. The human diversity requirements are located by conducting usability tests that investigated the requirements in the form of usability issues. The systematic literature review (SLR) process is followed in order to resolve the discovered usability issues. Both approaches resulted in a list of usability issues and guidelines. The usability tests returned 27 problems while the SLR came with a comprehensive set of universal usability guidelines that were grouped into eleven categories. The study concluded with some major outcomes. The results show evidence of critical usability problems that must be addressed during the design and development of smartphone apps. Moreover, the study also revealed that people with disabilities were three times severely affected by usability problems in such apps than people of different ages and their needs must be considered a top priority in the development of smartphone apps.

     

A Systematic and Generalizable Approach to the Heuristic Evaluation of User Interfaces

Heuristic evaluation is one of the most actively used techniques for analyzing usability, as it is quick and inexpensive. This technique is based on following a given set of heuristics, which are typically defined as broad rules of thumb. In this paper, we propose a systematic and generalizable approach to this type of evaluation based on using comprehensive taxonomies as a source for the heuristics. This approach contrasts with other typical approaches, such as following (or adapting) Jakob Nielsen’s heuristics or creating ad hoc heuristics (formally or informally). The usefulness of our approach is investigated in two ways. Firstly, we carry out an actual heuristic evaluation of a mobile app in this manner, which we describe in detail. Secondly, we compare our approach and Nielsen’s. Additionally, we identify some limitations in Nielsen’s heuristics and some inconsistencies between them and established usability models, including Nielsen’s own.     

A Study of the Evaluator Effect in Usability Testing

ABSTRACT

The evaluator effect names the observation that usability evaluators in similar conditions identify substantially different sets of usability problems. Yet little is known about the factors involved in the evaluator effect. We present a study of 50 novice evaluators' usability tests and subsequent comparisons, in teams and individually, of the resulting usability problems. The same problems were analyzed independently by 10 human–computer interaction experts. The study shows an agreement between evaluators of about 40%, indicating a substantial evaluator effect. Team matching of problems following the individual matching appears to improve the agreement, and evaluators express greater satisfaction with the teams' matchings. The matchings of individuals, teams, and independent experts show evaluator effects of similar sizes; yet individuals, teams, and independent experts fundamentally disagree about which problems are similar. Previous claims in the literature about the evaluator effect are challenged by the large variability in the matching of usability problems; we identify matching as a key determinant of the evaluator effect. An alternative view of usability problems and evaluator agreement is proposed in which matching is seen as an activity that helps to make sense of usability problems and where the existence of a correct matching is not assumed.     

Usability Principles for Augmented Reality Applications in a Smartphone Environment

Through the rapid spread of smartphones, users have access to many types of applications similar to those on desktop computer systems. Smartphone applications using augmented reality (AR) technology make use of users' location information. As AR applications will require new evaluation methods, improved usability and user convenience should be developed. The purpose of the current study is to develop usability principles for the development and evaluation of smartphone applications using AR technology. We develop usability principles for smartphone AR applications by analyzing existing research about heuristic evaluation methods, design principles for AR systems, guidelines for handheld mobile device interfaces, and usability principles for the tangible user interface. We conducted a heuristic evaluation for three popularly used smartphone AR applications to identify usability problems. We suggested new design guidelines to solve the identified problems. Then, we developed an improved AR application prototype of an Android-based smartphone, which later was conducted a usability testing to validate the effects of usability principles.     

Better discount evaluation: illustrating how critical parameters support heuristic creation

This paper describes a heuristic creation process based on the notion of critical parameters, and a comparison experiment that demonstrates the utility of heuristics created for a specific system class. We focus on two examples of using the newly created heuristics to illustrate the utility of the usability evaluation method, as well as to provide support for the creation process, and we report on successes and frustrations of two classes of users, novice evaluators and domain experts, who identified usability problems with the new heuristics. We argue that establishing critical parameters for other domains will support efforts in creating tailored evaluation tools.     

Correlation between safety assessments in the driver-car interaction design process

With the functional revolution in modern cars, evaluation methods to be used in all phases of driver–car interaction design have gained importance. It is crucial for car manufacturers to discover and solve safety issues early in the interaction design process. A current problem is thus to find a correlation between the formative methods that are used during development and the summative methods that are used when the product has reached the customer. This paper investigates the correlation between efficiency metrics from summative and formative evaluations, where the results of two studies on sound and navigation system tasks are compared. The first, an analysis of the J.D. Power and Associates APEAL survey, consists of answers given by about two thousand customers. The second, an expert evaluation study, was done by six evaluators who assessed the layouts by task completion time, TLX and Nielsen heuristics. The results show a high degree of correlation between the studies in terms of task efficiency, i.e. between customer ratings and task completion time, and customer ratings and TLX. However, no correlation was observed between Nielsen heuristics and customer ratings, task completion time or TLX. The results of the studies introduce a possibility to develop a usability evaluation framework that includes both formative and summative approaches, as the results show a high degree of consistency between the different methodologies. Hence, combining a quantitative approach with the expert evaluation method, such as task completion time, should be more useful for driver–car interaction design.     

Text Input on a Smartwatch QWERTY Keyboard: Tap vs. Trace

This study examines text input performance on a smartwatch using tap and trace input methods on a standard QWERTY keyboard (Swype^TM). Participants were either experts or novices to the tracing technique on their smartphone. No users had experience typing on a smartwatch. Participants were able to type at speeds comparable to, or exceeding, those reported in the literature for smartphones and small screen devices. Both novices and tracing experts typed faster overall using the trace input method than the tap method and experts typed the fastest using trace (37 WPM). Word error rates were also comparable to those reported for smartphone text input. These results suggest that using a standard QWERTY keyboard that allows both tap and trace for text input is a viable option on a smartwatch.     

Design of trustworthy smartphone-based multimedia services in cultural environments

Smartphone is a dynamic new media that faces high popularity due to its versatile services and the friendliness of its usage. It can be used in many activities of everyday life from e-commerce to e-tourism. In this work, we study smartphone’s secure usability in cultural heritage sites and environments. Our goal is to make a first attempt towards a trustworthy commercial multimedia guiding system targeting cultural sites that will be executed in a set of smartphones. More specifically, we are interested in how the needs of curators and visitors, experts or not, of a cultural heritage site can be facilitated by the provided multimedia guiding services of smartphones employing trustworthy implementations of smartphone services that are controlled by a central server. Furthermore, we make an attempt to propose a simple business model for the commercial exploitation of such services.     

Smartphone Guide Technology in Cultural Spaces: Measuring Visitor Experience with an iPhone Multimedia Guide in Shakespeare’s Church

Smartphone technologies are now used to deploy audio and multimedia guides in cultural spaces, including historic churches. It is important to measure what effect the use of such technologies has on visitor experience of the cultural space and the usability and user experience of the guide. An “in the wild” study was conducted to investigate visitor experience in a historic church, with two versions of a multimedia iPhone Guide and with a traditional paper guide. The Church Experience Scale (CES) and the Multimedia Guide Scale (MMGS) were used to measure the experience of 59 visitors to Holy Trinity Church, Stratford-upon-Avon, known as Shakespeare’s church. A total of 40 visitors used an iPhone guide, 21 a free choice version and 19 a guided tour version, and 19 visitors used a paper guide. Results showed that participants with a smartphone guide had a significantly more positive visitor experience and spent significantly longer on their visit to the church. There was a significant correlation between length of visit and the Enjoyment, Intellectual Stimulation, and Curiosity factor of the CES, but further work is needed to understand the direction of causality in this relationship. The usability and user experience of the multimedia guide conditions did not differ from each other, but the results of the MMGS showed that both guides could be improved in terms of general usability and quality of interaction. Challenges of conducting a study “in the wild” of a smartphone app in a cultural space are discussed.     

Usability guideline for banking software design

As the diversity of services in the financial market increases, it is critical to design usable banking software in order to overcome the complex structure of the system. The current study presents a usability guideline based on heuristics and their corresponding criteria that could be used during the early stages of banking software design process. In the design of a usability guideline, the heuristics and their criteria are categorized in terms of their effectiveness in solving usability problems grouped and ranging from usability catastrophe to cosmetic problems. The current study comprises of three main steps: First, actual usability problems from three banking software development projects are categorized according to their severity level. Secondly, usability criteria are rated for how well they explain the usability problems encountered. Finally, usability heuristics are categorized according to the severity level of usability problems through two analytical models; corresponding and cluster analyses. As the result, designers and project managers may give more importance to the heuristics related with the following usability problem categories: Usability catastrophe and then major usability problems. Furthermore, the proposed guideline can be used to understand which usability criteria would be helpful in explaining usability problems as well as preventing banking system catastrophes, by highlighting the critical parts in system design of banking software.     

The influence of user expertise and prototype fidelity in usability tests

An empirical study examined the impact of user expertise and prototype fidelity on the outcomes of a usability test. User expertise (expert vs. novice) and prototype fidelity (paper prototype, 3D mock-up, and fully operational appliance) were manipulated as independent variables in a 2 × 3 between-subjects design. Employing a floor scrubber as a model product, 48 users carried out several cleaning tasks. Usability problems identified by participants were recorded. Furthermore, performance, system management strategies and perceived usability were measured. The results showed that experts reported more usability problems than novices but these were considered to be less severe than those reported by novices. Reduced fidelity prototypes were generally suitable to predict product usability of the real appliance. The implications for the running of usability tests are specific to the fidelity of the prototype.     

Argumentation Models for Usability Problem Analysis in Individual and Collaborative Settings

Consolidating usability problems (UPs) from problem lists from several users can be a cognitively demanding task for evaluators. It has been suggested that collaboration between evaluators can help this process. In an attempt to learn how evaluators make decisions in this process, the authors studied what justification evaluators give for extracting UPs and their consolidation when working both individually and collaboratively. An experiment with eight novice usability evaluators was carried out where they extracted UPs and consolidated them individually and then collaboratively. The data were analyzed by using conventional content analysis and by creating argumentation models according to the Toulmin model. The results showed that during UP, extraction novice usability evaluators could put forward warrants leading to clear claims when probed but seldom added qualifiers or rebuttals. Novice usability evaluators could identify predefined criteria for a UP when probed and this could be acknowledged as a backing to warrants. In the individual settings, novice evaluators had difficulty in presenting claims and warrants for their decisions on consolidation. Although further study is needed, the results of the study indicated that collaborating pairs had a tendency to argue slightly better than individuals. Through the experiment novice evaluators’ reasoning patterns during problem extraction and consolidation as well as during their assessment of severity and confidence could be identified.     

Heuristic evaluation: Comparing ways of finding and reporting usability problems

Research on heuristic evaluation in recent years has focused on improving its effectiveness and efficiency with respect to user testing. The aim of this paper is to refine a research agenda for comparing and contrasting evaluation methods. To reach this goal, a framework is presented to evaluate the effectiveness of different types of support for structured usability problem reporting. This paper reports on an empirical study of this framework that compares two sets of heuristics, Nielsen’s heuristics and the cognitive principles of Gerhardt-Powals, and two media of reporting a usability problem, i.e. either using a web tool or paper. The study found that there were no significant differences between any of the four groups in effectiveness, efficiency and inter-evaluator reliability. A more significant contribution of this research is that the framework used for the experiments proved successful and should be reusable by other researchers because of its thorough structure.     

A prediction and auto-execution system of smartphone application services based on user context-awareness

Smartphones have emerged as suitable environments for user context-awareness and intelligent service provision due to the high penetration rate, the high usability, various embedded sensors, and so on. In particular, its most unique characteristic is the usage of various applications. However, the most of existing studies through the three steps process (log collection, context inference, and service provision) did not consider smartphone applications (Apps) as the target service. Smartphone users still have to use Apps with manual controls by own decision. Therefore, in this paper, we propose a system to predict smartphone applications based on inferring user context. We define a mobile context model with a new level of context (Situation) and its inference method to perceive a user’s intention or purpose related to the App usage. Based on the Situation context, the system predicts Apps which can be useful and helpful for a user and automatically executes it on his/her smartphone. With the proposed system, it will be possible to autonomously provide and manage smartphone application services without users’ perception or intervention.     

A systematic examination of universal design resources: part 1, heuristic evaluation

This paper presents the evaluation of eight published Universal Design Resources (UDRs) to measure how effectively they support typical design processes and design psychology. New heuristics and principles to evaluate the UDRs from the point of view of designers who were universal design novices were created. Established methodologies for heuristic evaluation were used with the new heuristics. The evaluators found numerous problems in seven of the eight UDRs, providing evidence in support of the hypothesis that the content UDRs does not facilitate the design process and is not commensurate with what is known about typical design psychology.     

The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods

Computer professionals have a need for robust, easy-to-use usability evaluation methods (UEMs) to help them systematically improve the usability of computer artifacts. However, cognitive walkthrough (CW), heuristic evaluation (HE), and thinking- aloud study (TA)-3 of the most widely used UEMs-suffer from a substantial evaluator effect in that multiple evaluators evaluating the same interface with the same UEM detect markedly different sets of problems. A review of 11 studies of these 3 UEMs reveals that the evaluator effect exists for both novice and experienced evaluators, for both cosmetic and severe problems, for both problem detection and severity assessment, and for evaluations of both simple and complex systems. The average agreement between any 2 evaluators who have evaluated the same system using the same UEM ranges from 5% to 65%, and no 1 of the 3 UEMs is consistently better than the others. Although evaluator effects of this magnitude may not be surprising for a UEM as informal as HE, it is certainly notable that a substantial evaluator effect persists for evaluators who apply the strict procedure of CW or observe users thinking out loud. Hence, it is highly questionable to use a TA with 1 evaluator as an authoritative statement about what problems an interface contains. Generally, the application of the UEMs is characterized by (a) vague goal analyses leading to variability in the task scenarios, (b) vague evaluation procedures leading to anchoring, or (c) vague problem criteria leading to anything being accepted as a usability problem, or all of these. The simplest way of coping with the evaluator effect, which cannot be completely eliminated, is to involve multiple evaluators in usability evaluations.     

Problem Prioritization in Usability Evaluation: From Severity Assessments Toward Impact on Design

Severity assessments enable prioritization of problems encountered during usability evaluations, and thereby provide a device for guiding the utilization of design resources. However, designers' response to usability evaluations is also influenced by other factors, which may overshadow severity. With the purpose of enhancing the impact of severity assessments, this study combines a field study of factors that influence the impact of evaluations with an experimental study of severity assessments made during usability inspections. The results show that even in a project receptive to input from evaluations, their impact was highly dependent on conducting evaluations early. This accorded with an informal method that blended elements of usability evaluation and participatory design and could be extended with user-made severity assessments. The major cost associated with the evaluations was not finding but fixing problems, emphasizing that, to be effective, severity assessments must be reliable, valid, and sufficiently persuasive to justify the cost of fixing problems. For the usability inspections, evaluators' ratings of problem impact and persistence were weakly correlated with the number of evaluators reporting a problem, indicating that different evaluators represent different subgroups of users or alternatively that evaluator-made severity assessments are of questionable reliability. To call designers' attention to the severe problems, the halving of the severity sum is proposed as a means of visualizing the large payoff of fixing a high-severity problem and, conversely, the modest potential of spending resources on low-severity problems.     

Evaluating visualizations: do expert reviews work?

Visualization research generates beautiful images and impressive interactive systems. Emphasis on evaluating visualizations is growing. Researchers have successfully used alternative evaluation techniques in human-computer interaction (HCI), including focus groups, field studies, and expert reviews. These methods tend to produce qualitative results and require fewer participants than controlled experiments. In this article, we focus on expert reviews that we used for the applications. We commonly use expert reviews to assess interface usability. Expert reviews can generate valuable feedback on visualization tools. We recommend i) including experts with experience in data display as well as usability, and ii) developing heuristics based on visualization guidelines as well as usability guidelines. Expert reviews should not be used exclusively, since experts might not hilly predict end-user actions. Furthermore, we encourage more experimentation with this technique, particularly to develop a good set of visualization heuristics and to compare it with other methods.