Applying a methodology to develop user eXperience heuristics

A heuristic evaluation method allows the evaluation of the usability of application domains. To evaluate applications that have specific domain features, researchers can use sets of specific usability heuristics in addition to the well-known (usually Nielsen's) heuristics. Heuristics can also focus on the User eXperience (UX) aspects other than the usability. In a previous work, we proposed a formal methodology for establishing usability/UX heuristics. The methodology has 8 stages including activities to formulate, specify, validate and refine a new set of heuristics for a specific application domain. The methodology was validated through expert opinion and several case studies. Although when specifying the methodology, we explained each of its stages in detail, some activities can be difficult to perform without a guide that helps the researcher determine how the stages should be carried out. This article presents a detailed explanation regarding how to apply each stage of the methodology to create a new set of heuristics for a specific domain. Additionally, this paper explains how to iterate the methodology's stages and when to stop the process of developing new heuristics.     

Developing usability heuristics with PROMETHEUS: A case study in virtual learning environments

Heuristic evaluation is one of the most widely-used methods for evaluating the usability of a software product. Proposed in 1990 by Nielsen and Molich, it consists in having a small group of evaluators performing a systematic revision of a system under a set of guiding principles known as usability heuristics. Although Nielsen’s 10 usability heuristics are used as the de facto standard in the process of heuristic evaluation, recent research has provided evidence not only for the need of custom domain specific heuristics, but also for the development of methodological processes to create such sets of heuristics. In this work we apply the PROMETHEUS methodology, recently proposed by the authors, to develop the VLEs heuristics: a novel set of usability heuristics for the domain of virtual learning environments. In addition to the development of these heuristics, our research serves as further empirical validation of PROMETHEUS. To validate our results we performed an heuristic evaluation using both VLEs and Nielsen’s heuristics. Our design explicitly controls the effect of evaluator variability by using a large number of evaluators. Indeed, for both sets of heuristics the evaluation was performed independently by 7 groups of 5 evaluators each. That is, there were 70 evaluators in total, 35 using VLEs and 35 using Nielsen’s heuristics. In addition, we perform rigorous statistical analyses to establish the validity of the novel VLEs heuristics. The results show that VLEs perform better than Nielsen’s heuristics, finding more problems, which are also more relevant to the domain, as well as satisfying other quantitative and qualitative criteria. Finally, in contrast to evaluators using Nielsen’s heuristics, evaluators using VLEs heuristics reported greater satisfaction regarding utility, clarity, ease of use, and need of additional elements.     

A Systematic and Generalizable Approach to the Heuristic Evaluation of User Interfaces

Heuristic evaluation is one of the most actively used techniques for analyzing usability, as it is quick and inexpensive. This technique is based on following a given set of heuristics, which are typically defined as broad rules of thumb. In this paper, we propose a systematic and generalizable approach to this type of evaluation based on using comprehensive taxonomies as a source for the heuristics. This approach contrasts with other typical approaches, such as following (or adapting) Jakob Nielsen’s heuristics or creating ad hoc heuristics (formally or informally). The usefulness of our approach is investigated in two ways. Firstly, we carry out an actual heuristic evaluation of a mobile app in this manner, which we describe in detail. Secondly, we compare our approach and Nielsen’s. Additionally, we identify some limitations in Nielsen’s heuristics and some inconsistencies between them and established usability models, including Nielsen’s own.     

Usability guideline for banking software design

As the diversity of services in the financial market increases, it is critical to design usable banking software in order to overcome the complex structure of the system. The current study presents a usability guideline based on heuristics and their corresponding criteria that could be used during the early stages of banking software design process. In the design of a usability guideline, the heuristics and their criteria are categorized in terms of their effectiveness in solving usability problems grouped and ranging from usability catastrophe to cosmetic problems. The current study comprises of three main steps: First, actual usability problems from three banking software development projects are categorized according to their severity level. Secondly, usability criteria are rated for how well they explain the usability problems encountered. Finally, usability heuristics are categorized according to the severity level of usability problems through two analytical models; corresponding and cluster analyses. As the result, designers and project managers may give more importance to the heuristics related with the following usability problem categories: Usability catastrophe and then major usability problems. Furthermore, the proposed guideline can be used to understand which usability criteria would be helpful in explaining usability problems as well as preventing banking system catastrophes, by highlighting the critical parts in system design of banking software.     

Expert vs novice collaborative heuristic evaluation (CHE) of a smartphone app for cultural heritage sites

This study was conducted to compare CHE between Human-Computer Interaction (HCI) experts and novices in evaluating the Smartphone app for the cultural heritage site. It uses the Smartphone Mobile Application heuRisTics (SMART), focusing on smartphone applications and traditional Nielsen heuristics, focusing on a wider range of interactive systems. Six experts and six novices used the severity rating scale to categorise the severity of the usability issues. These issues were mapped to both heuristics. The study found that experts and novice evaluators identified 19 and 14 usability issues, respectively, with ten as the same usability issues. However, these same usability issues have been rated differently. Although the t-test indicates no significant differences between experts and novices in their ratings for usability issues, these results nevertheless indicate the need for both evaluators in CHE to provide a more comprehensive perspective on the severity of the usability issues. Furthermore, the mapping of the usability issues for Nielsen and SMART heuristics concluded that more issues with the smartphone app could be addressed through smartphone-specific heuristics than general heuristics, indicating a better tool for heuristic evaluation of the smartphone app. This study also provides new insight into the required number of evaluators needed for CHE.

     

Evaluating visualizations: do expert reviews work?

Visualization research generates beautiful images and impressive interactive systems. Emphasis on evaluating visualizations is growing. Researchers have successfully used alternative evaluation techniques in human-computer interaction (HCI), including focus groups, field studies, and expert reviews. These methods tend to produce qualitative results and require fewer participants than controlled experiments. In this article, we focus on expert reviews that we used for the applications. We commonly use expert reviews to assess interface usability. Expert reviews can generate valuable feedback on visualization tools. We recommend i) including experts with experience in data display as well as usability, and ii) developing heuristics based on visualization guidelines as well as usability guidelines. Expert reviews should not be used exclusively, since experts might not hilly predict end-user actions. Furthermore, we encourage more experimentation with this technique, particularly to develop a good set of visualization heuristics and to compare it with other methods.     

A unified methodology for the evaluation of accessibility and usability of mobile applications

This article reports a unified methodology developed to evaluate the accessibility and usability of mobile computing applications, which is intended to guarantee universal access as far as possible. As a basis for the methodology, this paper presents an analysis of the accessibility guidelines, conducted to take into account the specificity of mobile systems, as well as a set of usability heuristics, specifically devised for mobile computing. Finally, it presents the results of the application of the proposed methodology to applications that have been semi-automatically developed by the MAIS Designer, a new design tool that provides applications suited to different mobile devices.     

A Heuristic Evaluation for Deaf Web User Experience (HE4DWUX)

Websites that are usable and accessible can have a positive impact on the overall user experience. Usability Inspection Methods (UIMs) can be applied to evaluate and measure the usability. The current research in the fields of Web Accessibility and Human –Computer Interaction (HCI) is in need of additional UIMs that can be applied to also measure the accessibility, in addition to the usability alone. In this article, a novel UIM in the form of a heuristic evaluation is presented. The heuristic evaluation aims to support HCI experts and Web developers in designing and evaluating websites that provide positive user experiences to users who are deaf. This article discusses the development of the Heuristic Evaluation for Deaf Web User Experience (HE4DWUX). Following an iteration cycle, version 2 of the HE4DWUX is presented in Appendix A. An existing three-phase process to develop heuristics for specific application domains was applied to construct the HE4DWUX. The outcome of this research is 12 heuristics, with each containing its own set of checklist items to operationalize its applicability in measuring the Web user experience for users who are deaf. The heuristics and their checklist items can identify important aspects of design that will impact the Web user experience for this particular user group.     

A Usable Security Heuristic Evaluation for the Online Health Social Networking Paradigm

The security, privacy, and usability of health social networking websites remain key concerns for their successful adoption by patients. As far as it is known, no studies have developed a usable security heuristic evaluation to test these websites for the usability of their security and privacy features. This article describes the development of such a usable security heuristic evaluation in the context of the online health social networking paradigm. The tailored method approach was applied to transform themes identified from relevant literature and convert them into high-level heuristics for usable security. Checklist items were created for each high-level heuristic. The result is 13 high-level heuristics with individualized checklist items that help examine usable security. The results can identify which security and privacy features are lacking from a usability perspective and would, therefore, require design improvement. The improvements would enhance the users' experiences, which in turn can assist in the adoption of health social networking websites.     

A framework for evaluating the usability of mobile phones based on multi-level, hierarchical model of usability factors

As a mobile phone has various advanced functionalities or features, usability issues are increasingly challenging. Due to the particular characteristics of a mobile phone, typical usability evaluation methods and heuristics, most of which are relevant to a software system, might not effectively be applied to a mobile phone. Another point to consider is that usability evaluation activities should help designers find usability problems easily and produce better design solutions. To support usability practitioners of the mobile phone industry, we propose a framework for evaluating the usability of a mobile phone, based on a multi-level, hierarchical model of usability factors, in an analytic way. The model was developed on the basis of a set of collected usability problems and our previous study on a conceptual framework for identifying usability impact factors. It has multi-abstraction levels, each of which considers the usability of a mobile phone from a particular perspective. As there are goal-means relationships between adjacent levels, a range of usability issues can be interpreted in a holistic as well as diagnostic way. Another advantage is that it supports two different types of evaluation approaches: task-based and interface-based. To support both evaluation approaches, we developed four sets of checklists, each of which is concerned, respectively, with task-based evaluation and three different interface types: Logical User Interface (LUI), Physical User Interface (PUI) and Graphical User Interface (GUI). The proposed framework specifies an approach to quantifying usability so that several usability aspects are collectively measured to give a single score with the use of the checklists. A small case study was conducted in order to examine the applicability of the framework and to identify the aspects of the framework to be improved. It showed that it could be a useful tool for evaluating the usability of a mobile phone. Based on the case study, we improved the framework in order that usability practitioners can use it more easily and consistently.     

Heuristic evaluation: Comparing ways of finding and reporting usability problems

Research on heuristic evaluation in recent years has focused on improving its effectiveness and efficiency with respect to user testing. The aim of this paper is to refine a research agenda for comparing and contrasting evaluation methods. To reach this goal, a framework is presented to evaluate the effectiveness of different types of support for structured usability problem reporting. This paper reports on an empirical study of this framework that compares two sets of heuristics, Nielsen’s heuristics and the cognitive principles of Gerhardt-Powals, and two media of reporting a usability problem, i.e. either using a web tool or paper. The study found that there were no significant differences between any of the four groups in effectiveness, efficiency and inter-evaluator reliability. A more significant contribution of this research is that the framework used for the experiments proved successful and should be reusable by other researchers because of its thorough structure.     

User‐interface design heuristics for developing usable industrial hypermedia applications

Shop floor operators urgently need help to cope with the pressures of the present manufacturing context. One proposed solution is industrial hypermedia applications (IHA). A review of published papers on IHAs has shown that while the potential benefits of such applications, if accepted and used by operators, are well documented, user‐interface design aspects are not. In other words, the rationale used for the user‐interface design is not explained. This article fills the gap by drawing on human–computer interaction and hypermedia literature to present user‐interface design guidelines or heuristics. The latter were incorporated in 2 IHAs developed for this study. Shop floor operators empirically assessed the IHAs through usability trials held in the workplace and provided data on their perceptions of the IHA's ease of use. The findings show that the 7 design heuristics contributed to the usability of the IHAs. However, developers should not apply the design heuristics in isolation but should also elicit user characteristics and task domain. © 2002 Wiley Periodicals, Inc.     

An interdisciplinary heuristic evaluation method for universal building design

This study highlights how heuristic evaluation as a usability evaluation method can feed into current building design practice to conform to universal design principles. It provides a definition of universal usability that is applicable to an architectural design context. It takes the seven universal design principles as a set of heuristics and applies an iterative sequence of heuristic evaluation in a shopping mall, aiming to achieve a cost-effective evaluation process. The evaluation was composed of three consecutive sessions. First, five evaluators from different professions were interviewed regarding the construction drawings in terms of universal design principles. Then, each evaluator was asked to perform the predefined task scenarios. In subsequent interviews, the evaluators were asked to re-analyze the construction drawings. The results showed that heuristic evaluation could successfully integrate universal usability into current building design practice in two ways: (i) it promoted an iterative evaluation process combined with multi-sessions rather than relying on one evaluator and on one evaluation session to find the maximum number of usability problems, and (ii) it highlighted the necessity of an interdisciplinary ad hoc committee regarding the heuristic abilities of each profession. A multi-session and interdisciplinary heuristic evaluation method can save both the project budget and the required time, while ensuring a reduced error rate for the universal usage of the built environments.     

Heuristics for the variable sized bin-packing problem

We investigate the one-dimensional variable-sized bin-packing problem. This problem requires packing a set of items into a minimum-cost set of bins of unequal sizes and costs. Six optimization-based heuristics for this problem are presented and compared. We analyze their empirical performance on a large set of randomly generated test instances with up to 2000 items and seven bin types. The first contribution of this paper is to provide evidence that a set covering heuristic proves to be highly effective and capable of delivering very-high quality solutions within short CPU times. In addition, we found that a simple subset-sum problem-based heuristic consistently outperforms heuristics from the literature while requiring extremely short CPU times.     

Extending the Usability of Heuristics for Design and Evaluation: Lead,Follow, and Get Out of the Way

User-centered design practitioners have often relied on discount usability engineering methods using heuristics. Top 10 lists of design and evaluation heuristics have proliferated during the 1990s, leading to a plethora of heterogeneous heuristic guidelines for a multitude of user interfaces. A simple, unified set of heuristics that might be applicable across different technologies, understood between disciplines, and equivalent to metrics used to measure ease of use can be beneficial.

Lead, follow, and get out of the way (LF&G) theorizes that the optimal human-computer interaction (HCI) experience is analogous to a facilitative learning relation-Like a good teacher, mentor, or coach, the usable user interface leads the user to successful completion of tasks and goals; follows the user's progress and provides appropriate feedback and information when needed; and gets out of the way of the user to allow efficient and effective completion of tasks as the user attains mastery of the system, its concepts, and operations. A set of 18 heuristics grouped under the 3 general principles of the title are provided for use across the tasks of design guidance, development refinement, and end-user evaluation of computer systems.

This article provides background into the development of LF&G, case studies on its use in real-world product design and development, and directions for further research needed to develop this approach into an empirically based method for defining, describing, designing, and predicting the ease of use of interactive human-computer systems.     

Better discount evaluation: illustrating how critical parameters support heuristic creation

This paper describes a heuristic creation process based on the notion of critical parameters, and a comparison experiment that demonstrates the utility of heuristics created for a specific system class. We focus on two examples of using the newly created heuristics to illustrate the utility of the usability evaluation method, as well as to provide support for the creation process, and we report on successes and frustrations of two classes of users, novice evaluators and domain experts, who identified usability problems with the new heuristics. We argue that establishing critical parameters for other domains will support efforts in creating tailored evaluation tools.     

Do patterns help novice evaluators? A comparative study

Evaluating e-learning systems is a complex activity which requires considerations of several criteria addressing quality in use as well as educational quality. Heuristic evaluation is a widespread method for usability evaluation, yet its output is often prone to subjective variability, primarily due to the generality of many heuristics. This paper presents the pattern-based (PB) inspection, which aims at reducing this drawback by exploiting a set of evaluation patterns to systematically drive inspectors in their evaluation activities. The application of PB inspection to the evaluation of e-learning systems is reported in this paper together with a study that compares this method to heuristic evaluation and user testing. The study involved 73 novice evaluators and 25 end users, who evaluated an e-learning application using one of the three techniques. The comparison metric was defined along six major dimensions, covering concepts of classical test theory and pragmatic aspects of usability evaluation. The study showed that evaluation patterns, capitalizing on the reuse of expert evaluators know-how, provide a systematic framework which reduces reliance on individual skills, increases inter-rater reliability and output standardization, permits the discovery of a larger set of different problems and decreases evaluation cost. Results also indicated that evaluation in general is strongly dependent on the methodological apparatus as well as on judgement bias and individual preferences of evaluators, providing support to the conceptualisation of interactive quality as a subjective judgement, recently brought forward by the UX research agenda.     

Writing Centers and Students with Disabilities: The User-centered Approach,Participatory Design,and Empirical Research as Collaborative Methodologies

This article discusses issues of accessibility and how user-centered and participatory approaches can inform empirical research to guide the Universal Design of virtual spaces and influence writing center efforts for students with disabilities. Because this article describes how to integrate usability/accessibility testing for online and in-person services, it can work as a model for writing centers struggling with the challenges of serving students with disabilities. Toward this end, the article discusses two generations of usability testing on a large, well-established online writing lab (the Purdue OWL), as well as the collaborative projects that emerged between the usability team and campus disabilities services as a result of this testing. The article concludes with heuristics and generative questions that may assist readers in developing similar projects tailored to their own contexts.     

Process of design and usability evaluation of a telepsychology web and virtual reality system for the elderly: Butler

Butler is a multi-application system based on Internet technology and virtual reality, designed to meet the needs of elderly users. Its objectives are to assist elderly populations in accessing technology and new forms of communication, replicating, as far as possible the way of lineal interaction that this population group had traditionally to interact with information in order to facilitate the first contacts with technology. Butler is equipped with e-mail, videoconferencing, blogging, access to the Internet through a well-known web search engine, image and sound gallery, new contacts searcher, and two virtual reality environments. In this paper we describe Butler's most relevant features taking into account the usability design process and the usability evaluation of the system, which, after pilot testing, have proven to be highly acceptable and satisfactory to users. We also present the results of evaluations pertaining to the iconography used in the program, the navigation and users' help design process, and the first functional prototype, all of which were designed and redesigned with the aid of web heuristics and guidelines for the web design for elderly users. Results suggest important features to be taken into account when designing and developing this sort of application for the elderly.     

一种基于虚拟截止时间制导的改进的Min-Min元任务调度算法

在网格环境下,资源状况和用户行为相当复杂,是一个异构计算环境,元任务（meta—task）调度比传统并行调度更为复杂。如何映射一组任务到一组机器上被证明是NP问题,其目的一般是最小化任务完成时间（makespan）。为解决这一问题,已经提出一些启发式任务调度算法,例如具有代表性的MinMin元任务调度算法。本文在Min-Min元任务调度算法的基础上,通过虚拟截止时间制导的方法来改进Min-Min算法。实验结果表明,本文提出的算法具有更短的任务完成时间。