Developing usability heuristics with PROMETHEUS: A case study in virtual learning environments

Heuristic evaluation is one of the most widely-used methods for evaluating the usability of a software product. Proposed in 1990 by Nielsen and Molich, it consists in having a small group of evaluators performing a systematic revision of a system under a set of guiding principles known as usability heuristics. Although Nielsen’s 10 usability heuristics are used as the de facto standard in the process of heuristic evaluation, recent research has provided evidence not only for the need of custom domain specific heuristics, but also for the development of methodological processes to create such sets of heuristics. In this work we apply the PROMETHEUS methodology, recently proposed by the authors, to develop the VLEs heuristics: a novel set of usability heuristics for the domain of virtual learning environments. In addition to the development of these heuristics, our research serves as further empirical validation of PROMETHEUS. To validate our results we performed an heuristic evaluation using both VLEs and Nielsen’s heuristics. Our design explicitly controls the effect of evaluator variability by using a large number of evaluators. Indeed, for both sets of heuristics the evaluation was performed independently by 7 groups of 5 evaluators each. That is, there were 70 evaluators in total, 35 using VLEs and 35 using Nielsen’s heuristics. In addition, we perform rigorous statistical analyses to establish the validity of the novel VLEs heuristics. The results show that VLEs perform better than Nielsen’s heuristics, finding more problems, which are also more relevant to the domain, as well as satisfying other quantitative and qualitative criteria. Finally, in contrast to evaluators using Nielsen’s heuristics, evaluators using VLEs heuristics reported greater satisfaction regarding utility, clarity, ease of use, and need of additional elements.     

A Textual Feedback Tool for Empowering Participants in Usability and UX Evaluations

The usability movement has historically always sought to empower end-users of computers so that they understand what is happening and can control the outcome. In this article, we develop and evaluate a “Textual Feedback” tool for usability and user experience (UX) evaluation that can be used to empower well-educated but low-status users in UX evaluations in countries and contexts with high power distances. The proposed tool contributes to the Human–Computer Interaction (HCI) community’s pool of localized UX evaluation tools. We evaluate the tool with 40 users from two socio-economic groups in real-life UX usability evaluations settings in Malaysia. The results indicate that the Textual Feedback tool may help participants to give their thoughts in UX evaluation in high power distance contexts. In particular, the Textual Feedback tool helps high status females and low status males express more UX problems than they can with traditional concurrent think aloud (CTA) alone. We found that classic concurrent think aloud UX evaluation works fine in high power contexts, but only with the addition of Textual Feedback to mitigate the effects of socio-economic status in certain user groups. We suggest that future research on UX evaluation look more into how to empower certain user groups, such as low status female users, in UX evaluations done in high power distance contexts.     

Interplay between User Experience (UX) evaluation and system development

User Experience (UX) is a maturing research area pertaining to as well as extending beyond the traditional usability. Issues in the realm of usability may be amplified in UX because of its larger scope. Four key non-orthogonal issues are definition, modeling, method selection, and interplay between evaluation and development. Leveraging the legacy of a series of related research activities, this Special Issue (SI) aims to develop a deeper understanding of how evaluation feedback shapes software development, especially when experiential qualities such as fun, trust, esthetic values are concerned. Three articles addressing this specific topic from different perspectives and with different approaches are included in this SI.     

How User Experience Changes over Time: A Case Study of Social Network Services

While interacting with a product or a service, people have diverse experiences, which constitute user experience (UX). In this study, the hierarchical dimensions of UX were considered, including overall UX, its elements (usability, affect, and user value) and their subelements. To determine how UX changes over time, the temporal changes of the influence of each element and subelement were analyzed. Twenty subjects participated in a longitudinal experiment by using a social network service (SNS). The most important element of UX changed from usability to user value after one week of use. These results suggest that adoption of an SNS occurs in two phases: a phase of affected by usability and a phase affected by user value. Identification of these phases and the systematic approach used in the study are expected to contribute to design and evaluation of new SNSs and other products and services.     

Do patterns help novice evaluators? A comparative study

Evaluating e-learning systems is a complex activity which requires considerations of several criteria addressing quality in use as well as educational quality. Heuristic evaluation is a widespread method for usability evaluation, yet its output is often prone to subjective variability, primarily due to the generality of many heuristics. This paper presents the pattern-based (PB) inspection, which aims at reducing this drawback by exploiting a set of evaluation patterns to systematically drive inspectors in their evaluation activities. The application of PB inspection to the evaluation of e-learning systems is reported in this paper together with a study that compares this method to heuristic evaluation and user testing. The study involved 73 novice evaluators and 25 end users, who evaluated an e-learning application using one of the three techniques. The comparison metric was defined along six major dimensions, covering concepts of classical test theory and pragmatic aspects of usability evaluation. The study showed that evaluation patterns, capitalizing on the reuse of expert evaluators know-how, provide a systematic framework which reduces reliance on individual skills, increases inter-rater reliability and output standardization, permits the discovery of a larger set of different problems and decreases evaluation cost. Results also indicated that evaluation in general is strongly dependent on the methodological apparatus as well as on judgement bias and individual preferences of evaluators, providing support to the conceptualisation of interactive quality as a subjective judgement, recently brought forward by the UX research agenda.     

Better discount evaluation: illustrating how critical parameters support heuristic creation

This paper describes a heuristic creation process based on the notion of critical parameters, and a comparison experiment that demonstrates the utility of heuristics created for a specific system class. We focus on two examples of using the newly created heuristics to illustrate the utility of the usability evaluation method, as well as to provide support for the creation process, and we report on successes and frustrations of two classes of users, novice evaluators and domain experts, who identified usability problems with the new heuristics. We argue that establishing critical parameters for other domains will support efforts in creating tailored evaluation tools.     

Developing SMASH: A set of SMArtphone's uSability Heuristics

The smartphone market is nowadays highly competitive. When buying a new device, users focus on visual esthetics, ergonomics, performance, and user experience, among others. Assessing usability issues allows improving these aspects. One popular method for detecting usability problems is heuristic evaluation, in which evaluators employ a set of usability heuristics as guide. Using proper heuristics is highly relevant. In this paper we present SMASH, a set of 12 usability heuristics for smartphones and mobile applications, developed iteratively. SMASH (previously named TMD: Usability heuristics for Touchscreen-based Mobile Devices) was experimentally validated. The results support its utility and effectiveness.     

A development of learning widget on m-learning and e-learning environments

This article describes the development of learning widget on m-learning and e-learning environments. A widget is a small, simple and useful application supporting user-oriented contents. The user may select and install widgets that are convenient as well as an auto-updating application including weather or calendar. These widgets are especially more useful, because they are able to be installed on a mobile device, a website or a desktop computer. If we take advantage of widgets for education, we may use this learning tool for delivering and pulling learning contents, essences of lessons or word learning. To that end, we developed an effective learning widget and then verified its usability, usefulness and effectiveness for m-learning and e-learning. That is, we evaluated the learning widget with a heuristic evaluation method. We identified 72 interface problems by using a set of 10 usability criteria or heuristics. In addition, we considered how to design the learning widget with consideration given to devices on m-learning and e-learning. Moreover, we experimented by conducting a pilot test with 34 students, a field test with 60 teachers and technology acceptance model (TAM) analysis with 15 teachers. We verified the effectiveness and usefulness of learning with a questionnaire, a quiz and TAM, where the subjects, after using the learning widget in real learning activities, rated the widget's efficacy. The result shows that the learning widget is useful for m-learning and e-learning environments.     

A framework for evaluating the usability of mobile phones based on multi-level, hierarchical model of usability factors

As a mobile phone has various advanced functionalities or features, usability issues are increasingly challenging. Due to the particular characteristics of a mobile phone, typical usability evaluation methods and heuristics, most of which are relevant to a software system, might not effectively be applied to a mobile phone. Another point to consider is that usability evaluation activities should help designers find usability problems easily and produce better design solutions. To support usability practitioners of the mobile phone industry, we propose a framework for evaluating the usability of a mobile phone, based on a multi-level, hierarchical model of usability factors, in an analytic way. The model was developed on the basis of a set of collected usability problems and our previous study on a conceptual framework for identifying usability impact factors. It has multi-abstraction levels, each of which considers the usability of a mobile phone from a particular perspective. As there are goal-means relationships between adjacent levels, a range of usability issues can be interpreted in a holistic as well as diagnostic way. Another advantage is that it supports two different types of evaluation approaches: task-based and interface-based. To support both evaluation approaches, we developed four sets of checklists, each of which is concerned, respectively, with task-based evaluation and three different interface types: Logical User Interface (LUI), Physical User Interface (PUI) and Graphical User Interface (GUI). The proposed framework specifies an approach to quantifying usability so that several usability aspects are collectively measured to give a single score with the use of the checklists. A small case study was conducted in order to examine the applicability of the framework and to identify the aspects of the framework to be improved. It showed that it could be a useful tool for evaluating the usability of a mobile phone. Based on the case study, we improved the framework in order that usability practitioners can use it more easily and consistently.     

A Systematic and Generalizable Approach to the Heuristic Evaluation of User Interfaces

Heuristic evaluation is one of the most actively used techniques for analyzing usability, as it is quick and inexpensive. This technique is based on following a given set of heuristics, which are typically defined as broad rules of thumb. In this paper, we propose a systematic and generalizable approach to this type of evaluation based on using comprehensive taxonomies as a source for the heuristics. This approach contrasts with other typical approaches, such as following (or adapting) Jakob Nielsen’s heuristics or creating ad hoc heuristics (formally or informally). The usefulness of our approach is investigated in two ways. Firstly, we carry out an actual heuristic evaluation of a mobile app in this manner, which we describe in detail. Secondly, we compare our approach and Nielsen’s. Additionally, we identify some limitations in Nielsen’s heuristics and some inconsistencies between them and established usability models, including Nielsen’s own.     

A unified methodology for the evaluation of accessibility and usability of mobile applications

This article reports a unified methodology developed to evaluate the accessibility and usability of mobile computing applications, which is intended to guarantee universal access as far as possible. As a basis for the methodology, this paper presents an analysis of the accessibility guidelines, conducted to take into account the specificity of mobile systems, as well as a set of usability heuristics, specifically devised for mobile computing. Finally, it presents the results of the application of the proposed methodology to applications that have been semi-automatically developed by the MAIS Designer, a new design tool that provides applications suited to different mobile devices.     

Evaluating user experience of adaptive digital educational games with Activity Theory

Adaptive digital educational games (DEGs) providing players with relevant interventions can enhance gameplay experience. This advance in game design, however, renders the user experience (UX) evaluation of DEGs even more challenging. To tackle this challenge, we developed a four-dimension evaluation framework (i.e., gaming experience, learning experience, adaptivity, and usability) and applied it to an empirical study with a DEG on teaching geography. Mixed-method approaches were adopted to collect data with 16 boys aged 10–11. Specifically, a so-called Dyadic User Experience Tests (DUxT) was employed; participants were paired up to assume different roles during gameplay. Learning efficacy was evaluated with a pre-post intervention measurement using a domain-specific questionnaire. Learning experience, gaming experiences and usability were evaluated with intensive in situ observations and interviews guided by a multidimensional scheme; content analysis of these transcribed audio data was supplemented by video analysis. Effectiveness of adaptivity algorithms was planned to be evaluated with automatic logfiles, which, unfortunately, could not be realised due to some technical problem. Nonetheless, the user-based data could offer some insights into this issue. Furthermore, we attempted to bridge the existing gap in UX research – the lack of theoretical frameworks in understanding user experience – by adopting Engeström's (1987) extended framework of Activity Theory (AT) that provides contextual information essential for understanding contradictions and breakdowns observed in the interactions between the game players. The dyadic gameplay setting allows us to explore the issue of group UX. Implications for further applications of the AT framework in the UX research, especially the interplay between evaluation and redesign (i.e., downstream utility of UX evaluation methods), are discussed.     

Using sketches and storyboards to assess impact of age difference in user experience

We compared two versions of a touch-screen digital thermostat using a framework encompassing several user experience (UX) characteristics, and here describe how the implementation of certain design factors (specialists, praises, tooltips and increased interactivity) was done on mixed-fidelity prototypes of the user interface. We illustrate how the experimental comparison, involving 20 university students and 20 older adults, revealed important differences in UX, including perceived ease of use, behavioral intentions, enjoyment, quality, satisfaction, trust and usability, measured mainly through established questionnaires.Analysis revealed that using that kind of artifacts is a very cost effective way to elicit interesting and useful results; many UX variables are significantly affected by design factors and by age differences, as expected; effects of design factors go well beyond usability and therefore could not be caught by running an investigation focused only on usability.Age difference matters: older adults do not respond to addition of specialists, praises and tooltips as younger users do. We argue that potential benefits of these design choices are outweigh by the increase in complexity of the user interface.From a methodological viewpoint we suggest using a particular array of UX characteristics and metrics when testing mixed-fidelity prototypes. Not all the metrics that we adopted were equally useful, and in particular perceived usability, subjective mental effort, and emotions did not help us highlighting differences.     

The user action framework: a reliable foundation for usability engineering support tools

Although various methods exist for performing usability evaluation, they lack a systematic framework for guiding and structuring the assessment and reporting activities. Consequently, analysis and reporting of usability data are ad hoc and do not live up to their potential in cost effectiveness, and usability engineering support tools are not well integrated. We developed the User Action Framework, a structured knowledge base of usability concepts and issues, as a framework on which to build a broad suite of usability engineering support tools. The User Action Framework helps to guide the development of each tool and to integrate the set of tools in the practitioner's working environment. An important characteristic of the User Action Framework is its own reliability in term of consistent use by practitioners. Consistent understanding and reporting of the underlying causes of usability problems are requirements for cost-effective analysis and redesign. Thus, high reliability in terms of agreement by users on what the User Action Framework means and how it is used is essential for its role as a common foundation for the tools. Here we describe how we achieved high reliability in the User Action Framework, and we support the claim with strongly positive results of a summative reliability study conducted to measure agreement among 10 usability experts in classifying 15 different usability problems. Reliability data from the User Action Framework are also compared to data collected from nine of the same usability experts using a classic heuristic evaluation technique.     

Empirical validation of an automatic usability evaluation method

Today, the success of a software application strongly depends on the usability of its interface, so the evaluation of interfaces has become a crucial aspect of software engineering. It is recognized that automatic tools for graphical user interface evaluation may greatly reduce the costs of traditional activities performed during expert evaluation or user testing in order to estimate the success probability of an application. However, automatic methods need to be empirically validated in order to prove their effectiveness with respect to the attributes they are supposed to evaluate.In this work, we empirically validate a usability evaluation method conceived to assess consistency aspects of a GUI with no need to analyze the back-end. We demonstrate the validity of the approach by means of a comparative experimental study, where four web sites and a stand-alone interactive application are analyzed and the results compared to those of a human-based usability evaluation. The analysis of the results and the statistical correlation between the tool׳s rating and humans׳ average ratings show that the proposed methodology can indeed be a useful complement to standard techniques of usability evaluation.     

Design-inclusive UX research: design as a part of doing user experience research

Since the third wave in human–computer interaction (HCI), research on user experience (UX) has gained momentum within the HCI community. The focus has shifted from systematic usability requirements and measures towards guidance on designing for experiences. This is a big change, since design has traditionally not played a large role in HCI research. Yet, the literature addressing this shift in focus is very limited. We believe that the field of UX research can learn from a field where design and experiential aspects have always been important: design research. In this article, we discuss why design is needed in UX research and how research that includes design as a part of research can support and advance UX design practice. We do this by investigating types of design-inclusive UX research and by learning from real-life cases of UX-related design research. We report the results of an interview study with 41 researchers in three academic research units where design research meets UX research. Based on our interview findings, and building on existing literature, we describe the different roles design can play in research projects. We also report how design research results can inform designing for experience methodologically or by providing new knowledge on UX. The results are presented in a structured palette that can help UX researchers reflect and focus more on design in their research projects, thereby tackling experience design challenges in their own research.     

An interdisciplinary heuristic evaluation method for universal building design

This study highlights how heuristic evaluation as a usability evaluation method can feed into current building design practice to conform to universal design principles. It provides a definition of universal usability that is applicable to an architectural design context. It takes the seven universal design principles as a set of heuristics and applies an iterative sequence of heuristic evaluation in a shopping mall, aiming to achieve a cost-effective evaluation process. The evaluation was composed of three consecutive sessions. First, five evaluators from different professions were interviewed regarding the construction drawings in terms of universal design principles. Then, each evaluator was asked to perform the predefined task scenarios. In subsequent interviews, the evaluators were asked to re-analyze the construction drawings. The results showed that heuristic evaluation could successfully integrate universal usability into current building design practice in two ways: (i) it promoted an iterative evaluation process combined with multi-sessions rather than relying on one evaluator and on one evaluation session to find the maximum number of usability problems, and (ii) it highlighted the necessity of an interdisciplinary ad hoc committee regarding the heuristic abilities of each profession. A multi-session and interdisciplinary heuristic evaluation method can save both the project budget and the required time, while ensuring a reduced error rate for the universal usage of the built environments.     

Process of design and usability evaluation of a telepsychology web and virtual reality system for the elderly: Butler

Butler is a multi-application system based on Internet technology and virtual reality, designed to meet the needs of elderly users. Its objectives are to assist elderly populations in accessing technology and new forms of communication, replicating, as far as possible the way of lineal interaction that this population group had traditionally to interact with information in order to facilitate the first contacts with technology. Butler is equipped with e-mail, videoconferencing, blogging, access to the Internet through a well-known web search engine, image and sound gallery, new contacts searcher, and two virtual reality environments. In this paper we describe Butler's most relevant features taking into account the usability design process and the usability evaluation of the system, which, after pilot testing, have proven to be highly acceptable and satisfactory to users. We also present the results of evaluations pertaining to the iconography used in the program, the navigation and users' help design process, and the first functional prototype, all of which were designed and redesigned with the aid of web heuristics and guidelines for the web design for elderly users. Results suggest important features to be taken into account when designing and developing this sort of application for the elderly.     

A Heuristic Evaluation for Deaf Web User Experience (HE4DWUX)

Websites that are usable and accessible can have a positive impact on the overall user experience. Usability Inspection Methods (UIMs) can be applied to evaluate and measure the usability. The current research in the fields of Web Accessibility and Human –Computer Interaction (HCI) is in need of additional UIMs that can be applied to also measure the accessibility, in addition to the usability alone. In this article, a novel UIM in the form of a heuristic evaluation is presented. The heuristic evaluation aims to support HCI experts and Web developers in designing and evaluating websites that provide positive user experiences to users who are deaf. This article discusses the development of the Heuristic Evaluation for Deaf Web User Experience (HE4DWUX). Following an iteration cycle, version 2 of the HE4DWUX is presented in Appendix A. An existing three-phase process to develop heuristics for specific application domains was applied to construct the HE4DWUX. The outcome of this research is 12 heuristics, with each containing its own set of checklist items to operationalize its applicability in measuring the Web user experience for users who are deaf. The heuristics and their checklist items can identify important aspects of design that will impact the Web user experience for this particular user group.