Developing usability heuristics with PROMETHEUS: A case study in virtual learning environments

Heuristic evaluation is one of the most widely-used methods for evaluating the usability of a software product. Proposed in 1990 by Nielsen and Molich, it consists in having a small group of evaluators performing a systematic revision of a system under a set of guiding principles known as usability heuristics. Although Nielsen’s 10 usability heuristics are used as the de facto standard in the process of heuristic evaluation, recent research has provided evidence not only for the need of custom domain specific heuristics, but also for the development of methodological processes to create such sets of heuristics. In this work we apply the PROMETHEUS methodology, recently proposed by the authors, to develop the VLEs heuristics: a novel set of usability heuristics for the domain of virtual learning environments. In addition to the development of these heuristics, our research serves as further empirical validation of PROMETHEUS. To validate our results we performed an heuristic evaluation using both VLEs and Nielsen’s heuristics. Our design explicitly controls the effect of evaluator variability by using a large number of evaluators. Indeed, for both sets of heuristics the evaluation was performed independently by 7 groups of 5 evaluators each. That is, there were 70 evaluators in total, 35 using VLEs and 35 using Nielsen’s heuristics. In addition, we perform rigorous statistical analyses to establish the validity of the novel VLEs heuristics. The results show that VLEs perform better than Nielsen’s heuristics, finding more problems, which are also more relevant to the domain, as well as satisfying other quantitative and qualitative criteria. Finally, in contrast to evaluators using Nielsen’s heuristics, evaluators using VLEs heuristics reported greater satisfaction regarding utility, clarity, ease of use, and need of additional elements.     

Heuristic evaluation: Comparing ways of finding and reporting usability problems

Research on heuristic evaluation in recent years has focused on improving its effectiveness and efficiency with respect to user testing. The aim of this paper is to refine a research agenda for comparing and contrasting evaluation methods. To reach this goal, a framework is presented to evaluate the effectiveness of different types of support for structured usability problem reporting. This paper reports on an empirical study of this framework that compares two sets of heuristics, Nielsen’s heuristics and the cognitive principles of Gerhardt-Powals, and two media of reporting a usability problem, i.e. either using a web tool or paper. The study found that there were no significant differences between any of the four groups in effectiveness, efficiency and inter-evaluator reliability. A more significant contribution of this research is that the framework used for the experiments proved successful and should be reusable by other researchers because of its thorough structure.     

Expert vs novice collaborative heuristic evaluation (CHE) of a smartphone app for cultural heritage sites

This study was conducted to compare CHE between Human-Computer Interaction (HCI) experts and novices in evaluating the Smartphone app for the cultural heritage site. It uses the Smartphone Mobile Application heuRisTics (SMART), focusing on smartphone applications and traditional Nielsen heuristics, focusing on a wider range of interactive systems. Six experts and six novices used the severity rating scale to categorise the severity of the usability issues. These issues were mapped to both heuristics. The study found that experts and novice evaluators identified 19 and 14 usability issues, respectively, with ten as the same usability issues. However, these same usability issues have been rated differently. Although the t-test indicates no significant differences between experts and novices in their ratings for usability issues, these results nevertheless indicate the need for both evaluators in CHE to provide a more comprehensive perspective on the severity of the usability issues. Furthermore, the mapping of the usability issues for Nielsen and SMART heuristics concluded that more issues with the smartphone app could be addressed through smartphone-specific heuristics than general heuristics, indicating a better tool for heuristic evaluation of the smartphone app. This study also provides new insight into the required number of evaluators needed for CHE.

     

Applying a methodology to develop user eXperience heuristics

A heuristic evaluation method allows the evaluation of the usability of application domains. To evaluate applications that have specific domain features, researchers can use sets of specific usability heuristics in addition to the well-known (usually Nielsen's) heuristics. Heuristics can also focus on the User eXperience (UX) aspects other than the usability. In a previous work, we proposed a formal methodology for establishing usability/UX heuristics. The methodology has 8 stages including activities to formulate, specify, validate and refine a new set of heuristics for a specific application domain. The methodology was validated through expert opinion and several case studies. Although when specifying the methodology, we explained each of its stages in detail, some activities can be difficult to perform without a guide that helps the researcher determine how the stages should be carried out. This article presents a detailed explanation regarding how to apply each stage of the methodology to create a new set of heuristics for a specific domain. Additionally, this paper explains how to iterate the methodology's stages and when to stop the process of developing new heuristics.     

An Optimized Image Segmentation Approach Based on Boltzmann Machine

Image segmentation with complex background is a tedious task. In our study, a convex spline is constructed based on Good Features to Track (GF2T) method’s region-based salient feature (i.e., corner) set. For an optimized edge-based segmentation, an ellipse shape prior based on this convex spline is useful in edge regularization procedure with region-based features. This kind of optimization is achieved by Boltzmann machine (BM) to automatically form an elliptical foreground mask of the GrabCut method. We demonstrated our approach’s usability through traveling salesman problem (TSP), thus, we consider that the TSP’s valid tour’s path solved by BM can be taken as an optimized convex spline for edge-based segmentation. In our experiments, proposed BM-based approach has the performance improvement of segmentation to stand-alone GF2T as 29.79% improvement based on bounding boxes evaluation and as 38.67% improvement based on the overlapping pixel regions for a quantitative evaluation via objective metrics.     

A framework for evaluating the usability of mobile phones based on multi-level, hierarchical model of usability factors

As a mobile phone has various advanced functionalities or features, usability issues are increasingly challenging. Due to the particular characteristics of a mobile phone, typical usability evaluation methods and heuristics, most of which are relevant to a software system, might not effectively be applied to a mobile phone. Another point to consider is that usability evaluation activities should help designers find usability problems easily and produce better design solutions. To support usability practitioners of the mobile phone industry, we propose a framework for evaluating the usability of a mobile phone, based on a multi-level, hierarchical model of usability factors, in an analytic way. The model was developed on the basis of a set of collected usability problems and our previous study on a conceptual framework for identifying usability impact factors. It has multi-abstraction levels, each of which considers the usability of a mobile phone from a particular perspective. As there are goal-means relationships between adjacent levels, a range of usability issues can be interpreted in a holistic as well as diagnostic way. Another advantage is that it supports two different types of evaluation approaches: task-based and interface-based. To support both evaluation approaches, we developed four sets of checklists, each of which is concerned, respectively, with task-based evaluation and three different interface types: Logical User Interface (LUI), Physical User Interface (PUI) and Graphical User Interface (GUI). The proposed framework specifies an approach to quantifying usability so that several usability aspects are collectively measured to give a single score with the use of the checklists. A small case study was conducted in order to examine the applicability of the framework and to identify the aspects of the framework to be improved. It showed that it could be a useful tool for evaluating the usability of a mobile phone. Based on the case study, we improved the framework in order that usability practitioners can use it more easily and consistently.     

Correlation between safety assessments in the driver-car interaction design process

With the functional revolution in modern cars, evaluation methods to be used in all phases of driver–car interaction design have gained importance. It is crucial for car manufacturers to discover and solve safety issues early in the interaction design process. A current problem is thus to find a correlation between the formative methods that are used during development and the summative methods that are used when the product has reached the customer. This paper investigates the correlation between efficiency metrics from summative and formative evaluations, where the results of two studies on sound and navigation system tasks are compared. The first, an analysis of the J.D. Power and Associates APEAL survey, consists of answers given by about two thousand customers. The second, an expert evaluation study, was done by six evaluators who assessed the layouts by task completion time, TLX and Nielsen heuristics. The results show a high degree of correlation between the studies in terms of task efficiency, i.e. between customer ratings and task completion time, and customer ratings and TLX. However, no correlation was observed between Nielsen heuristics and customer ratings, task completion time or TLX. The results of the studies introduce a possibility to develop a usability evaluation framework that includes both formative and summative approaches, as the results show a high degree of consistency between the different methodologies. Hence, combining a quantitative approach with the expert evaluation method, such as task completion time, should be more useful for driver–car interaction design.     

Developing SMASH: A set of SMArtphone's uSability Heuristics

The smartphone market is nowadays highly competitive. When buying a new device, users focus on visual esthetics, ergonomics, performance, and user experience, among others. Assessing usability issues allows improving these aspects. One popular method for detecting usability problems is heuristic evaluation, in which evaluators employ a set of usability heuristics as guide. Using proper heuristics is highly relevant. In this paper we present SMASH, a set of 12 usability heuristics for smartphones and mobile applications, developed iteratively. SMASH (previously named TMD: Usability heuristics for Touchscreen-based Mobile Devices) was experimentally validated. The results support its utility and effectiveness.     

Better discount evaluation: illustrating how critical parameters support heuristic creation

This paper describes a heuristic creation process based on the notion of critical parameters, and a comparison experiment that demonstrates the utility of heuristics created for a specific system class. We focus on two examples of using the newly created heuristics to illustrate the utility of the usability evaluation method, as well as to provide support for the creation process, and we report on successes and frustrations of two classes of users, novice evaluators and domain experts, who identified usability problems with the new heuristics. We argue that establishing critical parameters for other domains will support efforts in creating tailored evaluation tools.     

A Taxonomy-Based Usability Study of an Intelligent Speed Adaptation Device

Usability studies are often based on ad hoc definitions of usability. These studies can be difficult to generalize, they might have a steep learning curve, and there is always the danger of being inconsistent with the concept of usability as defined in standards and the literature. This alternative approach involves comprehensive, general-purpose, and hierarchically structured taxonomies that follow closely the main usability literature. These taxonomies are then instantiated for a specific product. To illustrate this approach, a usability study for a prototype of an Intelligent Speed Adaptation device is described. The usability study consists of usability requirements analysis, heuristic evaluation, and subjective analysis, which helped identify problems of clarity, operability, robustness, safety, and aesthetics. As a context-specific usability taxonomy for this particular field of application happened to exist, the way that real-world usability results can be mapped to that taxonomy compared to the taxonomy in this article is examined, with the argument that this study’s taxonomy is more complete and generalizable.     

Evaluating visualizations: do expert reviews work?

Visualization research generates beautiful images and impressive interactive systems. Emphasis on evaluating visualizations is growing. Researchers have successfully used alternative evaluation techniques in human-computer interaction (HCI), including focus groups, field studies, and expert reviews. These methods tend to produce qualitative results and require fewer participants than controlled experiments. In this article, we focus on expert reviews that we used for the applications. We commonly use expert reviews to assess interface usability. Expert reviews can generate valuable feedback on visualization tools. We recommend i) including experts with experience in data display as well as usability, and ii) developing heuristics based on visualization guidelines as well as usability guidelines. Expert reviews should not be used exclusively, since experts might not hilly predict end-user actions. Furthermore, we encourage more experimentation with this technique, particularly to develop a good set of visualization heuristics and to compare it with other methods.     

Usability guideline for banking software design

As the diversity of services in the financial market increases, it is critical to design usable banking software in order to overcome the complex structure of the system. The current study presents a usability guideline based on heuristics and their corresponding criteria that could be used during the early stages of banking software design process. In the design of a usability guideline, the heuristics and their criteria are categorized in terms of their effectiveness in solving usability problems grouped and ranging from usability catastrophe to cosmetic problems. The current study comprises of three main steps: First, actual usability problems from three banking software development projects are categorized according to their severity level. Secondly, usability criteria are rated for how well they explain the usability problems encountered. Finally, usability heuristics are categorized according to the severity level of usability problems through two analytical models; corresponding and cluster analyses. As the result, designers and project managers may give more importance to the heuristics related with the following usability problem categories: Usability catastrophe and then major usability problems. Furthermore, the proposed guideline can be used to understand which usability criteria would be helpful in explaining usability problems as well as preventing banking system catastrophes, by highlighting the critical parts in system design of banking software.     

A Usable Security Heuristic Evaluation for the Online Health Social Networking Paradigm

The security, privacy, and usability of health social networking websites remain key concerns for their successful adoption by patients. As far as it is known, no studies have developed a usable security heuristic evaluation to test these websites for the usability of their security and privacy features. This article describes the development of such a usable security heuristic evaluation in the context of the online health social networking paradigm. The tailored method approach was applied to transform themes identified from relevant literature and convert them into high-level heuristics for usable security. Checklist items were created for each high-level heuristic. The result is 13 high-level heuristics with individualized checklist items that help examine usable security. The results can identify which security and privacy features are lacking from a usability perspective and would, therefore, require design improvement. The improvements would enhance the users' experiences, which in turn can assist in the adoption of health social networking websites.     

A CSCW Requirements Engineering CASE Tool: Development and usability evaluation

ContextCSRML Tool 2012 is a Requirements Engineering CASE Tool for the Goal-Oriented Collaborative Systems Requirements Modeling Language (CSRML).ObjectiveThis work aims at evaluating the usability of CSRML Tool 2012, thus identifying any possible usability flaw to be solved in the next releases of the application, as well as giving a general overview on how to develop a DSL tool similar to the one evaluated in this work by means of Visual Studio Modelling SDK.MethodIn this evaluation, which was reported by following the ISO/IEC 25062:2006 Common Industry Format for usability tests, 28 fourth-course Computer Science students took part. They were asked to carry out a series of modifications to an incomplete CSRML requirements specification. Usability was assessed by measuring the task’s completion rate, the elapsed time, number of accesses to the help system of the tool and the instructor’s verbal assistance. The participants’ arousal and pleasantness were assessed by analyzing both facial expressions and a USE questionnaire.ResultsIn spite of obtaining high usability levels, the test outcome revealed some usability flaws that should be addressed.ConclusionThe important lessons learnt from this evaluation are also applicable to the success of other usability tests as well as to the development of new CASE tools.     

Integration of Usability Evaluation Studies via a Novel Meta-Analytic Approach: What are Significant Attributes for Effective Evaluation?

The overall discovery rates, which are the ratios of sum of unique usability problems detected by all experiment participants against the number of usability problems existed in the evaluated systems, were investigated to find significant factors of usability evaluation through a meta-analytic approach with the n-corrected effect sizes newly defined in this study. Since many studies of usability evaluation have been conducted under specific contexts showing some mixed findings, usability practitioners need holistic and more generalized conclusions. Due to the limited applicability of the traditional meta-analysis to usability evaluation studies, a new meta-analytic approach was established and applied to 38 experiments that reported overall discovery rates of usability problems as a criterion measure. Through the meta-analytic approach with the n-corrected effect sizes, we successfully combined 38 experiments and found evaluator's expertise, report type, and interaction between usability evaluation method and time constraint as significant factors. We suggest that in order to increase overall discovery rates of usability problems, (a) free-style written reports are better than structured written reports; (b) when heuristic evaluation or cognitive walkthrough is used, the usability evaluation experiments should be conducted without time constraint, but when think-aloud is used, time constraint is not an important experimental condition; (c) usability practitioners do not need to be concerned about unit of evaluation, fidelity of evaluated systems, and task type; and (d) HCI experts are better than novice users or evaluators. Our conclusions can guide usability practitioners when determining evaluation contexts, and the meta-analytic approach of this study provides an alternative way to combine the empirical results of usability evaluation besides the traditional meta-analysis.     

A new hybrid GA solution to combinatorial optimization problems— an application to the multiprocessor scheduling problem

The multiprocessor scheduling problem is one of the classic examples of NP-hard combinatorial optimization problems. Several polynomial time optimization algorithms have been proposed for approximating the multiprocessor scheduling problem. In this paper, we suggest a geneticizedknowledge genetic algorithm (gkGA) as an efficient heuristic approach for solving the multiprocessor scheduling and other combinatorial optimization problems. The basic idea behind the gkGA approach is that knowledge of the heuristics to be used in the GA is also geneticized alongiside the genetic chromosomes. We start by providing four conversion schemes based on heuristics for converting chromosomes into priority lists. Through experimental evaluation, we observe that the performance of our GA based on each of these schemes is instance-dependent. However, if we simultaneously incorporate these schemes into our GA through the gkGA approach, simulation results show that the approach is not problem-dependent, and that the approach outperforms that of the previous GA. We also show the effectiveness of the gkGA approach compared with other conventional schemes through experimental evaluation. This work was presented, in part, at the Second International Symposium on Artifiical Life and Robotics, Oita, Japan, February 18–20, 1997     

Extending the Usability of Heuristics for Design and Evaluation: Lead,Follow, and Get Out of the Way

User-centered design practitioners have often relied on discount usability engineering methods using heuristics. Top 10 lists of design and evaluation heuristics have proliferated during the 1990s, leading to a plethora of heterogeneous heuristic guidelines for a multitude of user interfaces. A simple, unified set of heuristics that might be applicable across different technologies, understood between disciplines, and equivalent to metrics used to measure ease of use can be beneficial.

Lead, follow, and get out of the way (LF&G) theorizes that the optimal human-computer interaction (HCI) experience is analogous to a facilitative learning relation-Like a good teacher, mentor, or coach, the usable user interface leads the user to successful completion of tasks and goals; follows the user's progress and provides appropriate feedback and information when needed; and gets out of the way of the user to allow efficient and effective completion of tasks as the user attains mastery of the system, its concepts, and operations. A set of 18 heuristics grouped under the 3 general principles of the title are provided for use across the tasks of design guidance, development refinement, and end-user evaluation of computer systems.

This article provides background into the development of LF&G, case studies on its use in real-world product design and development, and directions for further research needed to develop this approach into an empirically based method for defining, describing, designing, and predicting the ease of use of interactive human-computer systems.     

A holistic approach to collaborative ontology development based on change management

This paper describes our methodological and technological approach for collaborative ontology development in inter-organizational settings. It is based on the formalization of the collaborative ontology development process by means of an explicit editorial workflow, which coordinates proposals for changes among ontology editors in a flexible manner. This approach is supported by new models, methods and strategies for ontology change management in distributed environments: we propose a new form of ontology change representation, organized in layers so as to provide as much independence as possible from the underlying ontology languages, together with methods and strategies for their manipulation, version management, capture, storage and maintenance, some of which are based on existing proposals in the state of the art. Moreover, we propose a set of change propagation strategies that allow keeping distributed copies of the same ontology synchronized. Finally, we illustrate and evaluate our approach with a test case in the fishery domain from the United Nations Food and Agriculture Organisation (FAO). The preliminary results obtained from our evaluation suggest positive indication on the practical value and usability of the work here presented.     

An automated approach to dependability evaluation of virtual networks

Network virtualization has been pointed as a promising approach to solve Internet’s current ossification. A major challenge is the mapping of virtual networks (VNs) onto the substrate network due to its NP-hard nature, and, thus, several heuristics have been proposed aiming to achieve efficient allocations. However, the existing approaches only focus on performance, and dependability issues are usually neglected. Dependability involves metrics such as reliability and availability, which directly impact the quality of service, and a limitation is the absence of mechanisms for estimating dependability metrics in virtual networks. This paper proposes an automated approach for estimating dependability metrics in virtual network environments and a mapping algorithm based on GRASP metaheuristic. The approach is based on stochastic Petri nets (SPN) and reliability block diagrams (RBD), and a tool is also presented for automating model generation and evaluation. Experimental results demonstrate the feasibility of the proposed technique, as well as the trade-off between VN’s availability and cost.     

Request-driven swarming scheme for P2P data streaming

Data streaming by swarming over peer-to-peer (P2P) overlay networks has attracted much attention in recent years and initially the swarming solution is based on data-driven schemes. This paper presents a new request-driven swarming scheme. The scheme offers several advantages: high efficiency in data delivery, low control overhead, and flexibility in streaming control. Some key issues on the design of the relevant scheduling and forwarding mechanisms are discussed. In addition, we present a scheduling model for this scheme and propose some heuristics. Our evaluation of these heuristics reveals that there is a tradeoff among performance metrics such as delivery reliability, loading on the underlay network, and on the streaming overlay network. Comparing our proposed request-driven scheme with an existing data-driven scheme using computer simulation, we find that our request-driven scheme incurs much lower control overhead while providing comparable or better efficiency in data delivery.