A Usable Security Heuristic Evaluation for the Online Health Social Networking Paradigm

The security, privacy, and usability of health social networking websites remain key concerns for their successful adoption by patients. As far as it is known, no studies have developed a usable security heuristic evaluation to test these websites for the usability of their security and privacy features. This article describes the development of such a usable security heuristic evaluation in the context of the online health social networking paradigm. The tailored method approach was applied to transform themes identified from relevant literature and convert them into high-level heuristics for usable security. Checklist items were created for each high-level heuristic. The result is 13 high-level heuristics with individualized checklist items that help examine usable security. The results can identify which security and privacy features are lacking from a usability perspective and would, therefore, require design improvement. The improvements would enhance the users' experiences, which in turn can assist in the adoption of health social networking websites.     

Developing SMASH: A set of SMArtphone's uSability Heuristics

The smartphone market is nowadays highly competitive. When buying a new device, users focus on visual esthetics, ergonomics, performance, and user experience, among others. Assessing usability issues allows improving these aspects. One popular method for detecting usability problems is heuristic evaluation, in which evaluators employ a set of usability heuristics as guide. Using proper heuristics is highly relevant. In this paper we present SMASH, a set of 12 usability heuristics for smartphones and mobile applications, developed iteratively. SMASH (previously named TMD: Usability heuristics for Touchscreen-based Mobile Devices) was experimentally validated. The results support its utility and effectiveness.     

The scenario‐based usability checklist development for home appliance design: A case study

In the past decade, home appliances have been rapidly developed to satisfy the various requirements of users. Thus, there are increases in requirements to explore appropriate evaluation methods that reflect the usability of home appliances quickly and comprehensively. This study aims to develop a scenario‐based usability checklist for product designers early in the design process. In this study, the scenario‐based usability checklist consists of two parts: 1) a heuristic evaluation checklist, and 2) a scenario evaluation checklist. In heuristic evaluation checklist development, usability factors of home appliances are extracted and then coupled to user interface (UI) elements. In scenario evaluation checklist development, scenarios are developed through brainstorming by focus group interview (FGI), and then evaluation elements are extracted from previous scenarios analysis. The proposed scenario‐based usability checklists can enable designers to evaluate product design quickly and comprehensively early in the development process with users' viewpoints. © 2010 Wiley Periodicals, Inc.     

Understanding Usability and User Experience of Web-Based 3D Graphics Technology

As Web-based interactive 3D graphics (Web 3D), popularly referred to as Virtual Reality, continue to become more affordable, research and development groups in various fields have been adopting Web 3D technology. In addition to simulation of 3D content, the ability to instantly display alternative looks has been recognized as an innovative way to improve communication in such fields as product design, architecture, and e-commerce. Despite substantial adoption of Web 3D, how and how much the technology benefits target users as well as the providers who choose to adapt the Web 3D technology are not well understood. Previous research has established that interactive 3D graphics provide users with unique human–computer interaction (HCI). However, little is known about how users experience the Web 3D graphics technology and how user–system interaction contributes to system usability. The purpose of this study is to build new knowledge of the user experience with interactive-3D graphics systems used for product demonstration. By testing the impact of the technology on the user–system interaction and usability and comparing this impact with that of conventional two-dimensional (2D) graphics, this study tries to better understand the Web 3D technology from an interdisciplinary view of technology acceptance, sense of presence, and HCI. The study investigated how system usability is affected by HCI in the context of a furniture-style preference survey. The results of the study display the clear advantage for Web 3D for usability and show that perceived usefulness and sense of presence both mediate the effect of the technology treatment on the usability outcomes. The contribution of this study is that it includes empirical data to show how Web 3D benefits users when adopted in the context of a product demonstration and how the advantage is obtained through the user's interaction with the Web 3D technology.     

Extending the Usability of Heuristics for Design and Evaluation: Lead,Follow, and Get Out of the Way

User-centered design practitioners have often relied on discount usability engineering methods using heuristics. Top 10 lists of design and evaluation heuristics have proliferated during the 1990s, leading to a plethora of heterogeneous heuristic guidelines for a multitude of user interfaces. A simple, unified set of heuristics that might be applicable across different technologies, understood between disciplines, and equivalent to metrics used to measure ease of use can be beneficial.

Lead, follow, and get out of the way (LF&G) theorizes that the optimal human-computer interaction (HCI) experience is analogous to a facilitative learning relation-Like a good teacher, mentor, or coach, the usable user interface leads the user to successful completion of tasks and goals; follows the user's progress and provides appropriate feedback and information when needed; and gets out of the way of the user to allow efficient and effective completion of tasks as the user attains mastery of the system, its concepts, and operations. A set of 18 heuristics grouped under the 3 general principles of the title are provided for use across the tasks of design guidance, development refinement, and end-user evaluation of computer systems.

This article provides background into the development of LF&G, case studies on its use in real-world product design and development, and directions for further research needed to develop this approach into an empirically based method for defining, describing, designing, and predicting the ease of use of interactive human-computer systems.     

Understanding usability issues addressed by three user-system interface evaluation techniques

Three structured judgment evaluation techniques were applied to a system with a graphical direct manipulation style interface, to understand the types of usability problems they address. These evaluation techniques were cognitive walkthrough, heuristic evaluation, and the Smith and Mosier (1986) guidelines. The authors wanted to learn whether the techniques identify problems: across all stages of user activity, which noticeably affect users' performance with the system, and which are important to the usability of direct manipulation-style systems. Results showed that the cognitive walkthrough method identifies issues almost exclusively within the action specification stage, while guidelines covered more stages. The walkthrough was best, however, and the guidelines worst at predicting problems that cause users noticeable difficulty (as observed during a usability study). All the techniques could be improved in assessing semantic distance and addressing all stages on the evaluation side of the HCI activity cycle. To evaluate the directness of engagements, improved or new techniques are needed.     

Usability Inspection by Metaphors of Human Thinking Compared to Heuristic Evaluation

A new usability inspection technique based on metaphors of human thinking has been experimentally compared to heuristic evaluation (HE). The aim of metaphors of thinking (MOT) is to focus inspection on users' mental activity and to make inspection easily applicable to different devices and use contexts. Building on classical introspective psychology, MOT bases inspection on metaphors of habit formation, stream of thought, awareness and associations, the relation between utterances and thought, and knowing. An experiment was conducted in which 87 novices evaluated a large Web application, and its key developer assessed the problems found. Compared to HE, MOT uncovered usability problems that were assessed as more severe for users and also appeared more complex to repair. The evaluators using HE found more cosmetic problems. The time spent learning and performing an evaluation with MOT was shorter. A discussion of strengths and weaknesses of MOT and HE is provided, which shows how MOT can be an effective alternative or supplement to HE.     

Recommender Systems: An Introduction,by Dietmar Jannach,Markus Zanker,Alexander Felfernig,and Gerhard Friedrich

In addressing user experience issues, users’ perceptions and emotions need to be considered important. This study examines the relationships between perceived usability/aesthetics and emotional valence/arousal/engagement through an experiment using 15 existing websites from various domains and questionnaire items developed to measure users’ responses. According to the experimental results, both perceived usability and perceived aesthetics were positively correlated with emotional valence and negatively correlated with emotional engagement. No specific relationship was found between perceived usability/aesthetics and emotional arousal. Perceived aesthetics potentially had a greater impact on valence than perceived usability. Unlike valence, engagement could be more influenced by perceived usability than by perceived aesthetics. These findings can be utilized as bases for applying users’ emotional responses in each dimension to the product-use situations in the chain of perceptions, emotions, and behaviors.     

A Textual Feedback Tool for Empowering Participants in Usability and UX Evaluations

The usability movement has historically always sought to empower end-users of computers so that they understand what is happening and can control the outcome. In this article, we develop and evaluate a “Textual Feedback” tool for usability and user experience (UX) evaluation that can be used to empower well-educated but low-status users in UX evaluations in countries and contexts with high power distances. The proposed tool contributes to the Human–Computer Interaction (HCI) community’s pool of localized UX evaluation tools. We evaluate the tool with 40 users from two socio-economic groups in real-life UX usability evaluations settings in Malaysia. The results indicate that the Textual Feedback tool may help participants to give their thoughts in UX evaluation in high power distance contexts. In particular, the Textual Feedback tool helps high status females and low status males express more UX problems than they can with traditional concurrent think aloud (CTA) alone. We found that classic concurrent think aloud UX evaluation works fine in high power contexts, but only with the addition of Textual Feedback to mitigate the effects of socio-economic status in certain user groups. We suggest that future research on UX evaluation look more into how to empower certain user groups, such as low status female users, in UX evaluations done in high power distance contexts.     

Effectiveness of user testing and heuristic evaluation as a function of performance classification

For different levels of user performance, different types of information are processed and users will make different types of errors. Based on the error's immediate cause and the information being processed, usability problems can be classified into three categories. They are usability problems associated with skill-based, rule-based and knowledge-based levels of performance. In this paper, a user interface for a Web-based software program was evaluated with two usability evaluation methods, user testing and heuristic evaluation. The experiment discovered that the heuristic evaluation with human factor experts is more effective than user testing in identifying usability problems associated with skill-based and rule-based levels of performance. User testing is more effective than heuristic evaluation in finding usability problems associated with the knowledge-based level of performance. The practical application of this research is also discussed in the paper.     

A Usability Checklist for the Usability Evaluation of Mobile Phone User Interface

In the last decade, the research of the usability of mobile phones has been a newly evolving area with few established methodologies and realistic practices that ensure capturing usability in evaluation. Thus, there exists growing demand to explore appropriate evaluation methodologies that evaluate the usability of mobile phones quickly as well as comprehensively. This study aims to develop a task-based usability checklist based on heuristic evaluations in views of mobile phone user interface (UI) practitioners. A hierarchical structure of UI design elements and usability principles related to mobile phones were developed and then utilized to develop the checklist. To demonstrate the practical effectiveness of the proposed checklist, comparative experiments were conducted on the usability checklist and usability testing. The majority of usability problems found by usability testing and additional problems were discovered by the proposed checklist. It is expected that the usability checklist proposed in this study could be used quickly and efficiently by usability practitioners to evaluate the mobile phone UI in the middle of the mobile phone development process.     

Expert vs novice collaborative heuristic evaluation (CHE) of a smartphone app for cultural heritage sites

This study was conducted to compare CHE between Human-Computer Interaction (HCI) experts and novices in evaluating the Smartphone app for the cultural heritage site. It uses the Smartphone Mobile Application heuRisTics (SMART), focusing on smartphone applications and traditional Nielsen heuristics, focusing on a wider range of interactive systems. Six experts and six novices used the severity rating scale to categorise the severity of the usability issues. These issues were mapped to both heuristics. The study found that experts and novice evaluators identified 19 and 14 usability issues, respectively, with ten as the same usability issues. However, these same usability issues have been rated differently. Although the t-test indicates no significant differences between experts and novices in their ratings for usability issues, these results nevertheless indicate the need for both evaluators in CHE to provide a more comprehensive perspective on the severity of the usability issues. Furthermore, the mapping of the usability issues for Nielsen and SMART heuristics concluded that more issues with the smartphone app could be addressed through smartphone-specific heuristics than general heuristics, indicating a better tool for heuristic evaluation of the smartphone app. This study also provides new insight into the required number of evaluators needed for CHE.

     

Tracking the effectiveness of usability evaluation methods

. We present a case study that tracks usability problems predicted with six usability evaluation methods (claims analysis, cognitive walkthrough, GOMS, heuristic evaluation, user action notation, and simply reading the specification) through a development process. We assess the method's predictive power by comparing the predictions to the results of user tests. We assess the method's persuasive power by seeing how many problems led to changes in the implemented code. We assess design-change effectiveness by user testing the resulting new versions of the system. We concludethatpredictivemethodsarenot as effective as the HCI field would like and discuss directions for future research.     

A heuristic checklist for an accessible smartphone interface design

Smartphone technology has evolved into a multi-functional device with advanced capabilities, but this mobile technology remains inaccessible to many individuals with visual impairments or upper extremity disabilities. This paper provides a heuristic checklist for accessible smartphone interface design, developed through reviewing existing design standards and guidelines and validating these guidelines with user involvement. Specifically, a set of preliminary user requirements (59 items) was extracted from existing standards, guidelines, and user requirements regarding mobile handheld device accessibility. Subsequently, the requirement set was filtered using a participatory method and then integrated to create an operational version of design guidelines. These guidelines were then used in a heuristic evaluation and usability testing on high-fidelity prototypes produced by a commercial manufacturer. A heuristic checklist for designing accessible smartphones was formed, which may also be applicable to other touchscreen handheld devices (e.g., printer screen) in terms of accessibility features. The initial set of 59 user requirements was re-organized into 44 statements in six general categories: mechanical controls, display, speech and general operation controls, audio feedback controls, touch-operated controls, and others. Using results from both qualitative and quantitative methods provides support, though with some limitations, for this accessibility checklist. This checklist is intended as a practical design support tool for use in early design phases of handheld products. A number of challenges and limitations are discussed as well.     

Ethnography,Scenario-Based Observational Usability Study,and Other Reviews Inform the Design of a Web-Based E-Notebook

As users turn to the World Wide Web to accomplish an increasing variety of daily tasks, many engage in information assimilation (IA), a process defined as the gathering, editing, annotating, organizing, and saving of Web information, and the tracking of ongoing Web work processes. The process of IA, which is similar to traditional note taking but in the Web environment, emerges from a literature review and an ethnographic field study, as presented in this article. Despite strong evidence which suggests that IA is critical to many Web users, however, a scenario-based observational usability study and a heuristic evaluation indicate that it is currently not well supported by existing software applications. This article, which culminates in the presentation of NetNotes-a Web-based e-notebook developed specifically to support the process of IA-illustrates how design requirements can be effectively extracted and synthesized from a variety of complementary background user studies.     

Integration of Usability Evaluation Studies via a Novel Meta-Analytic Approach: What are Significant Attributes for Effective Evaluation?

The overall discovery rates, which are the ratios of sum of unique usability problems detected by all experiment participants against the number of usability problems existed in the evaluated systems, were investigated to find significant factors of usability evaluation through a meta-analytic approach with the n-corrected effect sizes newly defined in this study. Since many studies of usability evaluation have been conducted under specific contexts showing some mixed findings, usability practitioners need holistic and more generalized conclusions. Due to the limited applicability of the traditional meta-analysis to usability evaluation studies, a new meta-analytic approach was established and applied to 38 experiments that reported overall discovery rates of usability problems as a criterion measure. Through the meta-analytic approach with the n-corrected effect sizes, we successfully combined 38 experiments and found evaluator's expertise, report type, and interaction between usability evaluation method and time constraint as significant factors. We suggest that in order to increase overall discovery rates of usability problems, (a) free-style written reports are better than structured written reports; (b) when heuristic evaluation or cognitive walkthrough is used, the usability evaluation experiments should be conducted without time constraint, but when think-aloud is used, time constraint is not an important experimental condition; (c) usability practitioners do not need to be concerned about unit of evaluation, fidelity of evaluated systems, and task type; and (d) HCI experts are better than novice users or evaluators. Our conclusions can guide usability practitioners when determining evaluation contexts, and the meta-analytic approach of this study provides an alternative way to combine the empirical results of usability evaluation besides the traditional meta-analysis.     

Tracking the effectiveness of usability evaluation methods

Abstract

We present a case study that tracks usability problems predicted with six usability evaluation methods (claims analysis, cognitive walkthrough, GOMS, heuristic evaluation, user action notation, and simply reading the specification) through a development process. We assess the method's predictive power by comparing the predictions to the results of user tests. We assess the method's persuasive power by seeing how many problems led to changes in the implemented code. We assess design-change effectiveness by user testing the resulting new versions of the system. We concludethatpredictivemethodsarenot as effective as the HCI field would like and discuss directions for future research.     

Current practice in measuring usability: Challenges to usability studies and research

How to measure usability is an important question in HCI research and user interface evaluation. We review current practice in measuring usability by categorizing and discussing usability measures from 180 studies published in core HCI journals and proceedings. The discussion distinguish several problems with the measures, including whether they actually measure usability, if they cover usability broadly, how they are reasoned about, and if they meet recommendations on how to measure usability. In many studies, the choice of and reasoning about usability measures fall short of a valid and reliable account of usability as quality-in-use of the user interface being studied. Based on the review, we discuss challenges for studies of usability and for research into how to measure usability. The challenges are to distinguish and empirically compare subjective and objective measures of usability; to focus on developing and employing measures of learning and retention; to study long-term use and usability; to extend measures of satisfaction beyond post-use questionnaires; to validate and standardize the host of subjective satisfaction questionnaires used; to study correlations between usability measures as a means for validation; and to use both micro and macro tasks and corresponding measures of usability. In conclusion, we argue that increased attention to the problems identified and challenges discussed may strengthen studies of usability and usability research.     

Must evaluation methods be about usability? Devising and assessing the utility inspection method

Whereas research in usability evaluation abounds, few evaluation approaches focus on utility. We present the utility inspection method (UIM), which prompts evaluators about the utility of the system they evaluate. The UIM asks whether a system uses global platforms, provides support infrastructure, is robust, gives access to rich content, allows customisation, offers symbolic value and supports companionship among users and between users and developers. We compare 47 participants’ use of UIM and heuristic evaluation (HE). The UIM helps identify more than three times as many problems as HE about the context of activities; HE helps identify 2.5 times as many problems as UIM about the interface. Usability experts consider the problems found with UIM more severe and more complex to solve compared to those found with HE. We argue that UIM complements existing usability evaluation methods and discuss future research on utility inspection.     

Better discount evaluation: illustrating how critical parameters support heuristic creation

This paper describes a heuristic creation process based on the notion of critical parameters, and a comparison experiment that demonstrates the utility of heuristics created for a specific system class. We focus on two examples of using the newly created heuristics to illustrate the utility of the usability evaluation method, as well as to provide support for the creation process, and we report on successes and frustrations of two classes of users, novice evaluators and domain experts, who identified usability problems with the new heuristics. We argue that establishing critical parameters for other domains will support efforts in creating tailored evaluation tools.