Retrospective vs. concurrent think-aloud protocols: Testing the usability of an online library catalogue

Think-aloud protocols are a dominant method in usability testing. There is, however, only little empirical evidence on the actual validity of the method. This paper describes an experiment that compares concurrent and retrospective think-aloud protocols for a usability test of an online library catalogue. There were three points of comparison: usability problems detected, overall task performance, and participant experiences. Results show that concurrent and retrospective think-aloud protocols reveal comparable sets of usability problems, but that these problems come to light in different ways. In retrospective think-aloud protocols, more problems were detected by means of verbalisation, while in concurrent think-aloud protocols, more problems were detected by means of observation. Moreover, in the concurrent think-aloud protocols, the requirement to think aloud while working had a negative effect on the task performance. This raises questions about the reactivity of concurrent think-aloud protocols, especially in the case of high task complexity.     

Retrospective vs. concurrent think-aloud protocols: testing the usability of an online library catalogue

Think-aloud protocols are a dominant method in usability testing. There is, however, only little empirical evidence on the actual validity of the method. This paper describes an experiment that compares concurrent and retrospective think-aloud protocols for a usability test of an online library catalogue. There were three points of comparison: usability problems detected, overall task performance, and participant experiences. Results show that concurrent and retrospective think-aloud protocols reveal comparable sets of usability problems, but that these problems come to light in different ways. In retrospective think-aloud protocols, more problems were detected by means of verbalisation, while in concurrent think-aloud protocols, more problems were detected by means of observation. Moreover, in the concurrent think-aloud protocols, the requirement to think aloud while working had a negative effect on the task performance. This raises questions about the reactivity of concurrent think-aloud protocols, especially in the case of high task complexity.     

Employing think-aloud protocols and constructive interaction to test the usability of online library catalogues: a methodological comparison

This paper describes a comparative study of three usability test approaches: concurrent think-aloud protocols, retrospective think-aloud protocols, and constructive interaction. These three methods were compared by means of an evaluation of an online library catalogue, which involved four points of comparison: number and type of usability problems detected; relevance of the problems detected; overall task performance; and participant experiences. The results of the study showed that there were only few significant differences between the usability test approaches, mainly with respect to manner of problem detecting, task performance and participant experience. For the most part, the usability methods proved very much comparable, revealing similar numbers and types of problems that were equally relevant. Taking some practical aspects into account, a case can be made for preferring the concurrent think-aloud protocols over the other two methods.     

Making the law accessible to non-lawyers: effects of different kinds of expertise on perceived usability of online legal information services

To help design an environment in which professionals without legal training can make effective use of public sector legal information on planning and the environment – for Add-Wijzer, a European e-government project – we evaluated their perceptions of usefulness and usability. In concurrent think-aloud usability tests, lawyers and non-lawyers carried out information retrieval tasks on a range of online legal databases. We found that non-lawyers reported twice as many difficulties as those with legal training (p = 0.001), that the number of difficulties and the choice of database affected successful completion, and that the non-lawyers had surprisingly few problems understanding legal terminology. Instead, they had more problems understanding the syntactical structure of legal documents and collections. The results support the constraint attunement hypothesis (CAH) of the effects of expertise on information retrieval, with implications for the design of systems to support the effective understanding and use of information.     

Why usability gets lost or usability in in-house software development

This study tries to shed some light on what happens to usability and occupational health issues in a bespoke software development project. Usability is an essential quality in software, in particular in a work context where poor usability and other risk factors related to the software and computers may cause health problems. We have interviewed a number of software developers, usability people and users about their attitudes to and practices for integrating usability and users' health concerns in software development. The interviews were conducted in two Swedish organisations with in-house development of bespoke software. Our main conclusion is that several factors combine to push usability and occupational health matters aside, some of which are attitudes to usability and users' health issues, unclear responsibilities, poor support for user-centeredness and usability in software development models, ineffective user participation and usability and users' health being ignored or forgotten in decisions about the software, its use and its design.     

Empirical studies to investigate the usability of text- and image-based CAPTCHAs

This article reports the findings of two empirical studies investigating the usability of image-based and text-based CAPTCHAs that authenticate users accessing the web. The first investigated four image-based CAPTCHAs, Asirra, ESP-PIX, SQ-PIX and IMAGINATION, with twenty participants interacting with five of each type on a desktop computer. The participants then completed the computer system usability questionnaire (CSUQ), the NASA Task Load Index (NASA-TLX), and a final post-test subjective questionnaire ranking the CAPTCHAs based on preference. The results found statistically significant differences among the CAPTCHAs for all dependent variables except task completion time, with participants preferring Asirra and ESP-PIX. The second compared the usability of these two with the two most commonly used text-based CAPTCHAs, ReCAPTCHA and Google's CAPTCHA. After receiving training on Asirra and ESP-PIX, twenty-four participants completed 10 trials with each type on a desktop computer, subsequently completing the CSUQ and the NASA-TLX followed by a retrospective think-aloud session. The results of this second study indicated that Google's CAPTCHA and ReCAPTCHA took the least amount of time to complete. The workload associated with Asirra and ReCAPTCHA was lower than that for ESP-PIX and Google's CAPTCHA. Perceived overall usability tended to be highest for Asirra and ReCAPTCHA. Participant comments indicated that Google's CAPTCHA was too visually distorted and ESP-PIX had commonalities between the images that weren't immediately obvious, factors perhaps contributing to their lower perceived usability scores. Overall, ReCAPTCHA is recommended for use on a desktop computer because of its high perceived usability, low completion time, and low workload.     

Interacting With Hypertext: A Meta-Analysis of Experimental Studies

The meta-analysis compared and synthesized the results of 23 experimental studies on hypertext. The analysis was based on 56 pairs of effect sizes and significance levels of the impact of users, tasks, and tools on interactions with hypertext. This analysis focused on three factors that prevailingly influence the use of hypertext: the cognitive styles and spatial ability of users; the complexity of tasks; and the structure of information organization and the visualization of the structure. The meta-analysis found that this group of experimental studies reported significantly discrepant findings, indicating that substantial differences exist among individual experiments. Individual differences in cognition did not yield enough evidence to conclude that the effect sizes are significantly apart from zero. The meta-analysis showed that the overall performance of hypertext users tended to be more effective than that of nonhypertext users, but the differences in efficiency measures were consistently in favor of nonhypertext users. Users benefited more from hypertext tools for open tasks. Overall, the complexity of tasks has the largest combined effect sizes. Graphical maps that visualize the organization of hypertext have significant impact on the usefulness of a hypertext system. This meta-analysis raised two issues concerned with the present hypertext literature: (a) the absence of a taxonomy of tasks for analyzing and comparing hypertext usability across studies, and (b) the weaknesses of the connections between abstract hypertext reference models and specific hypertext systems. These weaknesses may considerably undermine the significance of individual findings on hypertext usability. Results of the meta-analysis suggest that the discrepancies among empirical findings are related to these weaknesses. Future work on hypertext usability should emphasize task taxonomies along with longitudinal and ethnographic studies for a deep understanding of the interactions between users and hypertext. Recommended research issues for the future are highlighted in Section 5.     

Problem Prioritization in Usability Evaluation: From Severity Assessments Toward Impact on Design

Severity assessments enable prioritization of problems encountered during usability evaluations, and thereby provide a device for guiding the utilization of design resources. However, designers' response to usability evaluations is also influenced by other factors, which may overshadow severity. With the purpose of enhancing the impact of severity assessments, this study combines a field study of factors that influence the impact of evaluations with an experimental study of severity assessments made during usability inspections. The results show that even in a project receptive to input from evaluations, their impact was highly dependent on conducting evaluations early. This accorded with an informal method that blended elements of usability evaluation and participatory design and could be extended with user-made severity assessments. The major cost associated with the evaluations was not finding but fixing problems, emphasizing that, to be effective, severity assessments must be reliable, valid, and sufficiently persuasive to justify the cost of fixing problems. For the usability inspections, evaluators' ratings of problem impact and persistence were weakly correlated with the number of evaluators reporting a problem, indicating that different evaluators represent different subgroups of users or alternatively that evaluator-made severity assessments are of questionable reliability. To call designers' attention to the severe problems, the halving of the severity sum is proposed as a means of visualizing the large payoff of fixing a high-severity problem and, conversely, the modest potential of spending resources on low-severity problems.     

The impact of two different think-aloud instructions in a usability test: a case of just following orders?

The instructions used in think-aloud studies can range from a simple request to think-aloud, to an explicit instruction to include certain types of content. The present study compared two think-aloud instructions: the classic neutral think-aloud instruction and an explicit instruction requesting explanations and content that is relevant to the user experience. Data from task performance, mental workload, think-aloud protocols and usability problems were collected from 16 participants, equally distributed between the two think-aloud instruction conditions. No differences were found in task performance, however, participants in the explicit instruction condition reported higher mental workload and a focus on finding interface problems. The explicit instruction condition also yielded more utterances about the user experience, expectations and explanations of behaviour than the neutral condition. An analysis of the resultant usability problems revealed that the explicit instruction led to a larger number of dialogue, navigation, layout and functionality problems, but that the problems which were unique to this condition were, in the main, at a low level of severity.     

Understanding usability issues addressed by three user-system interface evaluation techniques

Three structured judgment evaluation techniques were applied to a system with a graphical direct manipulation style interface, to understand the types of usability problems they address. These evaluation techniques were cognitive walkthrough, heuristic evaluation, and the Smith and Mosier (1986) guidelines. The authors wanted to learn whether the techniques identify problems: across all stages of user activity, which noticeably affect users' performance with the system, and which are important to the usability of direct manipulation-style systems. Results showed that the cognitive walkthrough method identifies issues almost exclusively within the action specification stage, while guidelines covered more stages. The walkthrough was best, however, and the guidelines worst at predicting problems that cause users noticeable difficulty (as observed during a usability study). All the techniques could be improved in assessing semantic distance and addressing all stages on the evaluation side of the HCI activity cycle. To evaluate the directness of engagements, improved or new techniques are needed.     

Effects of the design of mobile security notifications and mobile app usability on users’ security perceptions and continued use intention

The explosive global adoption of mobile applications (i.e., apps) has been fraught with security and privacy issues. App users typically have a poor understanding of information security; worse, they routinely ignore security notifications designed to increase security on apps. By considering both mobile app interface usability and mobile security notification (MSN) design, we investigate how security perceptions of apps are formed and how these perceptions influence users’ intentions to continue using apps. Accordingly, we designed and conducted a set of controlled survey experiments with 317 participants in different MSN interface scenarios by manipulating the types of MSN interfaces (i.e., high vs. low disruption), the context (hedonic vs. utilitarian scenarios), and the degree of MSN intrusiveness (high vs. low intrusiveness). We found that both app interface usability and the design of MSNs significantly impacted users’ perceived security, which, in turn, has a positive influence on users’ intention to continue using the app. In addition, we identified an important conundrum: disruptive MSNs—a common approach to delivering MSNs—irritate users and negatively influence their perceptions of app security. Thus, our results directly challenge current practice. If these results hold, current practice should shift away from MSNs that interrupt task performance.     

Contextual Navigation Aids for Two World Wide Web Systems

In spite of the radical enhancement of Web technologies, many users still continue to experience severe difficulties in navigating Web systems. One way to reduce the navigation difficulties is to provide context information that explains the current situation of Web users. In this study, we empirically examined the effects of 2 types of context information, structural and temporal context. In the experiment, we evaluated the effectiveness of the contextual navigation aids in 2 different types of Web systems, an electronic commerce system that has a well-defined structure and a content dissemination system that has an ill-defined structure. In our experiment, participants answered a set of postquestionnaires after performing several searching and browsing tasks. The results of the experiment reveal that the 2 types of contextual navigation aids significantly improved the performance of the given tasks regardless of different Web systems and different task types. Moreover, context information changed the users' navigation patterns and increased their subjective convenience of navigation. This study concludes with implications for understanding the users' searching and browsing patterns and for developing effective navigation systems.     

Understanding User Experience with Computer-Based Applications with Different Use Purposes

The concept of user experience emphasizes the importance of understanding users for applications that have various contextual features. To address this issue, this study examines the changes in the relationships among user satisfaction and users' perceptions of usability and aesthetics according to use situations. For data, an experiment was conducted using 15 existing websites with similar levels of usability and aesthetics. Forty-five engineering students participated in the experiment. The results indicated that the relationships among perceived usability, perceived aesthetics, and user satisfaction could be dependent on how users perceived the use purposes and interaction types of the websites. Specifically, the relationship between perceived usability and user satisfaction was stronger for websites requiring users' goal-directed activities, whereas the relationship between perceived aesthetics and user satisfaction was stronger for the websites mainly providing useful information, regardless of the existence of a specific use goal. Also, the strong relationship between perceived usability and perceived aesthetics was more obvious for websites where users mainly wanted to obtain information than for those where users primarily wanted to use the leisure and relaxation content. The findings can be utilized in the design of websites with different contextual features, especially related to use purposes and interaction types.     

What we know about dyslexia and Web accessibility: a research review

Compared to the online interaction behavior of other users, little is known about the difficulties dyslexic Web users encounter online. This paper reviews existing literature at the intersection of dyslexia and accessibility research to determine what useful knowledge exists regarding this important and relatively large group of users. This review uncovers that, although there are few published usability tests with dyslexic users, there is a considerable body of knowledge on dyslexia as well as many design guidelines for authoring dyslexic-accessible interfaces. Through a comparison of existing accessibility guidelines for dyslexic and non-dyslexic users and discussion of the plain language movement, it is argued that dyslexic-accessible practices may redress difficulties encountered by all Internet users. This conclusion suggests that usability testing yielding a clearer profile of the dyslexic user would further inform the practice of universal design, but also that enough knowledge is already available to allow doing more to increase accessibility for dyslexic Internet users.     

Designing universal access: web-applications for the elderly and disabled

Throughout a person’s life they are likely to experience some degree of impairment. What must be ensured is that these individuals can benefit from the opportunities afforded by technology in the same way as those who are able bodied. With the Internet becoming an increasingly popular medium for organizations to deliver information and services to the public many of these technological benefits are provided through web applications. However, due to possible visual, physical or cognitive impairments, older users and those with disabilities are more likely to experience difficulties. This paper addresses issues relating to usability of web applications for elderly and disabled users and investigates ways in which it can be improved, and whether doing so will affect usability for younger and more able users. Results from experiments undertaken indicated that it was possible to develop a web application for elderly and disabled users without degrading usability for other users.     

Supporting novice usability practitioners with usability engineering tools

Although usability practitioners have successfully applied usability engineering processes to increase the usability of interaction designs, the literature suggests that usability practitioners experience a number of difficulties that negatively impact the effectiveness of their work, which in turn affects the effectiveness of the usability engineering processes within which they work. These difficulties include identifying and recording critical usability data and understanding and establishing relationships among usability data. These difficulties are particularly pronounced for novice usability practitioners. One approach to addressing these difficulties is through appropriate usability engineering tool support. We argue that existing usability engineering tools offer excellent support for helping experienced usability practitioners more efficiently do their jobs, but they provide little support for helping novice usability practitioners more effectively do their jobs. We introduce a tool feature, usability problem instance records, to better support novice usability practitioners. We describe the results of a study of this feature, which suggest that the feature helps to improve two aspects of the effectiveness of novice usability practitioners: reliability and quality.     

A qualitative study of smartwatch usage and its usability

This study aims to explore usability issues of watch‐type wearable devices and to suggest guidelines for improved operation of smartwatches. To do so, we conducted a series of surveys, interviews, and task performance experiments. Thirty smartwatch users from ages 20 to 43 years were recruited. Users’ experiences of smartwatches were collected via a weeklong, online‐based diary study, which consisted of various tasks to be completed while smartwatches were in use. Our study assessed usability problems associated with those tasks, concurrent tasks conducted while interacting with smartwatches, pain points/discomfort that users had while interacting with their devices, and requirements/requests of the smartwatch users. During the week of tracking, participants were asked to complete the usability evaluation three times a day using usability principles we designed for the study: information display, control, learnability, interoperability, and preference. In addition, task performance tests were conducted for the tasks most frequently conducted on touch‐based displays: number entry, swiping, and scrolling. Specific usability issues of smartwatches were identified and summarized for each usability principle by triangulating survey, interview, and task performance evaluation results. Based on the insights from the results of the study, we conclude by suggesting guidelines for further enhancing users’ experience of future smartwatches.     

Defining and classifying IT interaction disability

Many people have significant difficulties while using information technology (IT) devices. This study suggests a new concept of IT interaction disability (ITID) to understand the significant difficulties that users may encounter while using IT devices. The task, interface, and context (TIC) framework for classifying the types of ITID is also developed; this framework includes three components: tasks, user interface elements, and context factors. A case study in which visually impaired people used smartphones was conducted to illustrate the types of ITID and to show the applicability of the TIC framework. The result of this study can provide insights to researchers to improve the understanding of the significant difficulties of using IT devices and to analyse these difficulties systematically.     

Video-based hypermedia: guiding design with users' questions

Abstract

Users' questions can provide important design input. In this paper we examine the questions that users asked after viewing a series of seven video clips and then classify them into three main categories: questions relating to perception difficulties; questions concerning lack of understanding due to some form of dissonance and questions requiring more content information. There are three benefits from analyzing these questions, which will enable designers to improve their designs. First, our approach provides a breakdown of die questions that users ask which can be used by designers to determine what kind of additional information should be made available. Second, it provides a user-centered way of determining where to make links. Third, it enables designers to detect usability and pedagogic problems early in the design of video-based hypermedia.