Measuring Perceived Usability: The CSUQ,SUS, and UMUX

The primary purpose of this research was to investigate the relationship between two widely used questionnaires designed to measure perceived usability: the Computer System Usability Questionnaire (CSUQ) and the System Usability Scale (SUS). The correlation between concurrently collected CSUQ and SUS scores was 0.76 (over 50% shared variance). After converting CSUQ scores to a 0–100-point scale (to match the range of the SUS scores), there was a small but statistically significant difference between CSUQ and SUS means. Although this difference (just under 2 scale points out of a possible 100) was statistically significant, it did not appear to be practically significant. Although usability practitioners should be cautious pending additional independent replication, it appears that CSUQ scores, after conversion to a 0–100-point scale, can be interpreted with the Sauro–Lewis curved grading scale. As a secondary research goal, investigation of variations of the Usability Metric for User Experience (UMUX) replicated previous findings that the regression-adjusted version of the UMUX-LITE (UMUX-LITEr) had the closest correspondence with concurrently collected SUS scores. Thus, even though these three standardized questionnaires were independently developed and have different item content and formats, they largely appear to be measuring the same thing, presumably, perceived usability.     

Measuring Perceived Usability: SUS,UMUX, and CSUQ Ratings for Four Everyday Products

This research continued previous investigation of the relationships among measures of perceived usability: the System Usability Scale (SUS), three metrics derived from the Usability Metric for User Experience (UMUX), and the Computer System Usability Questionnaire (CSUQ), this time with ratings of four everyday products (Excel, Word, Amazon, and Gmail). SUS ratings of these products were generally consistent with previous reports. Significant differences in SUS means across studies could be due to differences in frequency of use, with implications for using these data as usability benchmarks. Correspondence among the various measures of perceived usability was also consistent with previous research. Considering frequency of use, mean differences ranged from -2.0 to 1.8 (average shift in Sauro-Lewis grade range from -0.6 to 0.8). When SUS scores were above average, the range restriction of the UMUX-LITEr led to relatively large discrepancies with SUS, suggesting it might not always be better than unadjusted UMUXLITE.     

Psychometric Evaluation of the T-CSUQ: The Turkish Version of the Computer System Usability Questionnaire

This article describes the development of a standardized computer system usability questionnaire for use with speakers of the Turkish language, the Turkish Computer System Usability Questionnaire, Short Version (T-CSUQ-SV). This new questionnaire, based on the English-language CSUQ, underwent careful translation and transformation through comprehensive psychometric evaluation. The results of the psychometric evaluation revealed an acceptable level of reliability, appropriate construct validity, and sensitivity to manipulation, indicating that Turkish usability practitioners should be able to use the T-CSUQ-SV with confidence when conducting user research.     

Empirical studies to investigate the usability of text- and image-based CAPTCHAs

This article reports the findings of two empirical studies investigating the usability of image-based and text-based CAPTCHAs that authenticate users accessing the web. The first investigated four image-based CAPTCHAs, Asirra, ESP-PIX, SQ-PIX and IMAGINATION, with twenty participants interacting with five of each type on a desktop computer. The participants then completed the computer system usability questionnaire (CSUQ), the NASA Task Load Index (NASA-TLX), and a final post-test subjective questionnaire ranking the CAPTCHAs based on preference. The results found statistically significant differences among the CAPTCHAs for all dependent variables except task completion time, with participants preferring Asirra and ESP-PIX. The second compared the usability of these two with the two most commonly used text-based CAPTCHAs, ReCAPTCHA and Google's CAPTCHA. After receiving training on Asirra and ESP-PIX, twenty-four participants completed 10 trials with each type on a desktop computer, subsequently completing the CSUQ and the NASA-TLX followed by a retrospective think-aloud session. The results of this second study indicated that Google's CAPTCHA and ReCAPTCHA took the least amount of time to complete. The workload associated with Asirra and ReCAPTCHA was lower than that for ESP-PIX and Google's CAPTCHA. Perceived overall usability tended to be highest for Asirra and ReCAPTCHA. Participant comments indicated that Google's CAPTCHA was too visually distorted and ESP-PIX had commonalities between the images that weren't immediately obvious, factors perhaps contributing to their lower perceived usability scores. Overall, ReCAPTCHA is recommended for use on a desktop computer because of its high perceived usability, low completion time, and low workload.     

Users’ viewpoint of usability and user experience testing procedure - gaining methodological insights in a case of an interactive HbbTV application

We present a novel meta-methodological approach for the user experience and usability methodology procedure evaluation, shown in an example of the user experience and usability study of an interactive HbbTV application. The idea behind this research is not only to evaluate and improve the TV-WEB service but also to gain insights how the participants perceived the whole Ux evaluation procedure itself. The research questions focused mainly on the time complexity (temporal demand) and the frustration level of the TV-WEB evaluation procedure. Additionally the appropriateness of the selected content used, interface and interaction design, and the service impressions and satisfaction/payment related questions was sought. A special questionnaire based on the NASA TLX standard test is presented. The concept has been successfully implemented in several live field trials in three countries (BiH, Serbia and Montenegro). In the user experience and usability study of the service itself more than 150 participants were involved, of those 35 took part in the meta-methodology study. The feedback was quantitatively evaluated on a 7-point Likert scale, with “1” indicating the best positive feedback, “4” neutral/undecided feedback and “7” the most negative feedback. The quantitative average summary results obtained in the three evaluation studies were 1.30 for both the BiH and Serbia test study cases and the score of 1.22 for the test study case in Montenegro. These results show that a great majority of the participants found the whole evaluation procedure time-wise and frustration-wise undemanding, with appropriate content, presentation style and overall attitude towards them. By using this approach it was possible to improve the user experience and usability methodology used, producing more reliable results and providing better user experience in the final version of the product as well as providing a pleasant experience during the testing of the product.     

The System Usability Scale: Past,Present, and Future

The System Usability Scale (SUS) is the most widely used standardized questionnaire for the assessment of perceived usability. This review of the SUS covers its early history from inception in the 1980s through recent research and its future prospects. From relatively inauspicious beginnings, when its originator described it as a “quick and dirty usability scale,” it has proven to be quick but not “dirty.” It is likely that the SUS will continue to be a popular measurement of perceived usability for the foreseeable future. When researchers and practitioners need a measure of perceived usability, they should strongly consider using the SUS.     

IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use

This article describes recent research in subjective usability measurement at IBM, focused on evaluating the psychometric properties of questionnaires designed for use in scenario‐based usability evaluation. The questionnaires address evaluation at both a global overall system level and at a more detailed scenario level. The primary goals of this article are to (a) discuss the psychometric characteristics of IBM questionnaires that measure user satisfaction with computer system usability, and (b) provide the questionnaires, with administration and scoring instructions. For scenario‐level measurement, the 3‐item After‐Scenario Questionnaire (ASQ) has excellent internal consistency, with coefficient alphas across a set of scenarios ranging from .90 to .96. For more global assessment, the Post‐Study System Usability Questionnaire (PSSUQ) also has excellent internal consistency, with an overall coefficient alpha of .97. Preliminary principal factor analysis of 48 PSSUQ questionnaires suggested the presence of three factors named, after varimax rotation, System Usefulness, Information Quality, and Interface Quality, with corresponding coefficient alphas of .96, .91, and .91. Evaluation of 377 PSSUQ questionnaires (modified to allow mailing to respondents in their offices and referred to as the Computer System Usability Questionnaire, or CSUQ) confirmed the structure of the preliminary principal factor analysis. Consequently, usability practitioners can use these questionnaires to help them measure users’ satisfaction with the usability of computer systems in the context of scenario‐based usability studies.     

The Inference of Perceived Usability From Beauty

A review of 15 papers reporting 25 independent correlations of perceived beauty with perceived usability showed a remarkably high variability in the reported coefficients. This may be due to methodological inconsistencies. For example, products are often not selected systematically, and statistical tests are rarely performed to test the generality of findings across products. In addition, studies often restrict themselves to simply reporting correlations without further specification of underlying judgmental processes.

The present study's main objective is to re-examine the relation between beauty and usability, that is, the implication that “what is beautiful is usable.” To rectify previous methodological shortcomings, both products and participants were sampled in the same way and the data aggregated both by averaging over participants to assess the covariance across ratings of products and by averaging over products to assess the covariance across participants. In addition, we adopted an inference perspective to qualify underlying processes to examine the possibility that, under the circumstances pertaining in most studies of this kind where participants have limited experience of using a website or product, the relationship between beauty and usability is mediated by goodness.

A mediator analysis of the relationship between beauty, the overall evaluation (i.e., “goodness”) and pragmatic quality (as operationalization of usability) suggests that the relationship between beauty and usability has been overplayed as the correlation between pragmatic quality and beauty is wholly mediated by goodness. This pattern of relationships was consistent across four different data sets and different ways of data aggregation. Finally, suggestions are made regarding methodologies that could be used in future studies that build on these results.     

Usability Ratings for Everyday Products Measured With the System Usability Scale

This article characterizes the usability of 14 common, everyday products using the System Usability Scale (SUS). More than 1,000 users were queried about the usability of these products using an online survey methodology. The study employed two novel applications of the SUS. First, participants were not asked to perform specific tasks on these products before rating their usability but were rather asked to assess usability based on their overall integrated experience with a given product. Second, some of the evaluated products were assessed as a class of products (e.g., “microwaves”) rather than a specific make and model, as is typically done. The results show clear distinctions among different products and will provide practitioners and researchers with important known benchmarks as they seek to characterize and describe results from their own usability studies.     

Future technology on the flight deck: assessing the use of touchscreens in vibration environments

Abstract

Use of touchscreens in the flight deck has been steadily increasing, however, their usability may be severely impacted when turbulent conditions arise. Most previous research focusses on using touchscreens in static conditions; therefore, this study assessed touchscreen use whilst undergoing turbulent representative motion, generated using a 6-axis motion simulator. Touchscreens were tested in centre, side and overhead positions, to investigate how turbulence affected: (1) error rate, movement times and accuracy, (2) arm fatigue and discomfort. Two touchscreen technologies were compared: a 15” infra-red and a 17.3” projected capacitive touchscreen with force sensing capability. The potential of the force sensing capability to minimise unintentional interactions was also investigated. Twenty-six participants undertook multi-direction tapping (ISO 9241; ISO 2010 ISO (International Organization for Standardization). 2010. “ISO 9241 Ergonomics of Human System Interaction – Part 210: Human-centred Design for Interactive Systems.” Tech. Rep. Switzerland. [Google Scholar]) and gesture tasks, under four vibration conditions (control, light chop, light turbulence and moderate turbulence). Error rate, movement time and workload increased and usability decreased significantly, with screen position and increasing turbulence level.

Practitioner Summary: This study evaluated the use of infra-red and projected capacitive touchscreen technologies using multi-directional tapping and gesture tasks, whilst being subjected to different levels of turbulence representative motion. Performance degraded significantly with increasing turbulence level and touchscreen location. This has implications for future flight deck design.     

Templates for Cross-Cultural and Culturally Specific Usability Testing: Results From Field Studies and Ethnographic Interviewing in Three Countries

The cultural diversity of users of technology challenges our methods for usability testing. This article suggests templates for cross-culturally and culturally specific usability testing, based on studies of usability testing in companies in Mumbai, Beijing, and Copenhagen. Study 1 was a cross-cultural field study of think-aloud testing done by usability vendor companies in the three countries. The result was a grounded theory of cultural variations in the production of a usability problem list. Study 2 was a follow-up, ethnographic interview study of how the companies typically perform usability tests. The result was the construction of templates for usability testing. The culturally specific templates were in Mumbai “user-centered evaluation,” Copenhagen “client-centered evaluation,” and Beijing “evaluator-centered evaluation.” The findings are compared with related research, and the implications are pointed out. The templates can be seen as a simple and practical way to plan, compare, and improve the way usability testing is carried out in multiple, different cultures and countries.     

Exploring Relationships Between Eye Tracking and Traditional Usability Testing Data

This study explored the relationships between eye tracking and traditional usability testing data in the context of analyzing the usability of Algebra Nation?, an online system for learning mathematics used by hundreds of thousands of students. Thirty-five undergraduate students (20 females) completed seven usability tasks in the Algebra Nation? online learning environment. The participants were asked to log in, select an instructor for the instructional video, post a question on the collaborative wall, search for an explanation of a mathematics concept on the wall, find information relating to Karma Points (an incentive for engagement and learning), and watch two instructional videos of varied content difficulty. Participants’ eye movements (fixations and saccades) were simultaneously recorded by an eye tracker. Usability testing software was used to capture all participants’ interactions with the system, task completion time, and task difficulty ratings. Upon finishing the usability tasks, participants completed the System Usability Scale. Important relationships were identified between the eye movement metrics and traditional usability testing metrics such as task difficulty rating and completion time. Eye tracking data were investigated quantitatively using aggregated fixation maps, and qualitative examination was performed on video replay of participants’ fixation behavior. Augmenting the traditional usability testing methods, eye movement analysis provided additional insights regarding revisions to the interface elements associated with these usability tasks.     

Using simulation to aid decision making in managing the usability evaluation process

ContextThis paper is developed in the context of Usability Engineering. More specifically, it focuses on the use of modelling and simulation to help decision-making in the scope of usability evaluation.ObjectiveThe main goal of this paper is to present UESim: a System Dynamics simulation model to help decision-making in the make-up of the usability evaluation team during the process of usability evaluation.MethodTo develop this research we followed four main research phases: (a) study identification, (b) study development, (c) running and observation and finally, (d) reflexion. In relation with these phases the paper describes the literature revision, the model building and validation, the model simulation and its results and finally the reflexion on it.ResultsWe developed and validated a model to simulate the usability evaluation process. Through three different simulations we analysed the effects of different compositions of the evaluation team on the outcome of the evaluation. The simulation results show the utility of the model in the decision making of the usability evaluation process by changing the number and expertise of evaluators employed.ConclusionOne of the main advantages of using such a simulation model is that it allows developers to observe the evolution of the key indicators of the evaluation process over time. UESim represents a customisable tool to help decision-making in the management of the usability evaluation process, since it makes it possible to analyse how the key process indicators are affected by the main management options of the Usability Evaluation Process.     

Investigating users’ perspectives on e-learning: An integration of TAM and IS success model

The purpose of this paper is to examine an integrated model of TAM and D&M to explore the effects of quality features, perceived ease of use, perceived usefulness on users’ intentions and satisfaction, alongside the mediating effect of usability towards use of e-learning in Iran. Based on the e-learning user data collected through a survey, structural equations modeling (SEM) and path analysis were employed to test the research model. The results revealed that “intention” and “user satisfaction” both had positive effects on actual use of e-learning. “System quality” and “information quality” were found to be the primary factors driving users’ intentions and satisfaction towards use of e-learning. At last, “perceived usefulness” mediated the relationship between ease of use and users’ intentions. The sample consisted of e-learning users of four public universities in Iran. Past studies have seldom examined an integrated model in the context of e-learning in developing countries. Moreover, this paper tries to provide a literature review of recent published studies in the field of e-learning.     

Commentary: Usability in Theory and Practice

This commentary discusses Noam Tractinsky’s article “The Usability Construct: A Dead End?” (Human–Computer Interaction, 2017). That article debates the conceptual qualities of “usability” as a key concept in human–computer interaction. It provides two main conclusions: (a) the usefulness of the concept of “usability” to human–computer interaction theories has been limited and (b) a notion of “construct” is useful for identifying weaknesses of the concept of “usability” and thereby provides reasons for its limited usefulness. The article is interesting and provides an in-depth analysis of the concept of usability. Unfortunately, the use of this analysis is not convincing. Instead, redefinition of “usability” should be based on a constructive approach.     

A study on how usability flaws in GUI design increase mouse movements and consequently may affect users' health

The objective of this study is to discuss how software usability flaws may cause a significant increase in mouse movements and, as a potential side effect, may even affect users' health. During the literature review, this article examines the potential relationship between mouse movement and musculoskeletal disorders of the upper extremity, based on studies from medical sciences. Subsequently, in the main part of the present study, three software products were selected that had at least one usability flaw related to extra mouse movement (the selection of the software was made out of over 20 software programs that had been evaluated in our Software Quality Assessment Laboratory). For these products, all additional mouse movements were measured, involving actual users in various settings and computer configurations. The findings showed that even a single usability flaw may increase mouse movement to a magnitude of between 3.6 and 4.7 m/h. The article concludes that the role of software engineering is to focus on software usability as well, taking into account that a user friendly graphical user interface (GUI) which is able to eliminate unnecessary mouse movement may also eventually contribute to the reduction of fatigue and discomfort, caused by musculoskeletal disorders.     

Scoping Analytical Usability Evaluation Methods: A Case Study

ABSTRACT

Analytical usability evaluation methods (UEMs) can complement empirical evaluation of systems: for example, they can often be used earlier in design and can provide accounts of why users might experience difficulties, as well as what those difficulties are. However, their properties and value are only partially understood. One way to improve our understanding is by detailed comparisons using a single interface or system as a target for evaluation, but we need to look deeper than simple problem counts: we need to consider what kinds of accounts each UEM offers, and why. Here, we report on a detailed comparison of eight analytical UEMs. These eight methods were applied to a robotic arm interface, and the findings were systematically compared against video data of the arm in use. The usability issues that were identified could be grouped into five categories: system design, user misconceptions, conceptual fit between user and system, physical issues, and contextual ones. Other possible categories such as user experience did not emerge in this particular study. With the exception of Heuristic Evaluation, which supported a range of insights, each analytical method was found to focus attention on just one or two categories of issues. Two of the three “home-grown” methods (Evaluating Multimodal Usability and Concept-based Analysis of Surface and Structural Misfits) were found to occupy particular niches in the space, whereas the third (Programmable User Modeling) did not. This approach has identified commonalities and contrasts between methods and provided accounts of why a particular method yielded the insights it did. Rather than considering measures such as problem count or thoroughness, this approach has yielded insights into the scope of each method.     

A Measurement and Structural Model for Usability Evaluation of Shared Workspace Groupware

This study presents the development process of a set of questionnaire items to establish a measurement model for the usability of shared workspace groupware systems, which is suggested as a usability scale called SWUS, the Shared Workspace Usability Scale. Manifest variables and latent variables are based on the various dimensions of teamwork collated through the literature. A structural model was built on the measurement model. Models were evaluated through PLS-SEM methods. Data acquired on candidate questionnaire items from 398 international respondents who are users of five different online collaborative word processors was used for the model analysis. Of 37 candidate manifest variables, 22 were retained, which were measuring seven latent constructs: “3C Mechanisms,” “Grounding,” “Team Integration,” “Communication,” “Shared Access,” “Awareness,” and “Usability.” The data provided empirical evidence for the structural model based on these latent variables. The responses of the participants were not sensitive to differences between users in terms of gender and native language but showed sensitivity to age, experience with the evaluated software, and different shared workspace groupware evaluated in the study. Our structural model attempts to integrate several frameworks and models of usability for CSCW environments and provides empirical evidence for its reliability and validity based on subjective responses from users of shared workspace groupware.     

Usability evaluation of a virtual reality smartphone app for a living museum

This paper elaborates the empirical evidence of a usability evaluation of a VR and non-VR virtual tour application for a living museum. The System Usability Scale (SUS) was used in between participants experiments (Group 1: non-VR version and Group 2: VR version) with 40 participants. The results show that the mean scores of all components for the VR version are higher compared to the non-VR version, overall SUS score (72.10 vs 68.10), usability score (75.50 vs 71.70), and learnability (58.40 vs 57.00). Further analysis using a two-tailed independent t test showed no difference between the non-VR and VR versions. Additionally, no significant difference was observed between the groups in the context of gender, nationality, and prior experience (other VR tour applications) for overall SUS score, usability score, and learnability score. Α two-tailed independent t test indicated no significant difference in the usability score between participants with VR experience and no VR experience. However, a significant difference was found between participants with VR experience and no VR experience for both SUS score (t(38) = 2.17, p = 0.037) and learnability score (t(38) = 2.40, p = 0.021). The independent t test results indicated a significant difference between participant with and without previous visits to SCV for the usability score (t(38) = −2.31, p = 0.027), while there was no significant differences observed in other components. It can be concluded that both versions passed based on the SUS score. However, the sub-scale usability and learnability scores indicated some usability issue.

     

Sociability and usability in online communities: Determining and measuring success

Little attention has focused so far on evaluating the success of online communities. This paper begins to identify some key determinants of sociability and usability that help to determine their success. Determinants of sociability include obvious measures such as the number of participants in a community, the number of messages per unit of time, members' satisfaction, and some less obvious measures such as amount of reciprocity, the number of on-topic messages, trustworthiness and several others. Measures of usability include numbers of errors, productivity, user satisfaction and others. The list is not exhaustive but it is intended to provide a starting point for research on this important topic that will lead to develop of metrics. To avoid creating false impressions it is advisable to use several measures and to triangulate with qualitative data, particularly from ethnographic studies.