Development and Psychometric Evaluation of the Emotional Metric Outcomes (EMO) Questionnaire

This article describes the development and psychometric evaluation of the Emotional Metric Outcomes (EMO) questionnaire—a new questionnaire designed to assess the emotional outcomes of interaction, especially the interaction of customers with service-provider personnel or software. The EMO is a concise multifactor standardized questionnaire that provides an assessment of transaction-driven personal and relationship emotional outcomes, both positive and negative. The primary purpose of the EMO is to move beyond traditional assessment of satisfaction to achieve a more effective measurement of customers’ emotional responses to products and processes. Psychometric evaluation showed that the EMO and its component scales had high reliability and concurrent validity with loyalty and overall experience metrics in a variety of measurement contexts. Concurrent measurement with the System Usability Scale (SUS) indicated that reported significant correlation of the SUS with likelihood-to-recommend ratings are probably primarily due to emotional rather than utilitarian aspects of the SUS.     

Towards a Standard Arabic System Usability Scale: Psychometric Evaluation using Communication Disorder App

This research aims to customize the System Usability Scale (SUS) so it can be used as a standard usability measure on native Arabic-language speakers. A process of rigor translation followed by psychometric evaluation has been administered through a questionnaire in the Arabic-language that reflects a reasonable level of reliability, validity and sensitivity when compared to the original English-language SUS questionnaire. The final result of the Arabic-System Usability Scale (A-SUS) is then applied on students majoring in Communication Disorders Sciences (Kuwait University) to measure the usability of a mobile application. A-SUS encapsulates the essence of the original SUS and provides an adequate tool for professionals to use with Arabic native speaking users to evaluate systems usability.     

The System Usability Scale: Past,Present, and Future

The System Usability Scale (SUS) is the most widely used standardized questionnaire for the assessment of perceived usability. This review of the SUS covers its early history from inception in the 1980s through recent research and its future prospects. From relatively inauspicious beginnings, when its originator described it as a “quick and dirty usability scale,” it has proven to be quick but not “dirty.” It is likely that the SUS will continue to be a popular measurement of perceived usability for the foreseeable future. When researchers and practitioners need a measure of perceived usability, they should strongly consider using the SUS.     

Psychometric Evaluation of the T-CSUQ: The Turkish Version of the Computer System Usability Questionnaire

This article describes the development of a standardized computer system usability questionnaire for use with speakers of the Turkish language, the Turkish Computer System Usability Questionnaire, Short Version (T-CSUQ-SV). This new questionnaire, based on the English-language CSUQ, underwent careful translation and transformation through comprehensive psychometric evaluation. The results of the psychometric evaluation revealed an acceptable level of reliability, appropriate construct validity, and sensitivity to manipulation, indicating that Turkish usability practitioners should be able to use the T-CSUQ-SV with confidence when conducting user research.     

A Slovene Translation of the System Usability Scale: The SUS-SI

The System Usability Scale (SUS) is a widely adopted and studied questionnaire for usability evaluation. It is technology independent and has been used to evaluate the perceived usability of a broad range of products, including hardware, software, and websites. In this article we present a Slovene translation of the SUS (the SUS-SI) along with the procedure used in its translation and psychometric evaluation. The results indicated that the SUS-SI has properties similar to the English version. Slovene usability practitioners should be able to use the SUS-SI with confidence when conducting user research.     

Measuring Perceived Usability: The SUS,UMUX-LITE,and AltUsability

The purpose of this research was to investigate various measurements of perceived usability, in particular, to assess (a) whether a regression formula developed previously to bring Usability Metric for User Experience LITE (UMUX-LITE) scores into correspondence with System Usability Scale (SUS) scores would continue to do so accurately with an independent set of data; (b) whether additional items covering concepts such as findability, reliability, responsiveness, perceived use by others, effectiveness, and visual appeal would be redundant with the construct of perceived usability or would align with other potential constructs; and (c) the dimensionality of the SUS as a function of self-reported frequency of use and expertise. Given the broad use of and emerging interpretative norms for the SUS, it was encouraging that the regression equation for the UMUX-LITE worked well with this independent set of data, although there is still a need to investigate its efficacy with a broader set of products and methods. Results from a series of principal components analyses indicated that most of the additional concepts, such as findability, familiarity, efficiency, control, and visual appeal covered the same statistical ground as the other more standard metrics for perceived usability. Two of the other items (Reliable and Responsive) made up a reliable construct named System Quality. None of the structural analyses of the SUS as a function of frequency of use or self-reported expertise produced the expected components, indicating the need for additional research in this area and a need to be cautious when using the Usable and Learnable components described in previous research.     

Measuring Perceived Usability: SUS,UMUX, and CSUQ Ratings for Four Everyday Products

This research continued previous investigation of the relationships among measures of perceived usability: the System Usability Scale (SUS), three metrics derived from the Usability Metric for User Experience (UMUX), and the Computer System Usability Questionnaire (CSUQ), this time with ratings of four everyday products (Excel, Word, Amazon, and Gmail). SUS ratings of these products were generally consistent with previous reports. Significant differences in SUS means across studies could be due to differences in frequency of use, with implications for using these data as usability benchmarks. Correspondence among the various measures of perceived usability was also consistent with previous research. Considering frequency of use, mean differences ranged from -2.0 to 1.8 (average shift in Sauro-Lewis grade range from -0.6 to 0.8). When SUS scores were above average, the range restriction of the UMUX-LITEr led to relatively large discrepancies with SUS, suggesting it might not always be better than unadjusted UMUXLITE.     

The Impact of Personality on the Subjective Assessment of Usability

The system usability scale (SUS) has been widely employed in both the field and the laboratory as a valid and reliable measure of system usability. Although its psychometric properties are relatively well understood, the impact that differences in users’ personality traits have on their perceived usability of products, services, and systems has not been deeply explored—even though people’s scores on personality traits have been shown to be reliable and predict a staggering array of societally important outcomes in work, school, and life domains. In this study, 268 users assessed the usability of 20 different products retrospectively with the SUS. Five broad personality traits were then measured using the Mini-IPIP scale. Results indicated that measured personality traits do correlate with the rated usability of products, where measures of Openness to Experience and Agreeableness have the strongest positive correlations with subjective usability assessment. Implications for practitioners, designers, and researchers are discussed.     

Psychometric Evaluation of the PSSUQ Using Data from Five Years of Usability Studies

Factor analysis of Post Study System Usability Questionnaire (PSSUQ) data from 5 years of usability studies (with a heavy emphasis on speech dictation systems) indicated a 3-factor structure consistent with that initially described 10 years ago: factors for System Usefulness, Information Quality, and Interface Quality. Estimated reliabilities (ranging from .83-.96) were also consistent with earlier estimates. Analyses of variance indicated that variables such as the study, developer, stage of development, type of product, and type of evaluation significantly affected PSSUQ scores. Other variables, such as gender and completeness of responses to the questionnaire, did not. Norms derived from this data correlated strongly with norms derived from the original PSSUQ data. The similarity of psychometric properties between the original and this PSSUQ data, despite the passage of time and differences in the types of systems studied, provide evidence of significant generalizability for the questionnaire, supporting its use by practitioners for measuring participant satisfaction with the usability of tested systems.     

The IsoMetrics usability inventory: an operationalization of ISO 9241-10 supporting summative and formative evaluation of software systems

Aiming at a user-oriented approach in software evaluation on the basis of ISO 9241 Part 10, we present a questionnaire (IsoMetrics) which collects usability data for summative and formative evaluation, and document its construction. The summative version of IsoMetrics shows a high reliability of its subscales and gathers valid information about differences in the usability of different software systems. Moreover, we show that the formative version of IsoMetrics is a powerful tool for supporting the identification of software weaknesses. Finally, we propose a procedure to categorize and prioritize weak points, which subsequently can be used as basic input to usability reviews.     

Measuring Perceived Usability: The CSUQ,SUS, and UMUX

The primary purpose of this research was to investigate the relationship between two widely used questionnaires designed to measure perceived usability: the Computer System Usability Questionnaire (CSUQ) and the System Usability Scale (SUS). The correlation between concurrently collected CSUQ and SUS scores was 0.76 (over 50% shared variance). After converting CSUQ scores to a 0–100-point scale (to match the range of the SUS scores), there was a small but statistically significant difference between CSUQ and SUS means. Although this difference (just under 2 scale points out of a possible 100) was statistically significant, it did not appear to be practically significant. Although usability practitioners should be cautious pending additional independent replication, it appears that CSUQ scores, after conversion to a 0–100-point scale, can be interpreted with the Sauro–Lewis curved grading scale. As a secondary research goal, investigation of variations of the Usability Metric for User Experience (UMUX) replicated previous findings that the regression-adjusted version of the UMUX-LITE (UMUX-LITEr) had the closest correspondence with concurrently collected SUS scores. Thus, even though these three standardized questionnaires were independently developed and have different item content and formats, they largely appear to be measuring the same thing, presumably, perceived usability.     

Understanding of website usability: Specifying and measuring constructs and their relationships

Developing a usable website is pivotal for e-business success. Researchers have devoted effort to develop metrics, guidelines and theories of website usability, yet there still is a lack of consensus on the multifaceted dimensions of website usability and lack of investigation of the nomological networks among website usability constructs. This study first investigated the common dimensions of website usability by integrating the findings of previous studies and a focus group study with website usability experts. Instruments to measure the constructs were developed and empirically validated. Then nomological networks between website usability constructs and between those constructs and online purchase intention and purchase were examined. Three field studies including two questionnaire surveys and a causal mapping analysis were conducted. The research identified ten website usability constructs with strong psychometric properties. A number of nomological networks between usability constructs were discovered, contributing to identification of sources of variances of purchase intention and purchase behavior. Findings of this study are expected to provide useful insights for practitioners to develop more usable websites and for researchers to better assess the effect of website usability on online customer behavior.     

The Impact of Geographic Location on the Subjective Assessment of System Usability

Marketing researchers use geography to identify specific user groups for studies to more effectively describe their potential customer base. Since usability professionals often recruit users employing similar selection criteria as their marketing peers, the use of geographic information might also be relevant when selecting usability test participants. In total, 3,168 participants from across the United States rated the usability of different hardware, software, and web-based products using the System Usability Scale (SUS). SUS scores were compared across geographic divisions to determine if usability assessments differ by location. SUS scores were also compared across rural and urban areas to determine if usability assessment scores change with population density. There was a lack of evidence to support significant differences in usability scores across both US geographic areas and zones of population density. The findings suggest that people make similar system usability assessments regardless of the area of the United States in which they live.     

An evaluation of the Circadian Type Questionnaire

Abstract

This study evaluates the psychometric properties and assesses the test-retest reliability and longer-term stability of scores on the Circadian Type Questionnaire (CTQ) of Folkard et al. (1979) in a sample of 445 students. The scales were found to lack internal consistency and the factor structure originally proposed was not replicated. Cultural variation in CTQ scores was suggested. Test-retest reliability, over three months in a subgroup of 36, was poor for the V and M scales. Longer-term stability, over nine months in a subgroup of 36 who were exposed to shiftwork, was better but significant decreases were found in rigidity of sleeping habits (R_s) and morningness (M) scores. A factor analysis suggested that the scales should be constructed differently; however, the properties of such scales were still not optimal. The CTQ has psychometric flaws and needs to be improved before it could be expected to reliably function as a predictive test of adaptation to shiftwork.     

Assessing User Satisfaction in the Era of User Experience: Comparison of the SUS,UMUX, and UMUX-LITE as a Function of Product Experience

Nowadays, practitioners extensively apply quick and reliable scales of user satisfaction as part of their user experience analyses to obtain well-founded measures of user satisfaction within time and budget constraints. However, in the human–computer interaction literature the relationship between the outcomes of standardized satisfaction scales and the amount of product usage has been only marginally explored. The few studies that have investigated this relationship have typically shown that users who have interacted more with a product have higher satisfaction. The purpose of this article was to systematically analyze the variation in outcomes of three standardized user satisfaction scales (SUS, UMUX, UMUX-LITE) when completed by users who had spent different amounts of time with a website. In two studies, the amount of interaction was manipulated to assess its effect on user satisfaction. Measurements of the three scales were strongly correlated and their outcomes were significantly affected by the amount of interaction time. Notably, the SUS acted as a unidimensional scale when administered to people who had less product experience but was bidimensional when administered to users with more experience. Previous findings of similar magnitudes for the SUS and UMUX-LITE (after adjustment) were replicated but did not show the previously reported similarities of magnitude for the SUS and the UMUX. Results strongly encourage further research to analyze the relationships of the three scales with levels of product exposure. Recommendations for practitioners and researchers in the use of the questionnaires are also provided.     

Usability Ratings for Everyday Products Measured With the System Usability Scale

This article characterizes the usability of 14 common, everyday products using the System Usability Scale (SUS). More than 1,000 users were queried about the usability of these products using an online survey methodology. The study employed two novel applications of the SUS. First, participants were not asked to perform specific tasks on these products before rating their usability but were rather asked to assess usability based on their overall integrated experience with a given product. Second, some of the evaluated products were assessed as a class of products (e.g., “microwaves”) rather than a specific make and model, as is typically done. The results show clear distinctions among different products and will provide practitioners and researchers with important known benchmarks as they seek to characterize and describe results from their own usability studies.     

Measuring user-satisfaction with electronic consumer products: The Consumer Products Questionnaire

This paper details the development of the Consumer Products Questionnaire (CPQ), a psychometric questionnaire created to measure user-satisfaction with electronic consumer products (ECPs). The five-factor theoretical model of satisfaction proposed by Porteous et al. (1995) was selected as a starting point for further empirical validation. An iterative psychometric process was used where responses to three versions of the CPQ were gathered from three independent user samples. Factor analysis and item analysis were used to produce the final thirty-item instrument. The original five-factor model was reduced to three factors, namely, Efficiency, Helpfulness, and Transparency. Cronbach's alpha for the Global scale and the three subscales are all above 0.90. In addition, preliminary validation studies indicate that the questionnaire can distinguish between products that differ in terms of usability.     

Usability evaluation of a virtual reality smartphone app for a living museum

This paper elaborates the empirical evidence of a usability evaluation of a VR and non-VR virtual tour application for a living museum. The System Usability Scale (SUS) was used in between participants experiments (Group 1: non-VR version and Group 2: VR version) with 40 participants. The results show that the mean scores of all components for the VR version are higher compared to the non-VR version, overall SUS score (72.10 vs 68.10), usability score (75.50 vs 71.70), and learnability (58.40 vs 57.00). Further analysis using a two-tailed independent t test showed no difference between the non-VR and VR versions. Additionally, no significant difference was observed between the groups in the context of gender, nationality, and prior experience (other VR tour applications) for overall SUS score, usability score, and learnability score. Α two-tailed independent t test indicated no significant difference in the usability score between participants with VR experience and no VR experience. However, a significant difference was found between participants with VR experience and no VR experience for both SUS score (t(38) = 2.17, p = 0.037) and learnability score (t(38) = 2.40, p = 0.021). The independent t test results indicated a significant difference between participant with and without previous visits to SCV for the usability score (t(38) = −2.31, p = 0.027), while there was no significant differences observed in other components. It can be concluded that both versions passed based on the SUS score. However, the sub-scale usability and learnability scores indicated some usability issue.

     

The Relationship Between System Effectiveness and Subjective Usability Scores Using the System Usability Scale

This article examines the relationship between users’ subjective usability assessments, as measured using the System Usability Scale (SUS), and the ISO metric of effectiveness, using task success as the measure. The article reports the results of two studies designed to explore the relationship between SUS scores and user success rates for a variety of interfaces. The first study was a field study, where stereotypical usability assessments on a variety of products and services were performed. The second study was a well-controlled laboratory study where the level of success that users were able to achieve was controlled. For both studies, the relationship between SUS scores and their attendant performance were examined at both the individual level and the average system level. Although the correlations are far from perfect, there are reliable and reasonably strong positive correlations between subjective usability measures and task success rates, for both the laboratory and field studies at both the individual and system level.     

Measuring the Usability of Mobile Applications for Phones and Tablets

The use of applications on mobile devices has reached historic levels. Using the System Usability Scale (SUS), data were collected on the usability of applications used on two kinds of mobile platforms—phones and tablets—across two general classes of operating systems, iOS and Android. Over 4 experiments, 3,575 users rated the usability of 10 applications that had been selected based on their popularity, as well as 5 additional applications that users had identified as using frequently. The average SUS rating for the top 10 apps across all platforms was 77.7, with a nearly 20-point spread (67.7–87.4) between the highest and lowest rated apps. Overall, applications on phone platforms were judged to be more usable than applications on the tablet platforms. Practitioners can use the information in this article to make better design decisions and benchmark their progress against a known universe of apps for their specific mobile platform.