Remote Usability Evaluation: Overview and Case Studies

Traditional laboratory-based usability evaluations have many costs associated with them: financial costs, temporary loss in participant productivity, travel issues. For this reason, many human-computer interaction practitioners cannot conduct as many test iterations as they would like or may fail to secure representative samples, especially as user groups become more global. In recent years, we have begun to use less expensive remote evaluation techniques to complement traditional methods. This article describes some collaborative software that allows remote evaluations, along with the necessary hardware, to reproduce, as closely as possible, traditional evaluation techniques. It also describes the types of activities that lend themselves to this sort of approach and provides tips for conducting these sorts of sessions. Finally, this article provides data that suggest remote techniques are not only effective at improving interface designs, but also produce results comparable to traditional approaches for typical measures.     

The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods

Computer professionals have a need for robust, easy-to-use usability evaluation methods (UEMs) to help them systematically improve the usability of computer artifacts. However, cognitive walkthrough (CW), heuristic evaluation (HE), and thinking- aloud study (TA)-3 of the most widely used UEMs-suffer from a substantial evaluator effect in that multiple evaluators evaluating the same interface with the same UEM detect markedly different sets of problems. A review of 11 studies of these 3 UEMs reveals that the evaluator effect exists for both novice and experienced evaluators, for both cosmetic and severe problems, for both problem detection and severity assessment, and for evaluations of both simple and complex systems. The average agreement between any 2 evaluators who have evaluated the same system using the same UEM ranges from 5% to 65%, and no 1 of the 3 UEMs is consistently better than the others. Although evaluator effects of this magnitude may not be surprising for a UEM as informal as HE, it is certainly notable that a substantial evaluator effect persists for evaluators who apply the strict procedure of CW or observe users thinking out loud. Hence, it is highly questionable to use a TA with 1 evaluator as an authoritative statement about what problems an interface contains. Generally, the application of the UEMs is characterized by (a) vague goal analyses leading to variability in the task scenarios, (b) vague evaluation procedures leading to anchoring, or (c) vague problem criteria leading to anything being accepted as a usability problem, or all of these. The simplest way of coping with the evaluator effect, which cannot be completely eliminated, is to involve multiple evaluators in usability evaluations.     

Criteria For Evaluating Usability Evaluation Methods

The current variety of alternative approaches to usability evaluation methods (UEMs) designed to assess and improve usability in software systems is offset by a general lack of understanding of the capabilities and limitations of each. Practitioners need to know which methods are more effective and in what ways and for what purposes. However, UEMs cannot be evaluated and compared reliably because of the lack of standard criteria for comparison. In this article, we present a practical discussion of factors, comparison criteria, and UEM performance measures useful in studies comparing UEMs. In demonstrating the importance of developing appropriate UEM evaluation criteria, we offer operational definitions and possible measures of UEM performance. We highlight specific challenges that researchers and practitioners face in comparing UEMs and provide a point of departure for further discussion and refinement of the principles and techniques used to approach UEM evaluation and comparison.     

CBID系统的可用性评估

本文采用用户测试法和用户调查法对CBID(Case Based Industrial Design)系统进行评估分析。通过对可用性评估结果的分析,进一步为CBID系统与ICAID系统可用性对比研究提供根据。     

A Case (Study) For Usability in Secure Email Communication

The author's study on the usability and practical relevance of standard security mechanisms for email communication.     

A Usability Checklist for the Usability Evaluation of Mobile Phone User Interface

In the last decade, the research of the usability of mobile phones has been a newly evolving area with few established methodologies and realistic practices that ensure capturing usability in evaluation. Thus, there exists growing demand to explore appropriate evaluation methodologies that evaluate the usability of mobile phones quickly as well as comprehensively. This study aims to develop a task-based usability checklist based on heuristic evaluations in views of mobile phone user interface (UI) practitioners. A hierarchical structure of UI design elements and usability principles related to mobile phones were developed and then utilized to develop the checklist. To demonstrate the practical effectiveness of the proposed checklist, comparative experiments were conducted on the usability checklist and usability testing. The majority of usability problems found by usability testing and additional problems were discovered by the proposed checklist. It is expected that the usability checklist proposed in this study could be used quickly and efficiently by usability practitioners to evaluate the mobile phone UI in the middle of the mobile phone development process.     

Increasing the Usability of Online Information for Older Users: A Case Study in Participatory Design

This article describes one of the first documented participatory design (PD) efforts specifically aimed at older users. The goal of the project was to make an existing World Wide Web (WWW) site more user-friendly for older users, specifically in terms of display format issues. A PD team was assembled from a group of community-dwelling older adults and developers from a university research lab. After the developers established the trust and confidence of the participants and developed a conceptual user model (based on a survey and previous literature), the PD team evaluated the original design. Prototypes were iteratively developed and tested by the PD team to improve problems found in the original design. Specific design improvements and general design guidelines for older WWW users are discussed.     

一种基于可用度的Web服务可用性测量模型

Web服务通常运行于一个动态和开放的网络环境中,其底层平台与面向的用户千差万别,这使得Web服务的可用性经常不能得以保证。如何对Web服务可用性进行合理评估并提高服务质量是一个具有挑战性的研究课题。从服务请求者的角度出发,设计一种基于可用度的Web服务可用性测试模型,通过采集原始指标数据对服务质量进行可用度量化评估,为服务请求者选择更佳的Web服务提供参考依据。     

DEEP: Design-Oriented Evaluation of Perceived Usability

Existing perceived usability questionnaires detect the appearance of usability issues rather than the underlying design generating those issues. This limits the capability of existing instruments to directly inform design recommendations. To address this problem, a usability questionnaire structured around the analytical composition of the design was created and validated. A four-stage process was followed. First, 3 usability experts refined 54 questions from highly cited usability questionnaires and structured them around 6 design dimensions. Second, 12 raters scored the questions by their relevance to assess usability. Third, questions and dimensions were then improved through exploratory and confirmatory factor analysis (N?=?196) and, fourth, further enhanced through confirmatory factor analysis (N?=?362). The result is DEEP, a 19-question usability questionnaire based on 5 main design dimensions (content, information architecture, navigation, layout, and visual guidance). DEEP can be used to capture detailed usability feedback that more directly relates to specific aspects of design requirements.     

IT产品的可用性测试与评估

可用性测试与评估是可用性工程的重要组成部分。该文首先提出可用性质量的指标体系,它为衡量和改进产品的可用性质量提供了参照系和必要的基础,然后重点介绍几个用户评估与专家评估方法,最后给出一个可用性用户评估的实例。     

Evaluation of 3-D Graphics Software: A Case Study

Contracting for a commercially available 3-D graphics software system has advantages over developing such a system in-house, but a detailed evaluation process is a must.     

Evaluation of Environments for Portals Development: A Case Study

With the growing application of the Internet, business portal software is becoming increasingly complex given the wide variety of technologies it must integrate. It is therefore most important to have development environments that enable this type of software to be built efficiently. This justifies the need to evaluate the quality of business portal development environments.

The objective of this article is to propose the definition of a quality model for business portal development environments, based on ISO/IEC 9126, with the appropriate metrics for estimating quality. the estimation model was evaluated through a case study using a commercial business portal development environment. the rational unified process (RUP) methodology was chosen to build a business portal prototype.     

Ingredients and Meals Rather Than Recipes: A Proposal for Research That Does Not Treat Usability Evaluation Methods as Indivisible Wholes

To better support usability practice, most usability research focuses on evaluation methods. New ideas in usability research are mostly proposed as new evaluation methods. Many publications describe experiments that compare methods. Comparisons may indicate that some methods have important deficiencies, and thus often advise usability practitioners to prefer a specific method in a particular situation. An expectation persists in human–computer interaction (HCI) that results about evaluation methods should be the standard “unit of contribution” rather than favoring larger units (e.g., usability work as a whole) or smaller ones (e.g., the impact of specific aspects of a method). This article argues that these foci on comparisons and method innovations ignore the reality that usability evaluation methods are loose incomplete collections of resources, which successful practitioners configure, adapt, and complement to match specific project circumstances. Through a review of existing research on methods and resources, resources associated with specific evaluation methods, and ones that can complement existing methods, or be used separately, are identified. Next, a generic classification scheme for evaluation resources is developed, and the scheme is extended with project specific resources that impact the effective use of methods. With these reviews and analyses in place, implications for research, teaching, and practice are derived. Throughout, the article draws on culinary analogies. A recipe is nothing without its ingredients, and just as the quality of what is cooked reflects the quality of its ingredients, so too does the quality of usability work reflect the quality of resources as configured and combined. A method, like a recipe, is at best a guide to action for those adopting approaches to usability that are new to them. As with culinary dishes, HCI needs to focus more on what gets cooked, and how it gets cooked, and not just on how recipes suggest that it could be cooked.     

基于CMDB的业务可用性评估

本文提出了一种基于CMDB的业务可用性评估方法,对业务可用性评估的业务逻辑、数据流和CMDB集成进行了深入分析。该方法充分利用CMDB丰富的数据模型,将应用基础架构和它们对业务服务的影响映射起来。从最终用户、业务和服务水平等的多角度出发,实时检测关键业务流程和系统性能指数,帮助企业主动地发现应用性能问题。根据服务水平和业务优先级来管理应用的可用性。最终可以确保缩短平均故障鉴定时间,提升服务水平。目前该方法已经成功应用到本企业的IT集中运行监控系统,效果良好。     

A Weighted QFD-Based Usability Evaluation Method for Elderly in Smart Cars

The aim of this research is to develop a quantitative usability evaluation method (UEM) for elderly drivers, which has different weight values on each factor concerning physical and cognitive context of elderly drivers. An analysis of the relationship between universal design guidelines for elderly drivers and usability principles was conducted by using the quality function deployment method. In addition, developmental priorities are derived from analysis results of difficulty in achieving performance improvement, max relationship values, and relative weight. Furthermore, n and positive relationships among the universal design guidelines are defined by means of relationship analysis. Combining these results, a quantitative evaluation guideline for elderly drivers is derived, and based on the context and developmental goals of the developer, selective design is facilitated. The proposed UEM is compared with existing UEM in terms of thoroughness, validity, and effectiveness.     

示范表演驱动的运动数据检索方法及可用性评估

以直观、自然的方式来检索运动数据可有效地提高角色动画生产效率.提出一种通过现场示范表演来检索运动数据的方法,使动画制作人员能够直观、快速地从大容量运动数据库中找到所期望的运动片段.首先通过运动传感器获取用户的示范表演,然后利用经典的DTW时序数据匹配算法来实现基于样例的运动数据检索.通过一组观察实验对它的可用性进行分析和评估的结果表明,文中方法能够显著地提升运动数据检索的效率及用户体验.     

A Study of the Evaluator Effect in Usability Testing

ABSTRACT

The evaluator effect names the observation that usability evaluators in similar conditions identify substantially different sets of usability problems. Yet little is known about the factors involved in the evaluator effect. We present a study of 50 novice evaluators' usability tests and subsequent comparisons, in teams and individually, of the resulting usability problems. The same problems were analyzed independently by 10 human–computer interaction experts. The study shows an agreement between evaluators of about 40%, indicating a substantial evaluator effect. Team matching of problems following the individual matching appears to improve the agreement, and evaluators express greater satisfaction with the teams' matchings. The matchings of individuals, teams, and independent experts show evaluator effects of similar sizes; yet individuals, teams, and independent experts fundamentally disagree about which problems are similar. Previous claims in the literature about the evaluator effect are challenged by the large variability in the matching of usability problems; we identify matching as a key determinant of the evaluator effect. An alternative view of usability problems and evaluator agreement is proposed in which matching is seen as an activity that helps to make sense of usability problems and where the existence of a correct matching is not assumed.     

Web课程的可用性研究

提高Web课程的可用性可以增强课程的学习效果.文章提出了Web课程可用性应考虑的三个方面:课程界面设计、课程内容设计和课程站点设计,并从实际设计中分析了实现Web课程可用性的指导原则,提出了改进Web课程可用性的具体措施和方法.     

Web课程的可用性研究

提高Web课程的可用性可以增强课程的学习效果。文章提出了Web课程可用性应考虑的三个方面：课程界面设计、课程内容设计和课程站点设计，并从实际设计中分析了实现Web课程可用性的指导原则，提出了改进Web课程可用性的具体措施和方法。