Dogmas in the assessment of usability evaluation methods

Usability evaluation methods (UEMs) are widely recognised as an essential part of systems development. Assessments of the performance of UEMs, however, have been criticised for low validity and limited reliability. The present study extends this critique by describing seven dogmas in recent work on UEMs. The dogmas include using inadequate procedures and measures for assessment, focusing on win–lose outcomes, holding simplistic models of how usability evaluators work, concentrating on evaluation rather than on design and working from the assumption that usability problems are real. We discuss research approaches that may help move beyond the dogmas. In particular, we emphasise detailed studies of evaluation processes, assessments of the impact of UEMs on design carried out in real-world systems development and analyses of how UEMs may be combined.     

A usability study of iPhone built-in applications

The intuitiveness of gestural devices offers great potential for more physical ways of interacting, but it is imperative that such devices comply with existing, platform-independent usability principles that have proven their effect in heterogeneous settings over the last decades. This study focuses on iPhone, one of the most widely used haptic devices and reviews the usability problems of five built-in iPhone applications documented in related sources. The study investigates whether these problems are actually reported during testing with novice users, which is their severity and whether testing reveals additional issues. Overall, the study confirmed the list of usability problems documented and stressed the importance of conforming to established usability principles.     

Interpretation of a cross-cultural usability evaluation: A case study based on a hypermedia system for rare species management in Namibia

We present the results of a usability evaluation of a locally developed hypermedia information system aiming at conservation biologists and wildlife managers in Namibia. Developer and end user come from different ethnic backgrounds, as is common to software development in Namibia and many developing countries. To overcome both the cultural and the authoritarian gap between usability evaluator and user, the evaluation was held as a workshop with usability evaluators who shared the target users’ ethnic and social backgrounds. Different data collection methods were used and results as well as specific incidences recorded. Results suggest that it is difficult for Namibian computer users to evaluate functionality independently from content. Users displayed evidence of a passive search strategy and an expectation that structure is provided rather than self generated. The comparison of data collection methods suggests that questionnaires are inappropriate in Namibia because they do not elicit a truthful response from participants who tend to provide answers they think are “expected”. The paper concludes that usability goals and methods have to be determined and defined within the target users’ cultural context.     

Web系统可用性自动化评估系统的设计与实现

为了从用户真实行为中挖掘可能存在的可用性问题,避免启发式评估不可靠的问题,提出了基于大量用户行为数据,面向Web系统的可用性自动化评估系统.通过自动化匹配大量用户行为数据与需求分析阶段产生的用例,发现不一致的任务和任务前后置关系进而发现可用性问题.真实项目实践结果表明,输出的异常任务和异常任务前后置关系对可用性评估人员有指导价值,可以为进一步改进Web系统提供参考.     

A comparison of two analytical evaluation methods for educational computer games for young children

In this paper we describe a comparison of two analytical methods for educational computer games for young children. The methods compared in the study are the Structured Expert Evaluation Method (SEEM) and the Combined Heuristic Evaluation (HE) (based on a combination of Nielsen’s HE and the fun-related concepts from Malone and Lepper) with both usability and fun heuristics for children’s computer games. To verify SEEM’s relative quality, a study was set up in which adult evaluators predicted problems in computer games. Outcomes based on thoroughness (whether the analytical method finds all problems), validity (whether the analytical method uncovers problems that are likely to be true) and appropriateness (whether the method is applied correctly) are compared. The results show that both the thoroughness and validity of SEEM are higher than the thoroughness and validity of the Combined HE. The appropriateness scores indicate that SEEM gives evaluators more guidance when predicting problems than the Combined HE does.

A comparison of the four prominent user-based methods for evaluating the usability of computer software

The aim was to examine four prominent user-based computer software usability evaluation methods. Four evaluation methods (logged data, questionnaire, inter view, and verbal protocol analysis) were used to evaluate three different business software types (spreadsheet, word processor, and database) using a between-groups design, involving 148 individuals of mixed age and gender. Comparisons were made to examine the efficiency of each evaluation method in terms of its ability to highlight usability problems both between and within the evaluation strategy. Here, the verbal protocol analysis was found to be most efficient. The possibility of further efficiency gains by using two evaluation methods was also examined, where it was found that no statistically significant improvement was obtained over the verbal protocol analysis used by itself. Ways in which the utility of the methods may be enhanced was also discussed.     

An empirical study of factors affecting the perceived usability of websites for student Internet users

This study analysed Student Internet Users’ (SIUs’) perception of Web usability. Adopting a user testing method, seven Web Usability Factors (WUFs) were tested for their significance in affecting the ease of use of website. Several elements in websites were also tested for their significance in affecting the WUFs. Result shows the most significant WUF is related to the aesthetic appeal of a website, i.e., Use of Colour and Font. However, it was found that most Web developers are not focusing on this important WUF. Elements such as site map, site search, product image catalogue and others were also found to positively affect SIUs’ perception of the WUFs. The results presented in this paper can be used as guidelines for designing usable websites for SIUs.     

Empirical validation of an automatic usability evaluation method

Today, the success of a software application strongly depends on the usability of its interface, so the evaluation of interfaces has become a crucial aspect of software engineering. It is recognized that automatic tools for graphical user interface evaluation may greatly reduce the costs of traditional activities performed during expert evaluation or user testing in order to estimate the success probability of an application. However, automatic methods need to be empirically validated in order to prove their effectiveness with respect to the attributes they are supposed to evaluate.In this work, we empirically validate a usability evaluation method conceived to assess consistency aspects of a GUI with no need to analyze the back-end. We demonstrate the validity of the approach by means of a comparative experimental study, where four web sites and a stand-alone interactive application are analyzed and the results compared to those of a human-based usability evaluation. The analysis of the results and the statistical correlation between the tool׳s rating and humans׳ average ratings show that the proposed methodology can indeed be a useful complement to standard techniques of usability evaluation.     

Integration of usability evaluation and model-based software development

Model-based software development is carried out as a well defined process. Depending on the applied approach, different phases can be distinguished, e.g. requirements specification, design, prototyping, implementation and usability evaluation. During this iterative process manifold artifacts are developed and modified, including, e.g. models, source code and usability evaluation data. CASE tools support the development stages well, but lack a seamless integration of usability evaluation methods. We aim at bridging the gap between development and usability evaluation, through enabling the cooperative use of artifacts with the particular tools. As a result of integration usability experts save time to prepare an evaluation and evaluation results can be easier incorporated back into the development process. We show exemplary our work on enhancing the Eclipse framework to support usability evaluation for task model-based software development.     

Experience with usability evaluation of e-learning systems

Progress in the field of e-learning has been slow, with related problems mainly associated with the poor design of e-learning systems. Moreover, because of a depreciated importance of usability, usability studies are not very frequent. This paper reports the experience with the usability assessment of intelligent learning and teaching systems which are based on TEx-Sys model and are intended to enhance the process of knowledge acquisition in daily classroom settings. The applied scenario-based usability evaluation, as a combination of behaviour and opinion based measurements, enabled to quantify usability in terms of users’ (teachers’ and students’) performance and satisfaction. According to the achieved results, the main directions for interface redesign are offered. The acquired experience indicates that useful usability assessments with a significant identification of interface limitations can be performed quite easily and quickly. On the other hand, it raised a series of questions which, in order to be clarified, require further comprehensive research, the more so if the employment of universal design within e-learning context is considered.     

The influence of product aesthetics and usability over the course of time: a longitudinal field experiment

A longitudinal field experiment was carried out over a period of 2 weeks to examine the influence of product aesthetics and inherent product usability. A 2 × 2 × 3 mixed design was used in the study, with product aesthetics (high/low) and usability (high/low) being manipulated as between-subjects variables and exposure time as a repeated-measures variable (three levels). A sample of 60 mobile phone users was tested during a multiple-session usability test. A range of outcome variables was measured, including performance, perceived usability, perceived aesthetics and emotion. A major finding was that the positive effect of an aesthetically appealing product on perceived usability, reported in many previous studies, began to wane with increasing exposure time. The data provided similar evidence for emotion, which also showed changes as a function of exposure time. The study has methodological implications for the future design of usability tests, notably suggesting the need for longitudinal approaches in usability research. PRACTITIONER SUMMARY: This study indicates that product aesthetics influences perceived usability considerably in one-off usability tests but this influence wanes over time. When completing a usability test it is therefore advisable to adopt a longitudinal multiple-session approach to reduce the possibly undesirable influence of aesthetics on usability ratings.     

一种新的笔式交互范式的可用性评估

笔式用户界面是Post—WIMP界面的一种重要形态,具有自然、和谐的交互特性。尽管笔式用户界面广泛应用于无处不在计算环境中,但很少专门对它的可用性进行评估。针对笔式用户界面的特征,提出了一种新的笔式用户界面范式,基于此范式设计并实现了一个面向教学的笔式交互系统(PenOffice),利用此系统对范式从易学性、效率、出错率和用户满意度这几个方面进行了可用性评估,从中分析了笔式用户界面以及这种新的界面范式的优点和不足。     

C2C电子商务中基于声誉的信任评估模型

C2C模式电子交易平台以其便利、自由和高效性,而得到迅速发展,但由于其中信任评价机制的不完善,使得对交易参与者的监管难度增大,交易中仍存在较大的风险。针对目前信任评价机制的不完善,提出了基于声誉的信任机制,通过计算交易参与方的信任度以达到识别不诚实的个体的目的。通过进一步研究和分析影响评价的因素,使用时间窗口和交易窗口机制对参与计算的评价进行选择,并综合考虑评价者的自身信任度。对模型的有效性和效率都进行了理论分析和实验验证。结果表明,提出的模型能够有效地应用于C2C电子商务系统中。     

The effects of usability and web design attributes on user preference for e-commerce web sites

This study examines the relationships among perceived usability before actual use, task completion time, and preference, and the effects of design attributes on user preference for e-commerce web sites. Nine online bookstore web sites were used by ten participants. Results indicate: (1) pre-use usability and task completion time were correlated; (2) the relationship between pre-use usability and preference was greater than that of task completion time and preference; (3) design attribute assessments after actual use were highly intercorrelated; and (4) organizational structure and layout had a greater effect on user preference than aesthetic aspects, such as color and typography. These findings can be used to construct a conceptual framework for understanding user preferences and to develop design guidelines to yield more highly preferred e-commerce web sites. Also, the methodology in this study can be applied to other computerized-applications.     

Understanding user interface needs of e-commerce web sites

With the growth in the number of web sites, users must choose among many alternative web sites and designers must face the challenge of keeping the user's duration. Web site design is an iterative process of fine-tuning to enhance user satisfaction. According to the hygiene–motivation theory, the emotions of satisfaction and dissatisfaction are caused by two different categories of characteristics. Lack of dissatisfaction does not mean satisfaction, and lack of satisfaction does not mean dissatisfaction. This study proposes a parsimonious and effective multidimensional evaluation tool: a max–min approach to combine the hygiene–motivation factors and thereby provide a strategy to compare the competitive position of different web sites. The interfaces of eight e-retailer web sites for computer and communication products were targeted for evaluation. Four hygiene factors (navigation, information display, ease of learning and response time) and five motivation factors (interaction with people, screen complexity, user empowerment, visual appearance and achievement) are measured. The evaluations on hygiene and motivation factors were transformed into max and min effectiveness scores using the proposed models. Three zones (zone of intolerance, zone of efficiency and zone of satisfaction) were described to illustrate the role of strategic planning of a site interface in shaping a competitive position among the observed web sites.     

A methodology for evaluating the usability of audiovisual consumer electronic products

Usability evaluation is now considered an essential procedure in consumer product development. Many studies have been conducted to develop various techniques and methods of usability evaluation hoping to help the evaluators choose appropriate methods. However, planning and conducting usability evaluation requires considerations of a number of factors surrounding the evaluation process including the product, user, activity, and environmental characteristics. In this perspective, this study suggested a new methodology of usability evaluation through a simple, structured framework. The framework was outlined by three major components: the interface features of a product as design variables, the evaluation context consisting of user, product, activity, and environment as context variables, and the usability measures as dependent variables. Based on this framework, this study established methods to specify the product interface features, to define evaluation context, and to measure usability. The effectiveness of this methodology was demonstrated through case studies in which the usability of audiovisual products was evaluated by using the methods developed in this study. This study is expected to help the usability practitioners in consumer electronics industry in various ways. Most directly, it supports the evaluators' plan and conduct usability evaluation sessions in a systematic and structured manner. In addition, it can be applied to other categories of consumer products (such as appliances, automobiles, communication devices, etc.) with minor modifications as necessary.     

The importance of usability in product choice: A mobile phone case study

Usability has become established as an important aspect of product design. This paper describes an investigation that was carried out to identify the importance of usability in product choice related to other product attributes. Interviews were initially carried out to identify possible attributes that contribute to product choice. Experiments were then undertaken using the methods of active information search, structured preference elicitation, ranking and interviews in order to find out what attributes were important to people when choosing an example product – mobile phones. It was found that usability is indeed important in product choice but perhaps not as much as users themselves believe. Other attributes that were found to be more important were features, aesthetics and cost. The process of product choice was found to be complex and it may be the case that people have come to expect usability in their products. Further research is needed to understand more fully the changing role of usability in product choice and to further improve the quality of the user–product relationship.     

A survey of usability capability maturity models: implications for practice and research

Improving the position and effectiveness of user-centred design (UCD) in software and product development is a challenge in many companies. One step towards improvements is to carry out a usability capability maturity (UCM) assessment to evaluate the strengths and weaknesses of a development organisation in UCD. While several diverse UCM models have been introduced, little research has been published in the public domain on these models. The paper aims to fill this gap by analysing the main features of the models. The results show that most models represent different approaches – although some of them have the same roots – meaning that understanding one model is not enough to understand the basics of another model. All models can be used for examining the status of UCD in individual development projects. In addition, models provide various means for assessment of the status of UCD in other organisational areas. The level of documentation of models varies a lot, and very few empirical research results exist. Based on the results, implications for practice and research are suggested.     

Usability evaluation methods for the web: A systematic mapping study

Context

In recent years, many usability evaluation methods (UEMs) have been employed to evaluate Web applications. However, many of these applications still do not meet most customers’ usability expectations and many companies have folded as a result of not considering Web usability issues. No studies currently exist with regard to either the use of usability evaluation methods for the Web or the benefits they bring.

Objective

The objective of this paper is to summarize the current knowledge that is available as regards the usability evaluation methods (UEMs) that have been employed to evaluate Web applications over the last 14 years.

Method

A systematic mapping study was performed to assess the UEMs that have been used by researchers to evaluate Web applications and their relation to the Web development process. Systematic mapping studies are useful for categorizing and summarizing the existing information concerning a research question in an unbiased manner.

Results

The results show that around 39% of the papers reviewed reported the use of evaluation methods that had been specifically crafted for the Web. The results also show that the type of method most widely used was that of User Testing. The results identify several research gaps, such as the fact that around 90% of the studies applied evaluations during the implementation phase of the Web application development, which is the most costly phase in which to perform changes. A list of the UEMs that were found is also provided in order to guide novice usability practitioners.

Conclusions

From an initial set of 2703 papers, a total of 206 research papers were selected for the mapping study. The results obtained allowed us to reach conclusions concerning the state-of-the-art of UEMs for evaluating Web applications. This allowed us to identify several research gaps, which subsequently provided us with a framework in which new research activities can be more appropriately positioned, and from which useful information for novice usability practitioners can be extracted.