A framework for evaluating the usability of mobile phones based on multi-level, hierarchical model of usability factors

As a mobile phone has various advanced functionalities or features, usability issues are increasingly challenging. Due to the particular characteristics of a mobile phone, typical usability evaluation methods and heuristics, most of which are relevant to a software system, might not effectively be applied to a mobile phone. Another point to consider is that usability evaluation activities should help designers find usability problems easily and produce better design solutions. To support usability practitioners of the mobile phone industry, we propose a framework for evaluating the usability of a mobile phone, based on a multi-level, hierarchical model of usability factors, in an analytic way. The model was developed on the basis of a set of collected usability problems and our previous study on a conceptual framework for identifying usability impact factors. It has multi-abstraction levels, each of which considers the usability of a mobile phone from a particular perspective. As there are goal-means relationships between adjacent levels, a range of usability issues can be interpreted in a holistic as well as diagnostic way. Another advantage is that it supports two different types of evaluation approaches: task-based and interface-based. To support both evaluation approaches, we developed four sets of checklists, each of which is concerned, respectively, with task-based evaluation and three different interface types: Logical User Interface (LUI), Physical User Interface (PUI) and Graphical User Interface (GUI). The proposed framework specifies an approach to quantifying usability so that several usability aspects are collectively measured to give a single score with the use of the checklists. A small case study was conducted in order to examine the applicability of the framework and to identify the aspects of the framework to be improved. It showed that it could be a useful tool for evaluating the usability of a mobile phone. Based on the case study, we improved the framework in order that usability practitioners can use it more easily and consistently.     

Dogmas in the assessment of usability evaluation methods

Usability evaluation methods (UEMs) are widely recognised as an essential part of systems development. Assessments of the performance of UEMs, however, have been criticised for low validity and limited reliability. The present study extends this critique by describing seven dogmas in recent work on UEMs. The dogmas include using inadequate procedures and measures for assessment, focusing on win–lose outcomes, holding simplistic models of how usability evaluators work, concentrating on evaluation rather than on design and working from the assumption that usability problems are real. We discuss research approaches that may help move beyond the dogmas. In particular, we emphasise detailed studies of evaluation processes, assessments of the impact of UEMs on design carried out in real-world systems development and analyses of how UEMs may be combined.     

Training software developers and designers to conduct usability evaluations

Many efforts to improve the interplay between usability evaluation and software development rely either on better methods for conducting usability evaluations or on better formats for presenting evaluation results in ways that are useful for software designers and developers. Both of these approaches depend on a complete division of work between developers and evaluators. This article takes a different approach by exploring whether software developers and designers can be trained to conduct their own usability evaluations. The article is based on an empirical study where 36 teams with a total of 234 first-year university students on software development and design educations were trained through an introductory course in user-based website usability testing that was taught in 40 h. They used the techniques from this course for planning, conducting, and interpreting the results of a usability evaluation of an interactive website. They gained good competence in conducting the evaluation, defining user tasks and producing a usability report, while they were less successful in acquiring skills for identifying and describing usability problems.     

An empirical study of factors affecting the perceived usability of websites for student Internet users

This study analysed Student Internet Users’ (SIUs’) perception of Web usability. Adopting a user testing method, seven Web Usability Factors (WUFs) were tested for their significance in affecting the ease of use of website. Several elements in websites were also tested for their significance in affecting the WUFs. Result shows the most significant WUF is related to the aesthetic appeal of a website, i.e., Use of Colour and Font. However, it was found that most Web developers are not focusing on this important WUF. Elements such as site map, site search, product image catalogue and others were also found to positively affect SIUs’ perception of the WUFs. The results presented in this paper can be used as guidelines for designing usable websites for SIUs.     

The theoretical foundation and validity of a component-based usability questionnaire

Although software engineers extensively use a component-based software engineering (CBSE) approach, existing usability questionnaires only support a holistic evaluation approach, which focuses on the usability of the system as a whole. Therefore, this paper discusses a component-specific questionnaire for measuring the perceived ease-of-use of individual interaction components. A theoretical framework is presented for this compositional evaluation approach, which builds on Taylor's layered protocol theory. The application and validity of the component-specific measure is evaluated by re-examining the results of four experiments. Here, participants were asked to use the questionnaire to evaluate a total of nine interaction components used in a mobile phone, a room thermostat, a web-enabled TV set and a calculator. The applicability of the questionnaire is discussed in the setting of a new usability study of an MP3 player. The findings suggest that at least part of the perceived usability of a product can be evaluated on a component-based level.     

A survey of usability capability maturity models: implications for practice and research

Improving the position and effectiveness of user-centred design (UCD) in software and product development is a challenge in many companies. One step towards improvements is to carry out a usability capability maturity (UCM) assessment to evaluate the strengths and weaknesses of a development organisation in UCD. While several diverse UCM models have been introduced, little research has been published in the public domain on these models. The paper aims to fill this gap by analysing the main features of the models. The results show that most models represent different approaches – although some of them have the same roots – meaning that understanding one model is not enough to understand the basics of another model. All models can be used for examining the status of UCD in individual development projects. In addition, models provide various means for assessment of the status of UCD in other organisational areas. The level of documentation of models varies a lot, and very few empirical research results exist. Based on the results, implications for practice and research are suggested.     

Empirical validation of an automatic usability evaluation method

Today, the success of a software application strongly depends on the usability of its interface, so the evaluation of interfaces has become a crucial aspect of software engineering. It is recognized that automatic tools for graphical user interface evaluation may greatly reduce the costs of traditional activities performed during expert evaluation or user testing in order to estimate the success probability of an application. However, automatic methods need to be empirically validated in order to prove their effectiveness with respect to the attributes they are supposed to evaluate.In this work, we empirically validate a usability evaluation method conceived to assess consistency aspects of a GUI with no need to analyze the back-end. We demonstrate the validity of the approach by means of a comparative experimental study, where four web sites and a stand-alone interactive application are analyzed and the results compared to those of a human-based usability evaluation. The analysis of the results and the statistical correlation between the tool׳s rating and humans׳ average ratings show that the proposed methodology can indeed be a useful complement to standard techniques of usability evaluation.     

A usability study of iPhone built-in applications

The intuitiveness of gestural devices offers great potential for more physical ways of interacting, but it is imperative that such devices comply with existing, platform-independent usability principles that have proven their effect in heterogeneous settings over the last decades. This study focuses on iPhone, one of the most widely used haptic devices and reviews the usability problems of five built-in iPhone applications documented in related sources. The study investigates whether these problems are actually reported during testing with novice users, which is their severity and whether testing reveals additional issues. Overall, the study confirmed the list of usability problems documented and stressed the importance of conforming to established usability principles.     

Integration of usability evaluation and model-based software development

Model-based software development is carried out as a well defined process. Depending on the applied approach, different phases can be distinguished, e.g. requirements specification, design, prototyping, implementation and usability evaluation. During this iterative process manifold artifacts are developed and modified, including, e.g. models, source code and usability evaluation data. CASE tools support the development stages well, but lack a seamless integration of usability evaluation methods. We aim at bridging the gap between development and usability evaluation, through enabling the cooperative use of artifacts with the particular tools. As a result of integration usability experts save time to prepare an evaluation and evaluation results can be easier incorporated back into the development process. We show exemplary our work on enhancing the Eclipse framework to support usability evaluation for task model-based software development.     

大规模语料库可用性评测方法

提出了一种大规模语料库可用性评测方法。通过分析语料库工程的生命周期,构建了大规模语料库可用性评测指标体系,运用层次分析-模糊综合评价方法实现了语料库的可用性评测,给出了语料库的可用性级别。分析语料库的评测结果,确定影响语料库可用性的瓶颈因素,进而提出针对性的改进措施。最后,举例说明了该方法在语料库上的应用。     

Interpretation of a cross-cultural usability evaluation: A case study based on a hypermedia system for rare species management in Namibia

We present the results of a usability evaluation of a locally developed hypermedia information system aiming at conservation biologists and wildlife managers in Namibia. Developer and end user come from different ethnic backgrounds, as is common to software development in Namibia and many developing countries. To overcome both the cultural and the authoritarian gap between usability evaluator and user, the evaluation was held as a workshop with usability evaluators who shared the target users’ ethnic and social backgrounds. Different data collection methods were used and results as well as specific incidences recorded. Results suggest that it is difficult for Namibian computer users to evaluate functionality independently from content. Users displayed evidence of a passive search strategy and an expectation that structure is provided rather than self generated. The comparison of data collection methods suggests that questionnaires are inappropriate in Namibia because they do not elicit a truthful response from participants who tend to provide answers they think are “expected”. The paper concludes that usability goals and methods have to be determined and defined within the target users’ cultural context.     

Preventing user interface disasters

Abstract

This article defines a quantitative goal that is cheap to measure for the usability of a business application system for casual users. The article also describes a cost-effective method for attaining the goal. The goal is to eliminate all user interface disasters (UIDs) in a given system. UIDs are usability problems that seriously annoy users, or prevent them from accomplishing their work without help from a human being. The method consists of series of simple user tests without audio or video recording, and with little analysis after each user test. The article concludes by describing Baltica's results of applying the method to a medium-size business application for casual users.     

Experience with usability evaluation of e-learning systems

Progress in the field of e-learning has been slow, with related problems mainly associated with the poor design of e-learning systems. Moreover, because of a depreciated importance of usability, usability studies are not very frequent. This paper reports the experience with the usability assessment of intelligent learning and teaching systems which are based on TEx-Sys model and are intended to enhance the process of knowledge acquisition in daily classroom settings. The applied scenario-based usability evaluation, as a combination of behaviour and opinion based measurements, enabled to quantify usability in terms of users’ (teachers’ and students’) performance and satisfaction. According to the achieved results, the main directions for interface redesign are offered. The acquired experience indicates that useful usability assessments with a significant identification of interface limitations can be performed quite easily and quickly. On the other hand, it raised a series of questions which, in order to be clarified, require further comprehensive research, the more so if the employment of universal design within e-learning context is considered.     

What you get is what you see: revisiting the evaluator effect in usability tests

Usability evaluation is essential to user-centred design; yet, evaluators who analyse the same usability test sessions have been found to identify substantially different sets of usability problems. We revisit this evaluator effect by having 19 experienced usability professionals analyse video-recorded test sessions with five users. Nine participants analysed moderated sessions; 10 participants analysed unmoderated sessions. For the moderated sessions, participants reported an average of 33% of the problems reported by all nine of these participants and 50% of the subset of problems reported as critical or serious by at least one participant. For the unmoderated sessions, the percentages were 32% and 40%. Thus, the evaluator effect was similar for moderated and unmoderated sessions, and it was substantial for the full set of problems and still present for the most severe problems. In addition, participants disagreed in their severity ratings. As much as 24% (moderated) and 30% (unmoderated) of the problems reported by multiple participants were rated as critical by one participant and minor by another. The majority of the participants perceived an evaluator effect when merging their individual findings into group evaluations. We discuss reasons for the evaluator effect and recommend ways of managing it.     

A comparison of usability evaluation methods for evaluating e-commerce websites

The importance of evaluating the usability of e-commerce websites is well recognised. User testing and heuristic evaluation methods are commonly used to evaluate the usability of such sites, but just how effective are these for identifying specific problems? This article describes an evaluation of these methods by comparing the number, severity and type of usability problems identified by each one. The cost of employing these methods is also considered. The findings highlight the number and severity level of 44 specific usability problem areas which were uniquely identified by either user testing or heuristic evaluation methods, common problems that were identified by both methods, and problems that were missed by each method. The results show that user testing uniquely identified major problems related to four specific areas and minor problems related to one area. Conversely, the heuristic evaluation uniquely identified minor problems in eight specific areas and major problems in three areas.     

The importance of usability in product choice: A mobile phone case study

Usability has become established as an important aspect of product design. This paper describes an investigation that was carried out to identify the importance of usability in product choice related to other product attributes. Interviews were initially carried out to identify possible attributes that contribute to product choice. Experiments were then undertaken using the methods of active information search, structured preference elicitation, ranking and interviews in order to find out what attributes were important to people when choosing an example product – mobile phones. It was found that usability is indeed important in product choice but perhaps not as much as users themselves believe. Other attributes that were found to be more important were features, aesthetics and cost. The process of product choice was found to be complex and it may be the case that people have come to expect usability in their products. Further research is needed to understand more fully the changing role of usability in product choice and to further improve the quality of the user–product relationship.     

Web系统可用性自动化评估系统的设计与实现

为了从用户真实行为中挖掘可能存在的可用性问题,避免启发式评估不可靠的问题,提出了基于大量用户行为数据,面向Web系统的可用性自动化评估系统.通过自动化匹配大量用户行为数据与需求分析阶段产生的用例,发现不一致的任务和任务前后置关系进而发现可用性问题.真实项目实践结果表明,输出的异常任务和异常任务前后置关系对可用性评估人员有指导价值,可以为进一步改进Web系统提供参考.     

The effects of usability and web design attributes on user preference for e-commerce web sites

This study examines the relationships among perceived usability before actual use, task completion time, and preference, and the effects of design attributes on user preference for e-commerce web sites. Nine online bookstore web sites were used by ten participants. Results indicate: (1) pre-use usability and task completion time were correlated; (2) the relationship between pre-use usability and preference was greater than that of task completion time and preference; (3) design attribute assessments after actual use were highly intercorrelated; and (4) organizational structure and layout had a greater effect on user preference than aesthetic aspects, such as color and typography. These findings can be used to construct a conceptual framework for understanding user preferences and to develop design guidelines to yield more highly preferred e-commerce web sites. Also, the methodology in this study can be applied to other computerized-applications.