A Usability Checklist for the Usability Evaluation of Mobile Phone User Interface

In the last decade, the research of the usability of mobile phones has been a newly evolving area with few established methodologies and realistic practices that ensure capturing usability in evaluation. Thus, there exists growing demand to explore appropriate evaluation methodologies that evaluate the usability of mobile phones quickly as well as comprehensively. This study aims to develop a task-based usability checklist based on heuristic evaluations in views of mobile phone user interface (UI) practitioners. A hierarchical structure of UI design elements and usability principles related to mobile phones were developed and then utilized to develop the checklist. To demonstrate the practical effectiveness of the proposed checklist, comparative experiments were conducted on the usability checklist and usability testing. The majority of usability problems found by usability testing and additional problems were discovered by the proposed checklist. It is expected that the usability checklist proposed in this study could be used quickly and efficiently by usability practitioners to evaluate the mobile phone UI in the middle of the mobile phone development process.     

Usability risk level evaluation for physical user interface of mobile phone

The mobile phone has become an indispensable device in everyday life. However, many usability problems were generated by the multi-functionality of mobile phones. There was an important increase on the user interface (UI) design and usability. With the introduction of a variety of forms and keypads in mobile phones, the user's need of controllability, grip-stability, and usability were raised in the UI aspect. Therefore, in this study, we investigated the physical user interface (PUI) elements of mobile phones, and defined the degree of usability risk such as key level value, function level value, and grip level value. Also, this study proposed an evaluation framework to quantitatively measure the usability risk in the mobile phone PUI elements. For this, the mobile phone PUI elements were analyzed and classified by key type, function, and form-factor. Hence, in the case study was investigated and evaluated 133 mobile phones in the market by the proposed framework. As a result, the proposed evaluation framework enabled the measurement of the mobile phone usability risk level in the early stages of the concept design. This made possible the prediction of design problems related with the mobile phone PUI.     

The influence of user expertise and phone complexity on performance, ease of use and learnability of different mobile phones

The study focuses on usability, ease of use and learnability of three different mobile phones (Nokia 3210, Siemens C35i, Motorola P7389). The first independent variable refers to the complexity of the menu (depth/breadth of the menu tree) and navigation keys (number/functionality). The Nokia phone had the lowest and the Motorola the highest complexity, with the Siemens phone ranging between them. The second independent variable was user expertise: 30 novices and 30 experts solved six telephone tasks. In order to assess effects of learnability, tasks were presented twice. Differences between the mobile phones regarding effectiveness, efficiency and learnability were found: The best performance was shown by Nokia users. The remaining two phones did not differ significantly, although the most complex phone was superior to the phone of medium complexity which had the lowest performance. Moreover, an effect of expertise was confirmed, though suboptimal interfaces were identified as lessening the advantage of expertise. Specific weaknesses of the tested phones are discussed.     

The influence of user expertise and phone complexity on performance,ease of use and learnability of different mobile phones

The study focuses on usability, ease of use and learnability of three different mobile phones (Nokia 3210, Siemens C35i, Motorola P7389). The first independent variable refers to the complexity of the menu (depth/breadth of the menu tree) and navigation keys (number/functionality). The Nokia phone had the lowest and the Motorola the highest complexity, with the Siemens phone ranging between them. The second independent variable was user expertise: 30 novices and 30 experts solved six telephone tasks. In order to assess effects of learnability, tasks were presented twice. Differences between the mobile phones regarding effectiveness, efficiency and learnability were found: The best performance was shown by Nokia users. The remaining two phones did not differ significantly, although the most complex phone was superior to the phone of medium complexity which had the lowest performance. Moreover, an effect of expertise was confirmed, though suboptimal interfaces were identified as lessening the advantage of expertise. Specific weaknesses of the tested phones are discussed.     

Examining modality usage in a conversational multimodal application for mobile e-mail access

As Third Generation (3G) networks emerge they provide not only higher data transmission rates but also the ability to transmit both voice and low latency data within the same session. This paper describes the architecture and implementation of a multimodal application (voice and text) that uses natural language understanding combined with a WAP browser to access email messages on a cell phone. We present results from the use of the system by users as part of a laboratory trial that evaluated usage. The user trial also compared the multimodal system with a text-only system that is representative of current products in the market today. We discuss the observed modality issues and highlight implementation problems and usability concerns that were encountered in the trial. Findings indicate that speech was used the majority of the time by participants for both input and navigation even though most of the participants had little or no prior experience with speech systems (yet did have prior experience with text-only access to applications on their phones). To our knowledge this represents the first implementation and evaluation of its kind using this combination of technologies on an unmodified cell phone. Design implications resulting from the study findings and usability issues encountered are presented to inform the design of future conversational multimodal mobile applications.     

WebTracer: A new web usability evaluation environment using gazing point information

WebTracer is a new usability evaluation environment that supports recording, replaying, and analysis of a gazing point and operation while a user is browsing a website. WebTracer can record a user’s gazing point and operation compactly.Results of an experimental evaluation showed that the size of the operation history recorded by WebTracer was from 1/10 to 1/20 of the size of data recorded by an MPEG-2 or MPEG-4 format. With its compact form, the results of usability testing with the gazing point can be efficiently shared. It is expected to easily share empirical data between researchers. Also, evaluators can easily send the testing results as a feedback to the developers over the Internet.Moreover, the results show a possibility that gazing points related to usability. For example, when the menu of a Web page is divided into two panes, gazing points move more quickly. It seems that WebTracer efficiently improves usability evaluation, with gazing point data helping to identify problems on Web pages.     

Effectiveness of user testing and heuristic evaluation as a function of performance classification

For different levels of user performance, different types of information are processed and users will make different types of errors. Based on the error's immediate cause and the information being processed, usability problems can be classified into three categories. They are usability problems associated with skill-based, rule-based and knowledge-based levels of performance. In this paper, a user interface for a Web-based software program was evaluated with two usability evaluation methods, user testing and heuristic evaluation. The experiment discovered that the heuristic evaluation with human factor experts is more effective than user testing in identifying usability problems associated with skill-based and rule-based levels of performance. User testing is more effective than heuristic evaluation in finding usability problems associated with the knowledge-based level of performance. The practical application of this research is also discussed in the paper.     

Comparison of evaluation methods using structured usability problem reports

. Recent HCI research has produced analytic evaluation techniques which claim to predict potential usability problems for an interactive system.Validation of these methods has involved matching predicted problems against usability problems found during empirical user testing. This paper shows that the matching of predicted and actual problems requires careful attention, and that current approaches lack rigour or generality. Requirements for more rigorous and general matching procedures are presented. A solution to one key requirement is presented: a new report structure for usability problems.It is designed to improve the quality of matches made between usability problems found during empirical user testing and problems predicted by analytic methods. The use of this report format is placed within its design research context, an ongoing project on domain-specific methods for software visualizations.     

The relation between usability and product success in cell phones

The cell phone is not a simple communication tool. People use embedded mobile phone features such as games, cameras and wireless internet for various purposes of entertainment and shopping. In some cases usability problems impact negatively on product success, whereas in other cases they do not. The objective of our study was to develop a questionnaire for cell phones and investigate the relation between usability and product success. We reviewed existing usability studies and factors for product success to develop the questionnaire for cell phones. Subjects then evaluated the usability and success factors of cell phones. Finally, re-purchase intention and success scores including usability scores were analysed to determine whether they are positively related and which usability factors impact cell phone product success. Design, customer needs and innovativeness of cell phones were especially important among success factors, whereas more attention should be paid to satisfaction, feedback and efficiency among the usability dimensions.     

Investigating the usability of assistive user interfaces

A prototype e-mail system was developed for cognitively disabled users, with four different interfaces (free format, idea prompt, form fill and menu driven). The interfaces differed in the level of support provided for the user and complexity of facilities for composing e-mail messages. Usability evaluation demonstrated that no one interface was superior because of individual differences in usability problems, although the majority of users preferred interfaces which did not restrict their freedom of expression (free format). In contrast to traditional evaluation studies, no common pattern of usability errors emerged, demonstrating the need for customisation of interfaces for individual cognitively disabled users. A framework for customising user interfaces to individual users is proposed, and usability principles derived from the study are expressed as claims following the task artefact cycle.     

Comparison of evaluation methods using structured usability problem reports

Abstract

Recent HCI research has produced analytic evaluation techniques which claim to predict potential usability problems for an interactive system.Validation of these methods has involved matching predicted problems against usability problems found during empirical user testing. This paper shows that the matching of predicted and actual problems requires careful attention, and that current approaches lack rigour or generality. Requirements for more rigorous and general matching procedures are presented. A solution to one key requirement is presented: a new report structure for usability problems.It is designed to improve the quality of matches made between usability problems found during empirical user testing and problems predicted by analytic methods. The use of this report format is placed within its design research context, an ongoing project on domain-specific methods for software visualizations.     

可用性测试平台

阐明了软件可用性测试的重要性及其方法分类,着重说明了用户测试方法的步骤.介绍了所设计的软件可用性测试平台的特点和建设该平台的必要性,通过该平台对鱼眼菜单、单列菜单、级联式菜单进行了可用性测试.在测试用户较少的情况下,实验结果与其它主流实验室的结果相一致,表明了该实验平台的有效性和高效性.最后,对该测试平台可能的改进提出了设想.     

Usability Principles for Augmented Reality Applications in a Smartphone Environment

Through the rapid spread of smartphones, users have access to many types of applications similar to those on desktop computer systems. Smartphone applications using augmented reality (AR) technology make use of users' location information. As AR applications will require new evaluation methods, improved usability and user convenience should be developed. The purpose of the current study is to develop usability principles for the development and evaluation of smartphone applications using AR technology. We develop usability principles for smartphone AR applications by analyzing existing research about heuristic evaluation methods, design principles for AR systems, guidelines for handheld mobile device interfaces, and usability principles for the tangible user interface. We conducted a heuristic evaluation for three popularly used smartphone AR applications to identify usability problems. We suggested new design guidelines to solve the identified problems. Then, we developed an improved AR application prototype of an Android-based smartphone, which later was conducted a usability testing to validate the effects of usability principles.     

Mobility profiler: A framework for discovering mobility profiles of cell phone users

Mobility path information of cell phone users play a crucial role in a wide range of cell phone applications, including context based search and advertising, early warning systems, city-wide sensing applications such as air pollution exposure estimation and traffic planning. However, there is a disconnect between the low level location data logs available from the cell phones and the high level mobility path information required to support these cell phone applications. In this paper, we present formal definitions to capture the cell phone users’ mobility patterns and profiles, and provide a complete framework, Mobility Profiler, for discovering mobile cell phone user profiles starting from cell based location data. We use real-world cell phone log data (of over 350 K h of coverage) to demonstrate our framework and perform experiments for discovering frequent mobility patterns and profiles. Our analysis of mobility profiles of cell phone users expose a significant long tail in a user’s location-time distribution: A total of 15% of a cell phone user’s time is spent on average in locations that each appears with less than 1% of total time.     

The Easyline+ project: evaluation of a user interface developed to enhance independent living of elderly and disabled people

This paper reports the usability evaluation of interfaces developed to enable elderly and disabled people interact remotely with kitchen appliances in the home to enhance their independent living. A number of evaluation exercises were undertaken throughout the project??s development, including user-participative workshops and focus groups. This paper focuses on the summative usability evaluation exercise, which comprised a laboratory-based study in a simulated home environment, with a view to determining the appropriateness of employing this approach with potentially vulnerable participants. The study involved 27 participants interacting with the user interface. Their behaviour was observed and recorded, and their interaction with the system was analysed. They were also given a post-session questionnaire, where their opinions of the usability of the interface were solicited. The results of the usability testing were positive, and insight has been gained into how products of this nature can be further improved. The experience of conducting laboratory-based studies with vulnerable users was positive and led to propose in this paper a set of guidelines for future work in evaluating usability for work in this domain.     

The scenario‐based usability checklist development for home appliance design: A case study

In the past decade, home appliances have been rapidly developed to satisfy the various requirements of users. Thus, there are increases in requirements to explore appropriate evaluation methods that reflect the usability of home appliances quickly and comprehensively. This study aims to develop a scenario‐based usability checklist for product designers early in the design process. In this study, the scenario‐based usability checklist consists of two parts: 1) a heuristic evaluation checklist, and 2) a scenario evaluation checklist. In heuristic evaluation checklist development, usability factors of home appliances are extracted and then coupled to user interface (UI) elements. In scenario evaluation checklist development, scenarios are developed through brainstorming by focus group interview (FGI), and then evaluation elements are extracted from previous scenarios analysis. The proposed scenario‐based usability checklists can enable designers to evaluate product design quickly and comprehensively early in the development process with users' viewpoints. © 2010 Wiley Periodicals, Inc.     

Comparison of Mobile Web Browsers for Smartphones

Previous studies on mobile Web browsers have focused on phone features and technical performance, but not on the user interface. In this study, we compared three mobile Web browsers in the iPhone and ultimately proposed recommendations for the design of mobile Web browsers for smartphones. To develop such recommendations, we redefined the user interface (UI) components of mobile Web browsers from previous studies to conform to smartphones and compared the Chrome, Safari, and Opera Mini-browsers. To do this, we conducted experiments on usability tests with several dependent variables, including usefulness, perceived time reduction, minimal action, usability, ease to memorize, learnability, fun, and satisfaction. Finally, we developed several recommendations, based on the experimental results and participant interviews, which will fulfill user needs for the interface design of mobile Web browsers for smartphones. Multi-windows need to be implemented in a 3D environment. Developers should consider UI components and features that ease learning how to use the browser for novice users.     

A Return on Investment Assessment Model for a Mobile User Interface Project at the Predevelopment Stage

We present a new approach for assessing the business performance of a mobile user interface (UI) project at its predevelopment stage. Traditionally, the performance measurement of a mobile UI is made from the perspective of usability. Yet, as the number of mobile phone users is rapidly increasing, and, given that stakeholders are beginning to realize the importance of the UI, there is a growing demand to measure its business performance quantitatively as well as qualitatively. This article presents a model that can facilitate Return on Investment (ROI) estimations pertaining to the development of a mobile UI before the service with the new UI is launched. We base our method on stated preference data for the purpose of applying a conjoint analysis. Subsequently, we evaluate the viability of the proposed model quantitatively by comparing the estimated ROI at the predevelopment stage with the actual sales revenue achieved after the launch of the service. The proposed model was successfully applied to real‐world mobile UI design projects at Korea Telecom. Its benefits are demonstrated here through a case study. © 2012 Wiley Periodicals, Inc.     

To twist or poke? A method for identifying usability issues with the rotary controller and touch screen for control of in-vehicle information systems

In-vehicle information systems (IVIS) can be controlled by the user via direct or indirect input devices. In order to develop the next generation of usable IVIS, designers need to be able to evaluate and understand the usability issues associated with these two input types. The aim of this study was to investigate the effectiveness of a set of empirical usability evaluation methods for identifying important usability issues and distinguishing between the IVIS input devices. A number of usability issues were identified and their causal factors have been explored. These were related to the input type, the structure of the menu/tasks and hardware issues. In particular, the translation between inputs and on-screen actions and a lack of visual feedback for menu navigation resulted in lower levels of usability for the indirect device. This information will be useful in informing the design of new IVIS, with improved usability.

Statement of Relevance: This paper examines the use of empirical methods for distinguishing between direct and indirect IVIS input devices and identifying usability issues. Results have shown that the characteristics of indirect input devices produce more serious usability issues, compared with direct devices and can have a negative effect on the driver–vehicle interaction.     

A comparison of usability evaluation methods for evaluating e-commerce websites

The importance of evaluating the usability of e-commerce websites is well recognised. User testing and heuristic evaluation methods are commonly used to evaluate the usability of such sites, but just how effective are these for identifying specific problems? This article describes an evaluation of these methods by comparing the number, severity and type of usability problems identified by each one. The cost of employing these methods is also considered. The findings highlight the number and severity level of 44 specific usability problem areas which were uniquely identified by either user testing or heuristic evaluation methods, common problems that were identified by both methods, and problems that were missed by each method. The results show that user testing uniquely identified major problems related to four specific areas and minor problems related to one area. Conversely, the heuristic evaluation uniquely identified minor problems in eight specific areas and major problems in three areas.