Development of a Weighted Heuristic for Website Evaluation for Older Adults

Older adults are the fastest growing population of Internet users. As websites acquire a greater number of older visitors, it is vital that they are designed with this demographic in mind. Older users typically have different user characteristics than younger users; they may have changes in perceptual abilities, motor skills, cognitive abilities, mental models, and confidence in the use of technology. This research documents the development of a new weighted heuristic measure for evaluating the usability of websites for older adults and its validation with performance testing. Results from a repeated measures analysis of variance indicated that websites with different heuristic classifications were significantly different with respect to performance metrics and System Usability Scale ratings. Conclusions point to the need for web design that takes into account preferences and abilities of older web users.     

Evaluating expert systems

Abstract: Evaluation is crucial for improving expert system design and performance. This paper stresses the need for considering system evaluation throughout the development process. It highlights the importance of evaluating system usability and discusses key usability issues. A number of basic evaluation methods are described, including interviews, questionnaires, observation, system logging, user diaries, laboratory experiments and field trials. Finally, the paper looks at evaluating systems within organisations, and assessing other long term effects of expert systems.     

Understanding Usability and User Experience of Web-Based 3D Graphics Technology

As Web-based interactive 3D graphics (Web 3D), popularly referred to as Virtual Reality, continue to become more affordable, research and development groups in various fields have been adopting Web 3D technology. In addition to simulation of 3D content, the ability to instantly display alternative looks has been recognized as an innovative way to improve communication in such fields as product design, architecture, and e-commerce. Despite substantial adoption of Web 3D, how and how much the technology benefits target users as well as the providers who choose to adapt the Web 3D technology are not well understood. Previous research has established that interactive 3D graphics provide users with unique human–computer interaction (HCI). However, little is known about how users experience the Web 3D graphics technology and how user–system interaction contributes to system usability. The purpose of this study is to build new knowledge of the user experience with interactive-3D graphics systems used for product demonstration. By testing the impact of the technology on the user–system interaction and usability and comparing this impact with that of conventional two-dimensional (2D) graphics, this study tries to better understand the Web 3D technology from an interdisciplinary view of technology acceptance, sense of presence, and HCI. The study investigated how system usability is affected by HCI in the context of a furniture-style preference survey. The results of the study display the clear advantage for Web 3D for usability and show that perceived usefulness and sense of presence both mediate the effect of the technology treatment on the usability outcomes. The contribution of this study is that it includes empirical data to show how Web 3D benefits users when adopted in the context of a product demonstration and how the advantage is obtained through the user's interaction with the Web 3D technology.     

Usability measurement and metrics: A consolidated model

Usability is increasingly recognized as an important quality factor for interactive software systems, including traditional GUIs-style applications, Web sites, and the large variety of mobile and PDA interactive services. Unusable user interfaces are probably the single largest reasons why encompassing interactive systems – computers plus people, fail in actual use. The design of this diversity of applications so that they actually achieve their intended purposes in term of ease of use is not an easy task. Although there are many individual methods for evaluating usability; they are not well integrated into a single conceptual framework that facilitate their usage by developers who are not trained in the filed of HCI. This is true in part because there are now several different standards (e.g., ISO 9241, ISO/IEC 9126, IEEE Std.610.12) or conceptual models (e.g., Metrics for Usability Standards in Computing [MUSiC]) for usability, and not all of these standards or models describe the same operational definitions and measures. This paper first reviews existing usability standards and models while highlighted the limitations and complementarities of the various standards. It then explains how these various models can be unified into a single consolidated, hierarchical model of usability measurement. This consolidated model is called Quality in Use Integrated Measurement (QUIM). Included in the QUIM model are 10 factors each of which corresponds to a specific facet of usability that is identified in an existing standard or model. These 10 factors are decomposed into a total of 26 sub-factors or measurable criteria that are furtherdecomposed into 127 specific metrics. The paper explains also how a consolidated model, such as QUIM, can help in developing a usability measurement theory. Ahmed Seffah interests are at the intersection of human-computer interaction and software engineering, with an emphasis on human-centered software engineering, empirical studies, theoretical models for quality in use measurement, as well as patterns as a vehicle for capturing and incorporating empirically valid design practices in software engineering practices. He is a co-founder of the Usability and Empirical Studies Lab and the founder and the chair of the Human-Centered Software Engineering Research Group at Concordia University. Harkirat K. Padda is a Ph.D. candidate at Concordia University (Montreal) since 2003, where she completed her masters’ degree in computer science. Her research interests are software quality measurement, metrics, and empirical software evaluation. Being a member of Human Computer Software Engineering Group, she is exploring the comprehension and usability of software systems—in particular the visualization systems. In her masters’ work, she already defined a repository of different metrics to measure the ‘quality in use’ factors of software products in general. Currently, she is proposing a pattern-oriented measurement framework to measure comprehension of visualization systems.     

Survey of public access information retrieval systems

A London HCI Centre (LHC) project to produce a user interface development environment for screen-based, nondirect manipulation, public use computer systems is described. A survey of available public access computer systems was undertaken to characterize and critically assess the usability of current designs. That part of the survey relating to four particular information retrieval systems is presented. Some serious weaknesses in the user interfaces to these systems are highlighted along with recommendations for improvements. The paper concludes with a general discussion of the major areas of weakness and how human factors guidelines can be used to improve usability.     

Learning gaze typing: what are the obstacles and what progress to expect?

Gaze interaction is a promising input modality for people who are unable to control their fingers and arms. This paper suggests a number of new metrics that can be applied to the analysis of gaze typing interfaces and to the evaluation of user performance. These metrics are derived from a close examination of eight subjects typing text by gazing on a dwell-time activated onscreen keyboard during a seven-day experiment. One of the metrics, termed “Attended keys per character”, measures the number of keys that are attended for each typed character. This metric turned out to be particularly well correlated to the actual numbers of errors committed (r = 0.915). In addition to introducing metrics specific for gaze typing, the paper discusses how the metrics could make remote progress monitoring possible and provides some general advice on how to introduce gaze typing for novice users.     

Comparative Usability Analysis of Selected Data Entry Methods for Web Systems

Abstract

This paper addresses usability problems of data entry techniques and patterns in web systems and web sites. In order to carry out a comparative analysis of the solutions a dedicated web application was developed as an experimental tool for performing usability testing. The application was designed in the form of a test divided into scenarios and exercises on filling up selected models. Furthermore, the application recorded usability metrics, questionnaires, open and closed questions, as well as users’ personal data. 40 participants, who took part remotely in the experiment, were split into two distinct groups and completed two separate tests that differed in input data and tested solutions. Some measures referring to the effectiveness, efficiency, and user satisfaction were collected. The results were thoroughly analyzed and the recommendations for the usage of particular models were formulated.     

Performance evaluation of object detection algorithms for video surveillance

In this paper, we propose novel methods to evaluate the performance of object detection algorithms in video sequences. This procedure allows us to highlight characteristics (e.g., region splitting or merging) which are specific of the method being used. The proposed framework compares the output of the algorithm with the ground truth and measures the differences according to objective metrics. In this way it is possible to perform a fair comparison among different methods, evaluating their strengths and weaknesses and allowing the user to perform a reliable choice of the best method for a specific application. We apply this methodology to segmentation algorithms recently proposed and describe their performance. These methods were evaluated in order to assess how well they can detect moving regions in an outdoor scene in fixed-camera situations.     

Improving the usability of information systems: the role of the technical communicator

A successful information system implementation requires effective logical analysis, sound technical construction, effective design for users and successful change management during implementation. If these conditions are not present then in all probability the system will be a failure. Ensuring a system is successful from a user's perspective is primarily about usability. There are many aspects of information system design that impact on usability including: online help, the design of the user interface, system and error messages. These aspects need consideration early in the development cycle. This paper discusses the need for improved usability design in information systems and presents results of recent research which examined the role of technical communicators in systems development and explored the extent to which their work improved system usability. The study found that from a user's perspective an improved system results when technical communicators are involved particularly in the early stages of the development process.     

Electronic voting eliminates hanging chads but introduces new usability challenges

The arrival of electronic voting has generated considerable controversy, mostly about its vulnerability to fraud. By comparison, virtually no attention has been given to its usability, i.e., voters’ ability to vote as they intend, which was central to the controversy surrounding the 2000 US presidential election. Yet it is hard to imagine a domain of human–computer interaction where usability has more impact on how democracy works. This article reports a laboratory investigation of the usability of six electronic voting systems chosen to represent the features of systems in current use and potentially in future use. The primary question was whether e-voting systems are sufficiently hard to use that voting accuracy and satisfaction are compromised. We observed that voters often seemed quite lost taking far more than the required number of actions to cast individual votes, especially when they ultimately voted inaccurately. Their satisfaction went down as their effort went up. And accuracy with some systems was disturbingly low. While many of these problems are easy to fix, manufacturers will need to adopt usability engineering practices that have vastly improved user interfaces throughout the software industry.     

Expert systems as extensions of the human mind: A user oriented,holistic approach to the design of multiple reasoning system environments and interfaces

Expert systems have had little impact as computing artifacts. In this paper we argue that the reason for this stems from the underlying assumption of most builders of expert systems that an expert system needs to acquire information and to control the interaction between the human user and itself. We show that this assumption has serious linguistic and usability flaws which diminish the likelihood of producing socially acceptable expert systems. We propose a reversal of this paradigm, for the design of expert systems, by assuming that it is the human user who needs to acquire information and to control the interaction between themselves and the system.     

The validity of driving simulation for assessing differences between in-vehicle informational interfaces: A comparison with field testing

Data from on-road and simulation studies were compared to assess the validity of measures generated in the simulator. In the on-road study, driver interaction with three manual address entry methods (keypad, touch screen and rotational controller) was assessed in an instrumented vehicle to evaluate relative usability and safety implications. A separate group of participants drove a similar protocol in a medium fidelity, fixed-base driving simulator to assess the extent to which simulator measures mirrored those obtained in the field. Visual attention and task measures mapped very closely between the two environments. In general, however, driving performance measures did not differentiate among devices at the level of demand employed in this study. The findings obtained for visual attention and task engagement suggest that medium fidelity simulation provides a safe and effective means to evaluate the effects of in-vehicle information systems (IVIS) designs on these categories of driver behaviour.

Statement of Relevance: Realistic evaluation of the user interface of IVIS has significant implications for both user acceptance and safety. This study addresses the validity of driving simulation for accurately modelling differences between interface methodologies by comparing results from the field with those from a medium fidelity, fixed-base simulator.     

Modelling the hare and the tortoise: predicting the range of in-vehicle task times using critical path analysis

Analytic models can enable predictions about important aspects of the usability of in-vehicle information systems (IVIS) to be made at an early stage of the product development process. Task times provide a quantitative measure of user performance and are therefore important in the evaluation of IVIS usability. In this study, critical path analysis (CPA) was used to model IVIS task times in a stationary vehicle, and the technique was extended to produce predictions for slowperson and fastperson performance, as well as average user (middleperson) performance. The CPA-predicted task times were compared to task times recorded in an empirical simulator study of IVIS interaction, and the predicted times were, on average, within acceptable precision limits. This work forms the foundation for extension of the CPA model to predict IVIS task times in a moving vehicle, to reflect the demands of the dual-task driving scenario.

Practitioner Summary: The CPA method was extended for the prediction of slowperson and fastperson IVIS task times. Comparison of the model predictions with empirical data demonstrated acceptable precision. The CPA model can be used in early IVIS evaluation; however, there is a need to extend it to represent the dual-task driving scenario.     

A structured methodology for comparative evaluation of user interface designs using usability criteria and measures

Previous research in software ergonomics has indicated the importance of evaluating the usability of computer user interfaces. This paper presents a quantitative basis for selecting from among multiple alternative interfaces relative to multiple criteria of usability concern. The proposed model consists of two main phases: the prescreening phase and the evaluation phase. The prescreening phase involves expert judgment-based assessment with qualitative criteria. The first phase uses absolute measurement analytic hierarchy process (AHP) to filter possible alternative interfaces to a reasonable subset. The evaluation phase involves user-based assessment such as user testing, with quantitative criteria. The objective of the second phase is to evaluate a subset of alternatives using objective measures and to select the best alternative. A set of criteria and measures for evaluating the usability of computer software designs is presented. The proposed methodology is demonstrated in the interface design of a database system used to analyze trip cases information of nuclear power plant.Relevance to industry

There is currently a focus on the usability of interactive computer software. Software developers, interface designers or human factors engineers often confront the task of comparative evaluation among systems, versions or interface designs. The proposed methodology provides practitioners with a structured approach to select the best interface based on usability criteria and measures.     

Exploring user satisfaction for e-learning systems via usage-based metrics and system usability scale analysis

The use of e-learning technology is incontestably recognized as an important and integral part of the educational process. Considerable research studies are carried out in order to apprehend how effective and usable e-learning systems. In this paper, an empirical-based study is conducted to explore how lecturers interact with an e-learning environment based on a predefined task model describing low-level interactions. Client-side log data is collected from university lecturers from the Electrical and Computer Science departments. Subsequently, data analysis is conducted to infer the usability degree from the estimated usage metrics together with further exploratory analysis from user feedback via System Usability Scale. Experimental results reveal that the System Usability Scale score is not a sufficient measure to express the true acceptance and satisfaction level of lecturers for using the e-learning systems. The evaluation must be fulfilled in tandem with analyzing the usage metrics derived from interaction traces in a non-intrusive fashion. The proposed approach is a milestone towards usability evaluation to improve the acceptance and user experience for academic staff and students.     

Using decision tree modelling to support Peircian abduction in IS research: a systematic approach for generating and evaluating hypotheses for systematic theory development

Since their early development, computers have had a profound impact on how we conduct modern scientific research. The disciplines of mathematics and operations research are perhaps the earliest to be dramatically transformed by information technology. However, over the years, computing technologies have provided many new opportunities for information processing, problem solving and knowledge creation. In this paper, we explore the potential of data mining technology for providing support for systematic theory testing based on Peirce's theory of abduction. We propose a data mining approach to abducting and evaluating hypotheses based on Peirce's scientific method. We believe that this approach could assist scientist to more efficiently explore alternative hypotheses for existing theories. We demonstrate our approach with empirical observations collected using instruments from the well known user performance area of information systems research.     

Usability testing in 2000 and beyond

Usability testing is a widely used technique to evaluate user performance and acceptance of products and systems. It was introduced in the late 1980s and rose to popularity in the past decade. This paper provides a view of the current status of usability testing as a method and describes how it will be used in the 21st century. Although usability testing may not be the most efficient technique for discovery of usability problems, it is a reliable way to estimate quantitatively users' performance and subjective satisfaction with products. Four major trends in usability testing include: common reporting formats and methods for industry; Internet application and website testing; testing of mobile, handheld devices; and testing in more naturalistic environments such as simulated homes and classrooms. In the 21st century, 'quick and clean' usability testing methods are needed to provide valid and reliable data on how well people use products and systems, and how they like using them.     

Usability Principles for Augmented Reality Applications in a Smartphone Environment

Through the rapid spread of smartphones, users have access to many types of applications similar to those on desktop computer systems. Smartphone applications using augmented reality (AR) technology make use of users' location information. As AR applications will require new evaluation methods, improved usability and user convenience should be developed. The purpose of the current study is to develop usability principles for the development and evaluation of smartphone applications using AR technology. We develop usability principles for smartphone AR applications by analyzing existing research about heuristic evaluation methods, design principles for AR systems, guidelines for handheld mobile device interfaces, and usability principles for the tangible user interface. We conducted a heuristic evaluation for three popularly used smartphone AR applications to identify usability problems. We suggested new design guidelines to solve the identified problems. Then, we developed an improved AR application prototype of an Android-based smartphone, which later was conducted a usability testing to validate the effects of usability principles.     

DaisyViz: A model-based user interface toolkit for interactive information visualization systems

While information visualization technologies have transformed our life and work, designing information visualization systems still faces challenges. Non-expert users or end-users need toolkits that allow for rapid design and prototyping, along with supporting unified data structures suitable for different data types (e.g., tree, network, temporal, and multi-dimensional data), various visualization, interaction tasks. To address these issues, we designed DaisyViz, a model-based user interface toolkit, which enables end-users to rapidly develop domain-specific information visualization applications without traditional programming. DaisyViz is based on a user interface model for information (UIMI), which includes three declarative models: data model, visualization model, and control model. In the development process, a user first constructs a UIMI with interactive visual tools. The results of the UIMI are then parsed to generate a prototype system automatically. In this paper, we discuss the concept of UIMI, describe the architecture of DaisyViz, and show how to use DaisyViz to build an information visualization system. We also present a usability study of DaisyViz we conducted. Our findings indicate DaisyViz is an effective toolkit to help end-users build interactive information visualization systems.