A decision support system for usability evaluation of web-based information systems

In this study, a decision support system (DSS) for usability assessment and design of web-based information systems (WIS) is proposed. It employs three machine learning methods (support vector machines, neural networks, and decision trees) and a statistical technique (multiple linear regression) to reveal the underlying relationships between the overall WIS usability and its determinative factors. A sensitivity analysis on the predictive models is performed and a new metric, criticality index, is devised to identify the importance ranking of the determinative factors. Checklist items with the highest and the lowest contribution to the usability performance of the WIS are specified by means of the criticality index. The most important usability problems for the WIS are determined with the help of a pseudo-Pareto analysis. A case study through a student information system at Fatih University is carried out to validate the proposed DSS. The proposed DSS can be used to decide which usability problems to focus on so as to improve the usability and quality of WIS.     

Comparative usability evaluation

This paper reports on a study assessing the consistency of usability testing across organisations. Nine independent organisations evaluated the usability of the same website, Microsoft Hotmail. The results document a wide difference in selection and application of methodology, resources applied, and problems reported. The organizations reported 310 different usability problems. Only two problems were reported by six or more organizations, while 232 problems (75%) were uniquely reported, that is, no two teams reported the same problem. Some of the unique findings were classified as serious. Even the tasks used by most or all teams produced very different results - around 70% of the findings for each of these tasks were unique. Our main conclusion is that our simple assumption that we are all doing the same and getting the same results in a usability test is plainly wrong.     

Comparative usability evaluation

This paper reports on a study assessing the consistency of usability testing across organisations. Nine independent organisations evaluated the usability of the same website, Microsoft Hotmail. The results document a wide difference in selection and application of methodology, resources applied, and problems reported. The organizations reported 310 different usability problems. Only two problems were reported by six or more organizations, while 232 problems (75%) were uniquely reported, that is, no two teams reported the same problem. Some of the unique findings were classified as serious. Even the tasks used by most or all teams produced very different results – around 70% of the findings for each of these tasks were unique. Our main conclusion is that our simple assumption that we are all doing the same and getting the same results in a usability test is plainly wrong.     

Involving users in the design and usability evaluation of a clinical decision support system

AIM: To design and evaluate a clinical decision support system (CDSS) to support cardiovascular risk prevention in type 2 diabetes. METHODS: A preliminary requirements specification and three prototype CDSS interface designs were developed. Seven patients and seven clinicians conducted 'usability tests' on five different task scenarios with the CDSS prototypes to test its effectiveness, efficiency and 'user-friendliness'. Structured, qualitative questions explored their preferences for the different designs and overall impressions of clinical usefulness. RESULTS: Patients and clinicians were enthusiastic about the CDSS and used it confidently after a short learning period. Some patients had difficulty interpreting clinical data, but most were keen to see the CDSS used to help them understand their diabetes, provided a clinician explained their results. Clinicians' main concern was that the CDSS would increase consultation times. Changes suggested by users were incorporated into the final interface design. CONCLUSION: We have successfully incorporated patients' and clinicians' views into the design of a CDSS, but it was an arduous process.     

New techniques for usability evaluation of mobile systems

Usability evaluation of systems for mobile computers and devices is an emerging area of research. This paper presents and evaluates six techniques for evaluating the usability of mobile computer systems in laboratory settings. The purpose of these techniques is to facilitate systematic data collection in a controlled environment and support the identification of usability problems that are experienced in mobile use. The proposed techniques involve various aspects of physical motion combined with either needs for navigation in physical space or division of attention. The six techniques are evaluated through two usability experiments where walking in a pedestrian street was used as a reference. Each of the proposed techniques had some similarities to testing in the pedestrian street, but none of them turned out to be completely comparable to that form of field-evaluation. Seating the test subjects at a table supported identification of significantly more usability problems than any of the other proposed techniques. However a large number of the additional problems identified using this technique were categorized as cosmetic. When increasing the amount of physical activity, the test subjects also experienced a significantly increased subjective workload.     

Cultural cognition in usability evaluation

We discuss the impact of cultural differences on usability evaluations that are based on the thinking-aloud method (TA). The term ‘cultural differences’ helps distinguish differences in the perception and thinking of Westerners (people from Western Europe and US citizens with European origins) and Easterners (people from China and the countries heavily influenced by its culture). We illustrate the impact of cultural cognition on four central elements of TA: (1) instructions and tasks, (2) the user’s verbalizations, (3) the evaluator’s reading of the user, and (4) the overall relationship between user and evaluator. In conclusion, we point to the importance of matching the task presentation to users’ cultural background, the different effects of thinking aloud on task performance between Easterners and Westerners, the differences in nonverbal behaviour that affect usability problem detection, and, finally, the complexity of the overall relationship between a user and an evaluator with different cultural backgrounds.     

Automatic usability and stress analysis in mobile biometrics

This article focuses on the usability evaluation of biometric recognition systems in mobile devices. In particular, a behavioural modality has been used: the dynamic handwritten signature. Testing usability in behavioural modalities involves a big challenge due to the number of degrees of freedom that users have in interacting with sensors, as well as the variety of capture devices to be used. In this context we propose a usability evaluation that allows users to interact freely with the system while minimizing errors at the same time. The participants signed in a smartphone with a stylus through the different phases in the use of a biometric system: training, enrolment and verification. In addition, a profound study on the automation of the evaluation processes has been done, so as to reduce the resources employed. The influence of the users' stress has also been studied, to obtain conclusions on its impact on both the usability systems in scenarios where the user may suffer a certain level of stress, such as in courts, banks or even shopping. In brief, the results shown in this paper prove not only that a dynamic handwritten signature is a trustable solution for a large number of applications in the real world, but also that the evaluation of the usability of biometric systems can be carried out at lower costs and shorter duration.     

A comparison of usability evaluation methods for evaluating e-commerce websites

The importance of evaluating the usability of e-commerce websites is well recognised. User testing and heuristic evaluation methods are commonly used to evaluate the usability of such sites, but just how effective are these for identifying specific problems? This article describes an evaluation of these methods by comparing the number, severity and type of usability problems identified by each one. The cost of employing these methods is also considered. The findings highlight the number and severity level of 44 specific usability problem areas which were uniquely identified by either user testing or heuristic evaluation methods, common problems that were identified by both methods, and problems that were missed by each method. The results show that user testing uniquely identified major problems related to four specific areas and minor problems related to one area. Conversely, the heuristic evaluation uniquely identified minor problems in eight specific areas and major problems in three areas.     

SPI success factors within product usability evaluation

This article presents an experience report where we compare 8 years of experience of product related usability testing and evaluation with principles for software process improvement (SPI). In theory the product and the process views are often seen to be complementary, but studies of industry have demonstrated the opposite. Therefore, more empirical studies are needed to understand and improve the present situation. We find areas of close agreement as well as areas where our work illuminates new characteristics. It has been identified that successful SPI is dependent upon being successfully combined with a business orientation. Usability and business orientation also have strong connections although this has not been extensively addressed in SPI publications. Reasons for this could be that usability focuses on product metrics whilst today's SPI mainly focuses on process metrics. Also because today's SPI is dominated by striving towards a standardized, controllable, and predictable software engineering process; whilst successful usability efforts in organisations are more about creating a creative organisational culture advocating a useful product throughout the development and product life cycle. We provide a study and discussion that supports future development when combining usability and product focus with SPI, in particular if these efforts are related to usability process improvement efforts.     

Tracking the effectiveness of usability evaluation methods

Abstract

We present a case study that tracks usability problems predicted with six usability evaluation methods (claims analysis, cognitive walkthrough, GOMS, heuristic evaluation, user action notation, and simply reading the specification) through a development process. We assess the method's predictive power by comparing the predictions to the results of user tests. We assess the method's persuasive power by seeing how many problems led to changes in the implemented code. We assess design-change effectiveness by user testing the resulting new versions of the system. We concludethatpredictivemethodsarenot as effective as the HCI field would like and discuss directions for future research.     

Experience with usability evaluation of e-learning systems

Progress in the field of e-learning has been slow, with related problems mainly associated with the poor design of e-learning systems. Moreover, because of a depreciated importance of usability, usability studies are not very frequent. This paper reports the experience with the usability assessment of intelligent learning and teaching systems which are based on TEx-Sys model and are intended to enhance the process of knowledge acquisition in daily classroom settings. The applied scenario-based usability evaluation, as a combination of behaviour and opinion based measurements, enabled to quantify usability in terms of users’ (teachers’ and students’) performance and satisfaction. According to the achieved results, the main directions for interface redesign are offered. The acquired experience indicates that useful usability assessments with a significant identification of interface limitations can be performed quite easily and quickly. On the other hand, it raised a series of questions which, in order to be clarified, require further comprehensive research, the more so if the employment of universal design within e-learning context is considered.     

Analytic review of usability evaluation in ISMAR

There has been a rapid increase in research evaluating usability of Augmented Reality (AR) systems in recent years. Although many different styles of evaluation are used, there is no clear consensus on the most relevant approaches. We report a review of papers published in International Symposium of Mixed and Augmented Reality (ISMAR) proceedings in the past decade, building on the previous work of Swan and Gabbard (2005). Firstly, we investigate the evaluation goal, measurement and method of ISMAR papers according to their usability research in four categories: performance, perception and cognition, collaboration and User Experience (UX). Secondly, we consider the balance of evaluation approaches with regard to empirical–analytical, quantitative–qualitative and participant demographics. Finally we identify potential emphases for usability study of AR systems in the future. These analyses provide a reference point for current evaluation techniques, trends and challenges, which benefit researchers intending to design, conduct and interpret usability evaluations for future AR systems.     

Contrasting usability evaluation methods with blind users

Usability tests are a part of user-centered design. Usability testing with disabled people is necessary, if they are among the potential users. Several researchers have already investigated usability methods with sighted people. However, research with blind users is insufficient, for example, due to different knowledge on the use of assistive technologies and the ability to analyze usability issues from inspection of non-visual output of assistive devices. From here, the authors aspire to extend theory and practice by investigating four usability methods involving the blind, visually impaired and sighted people. These usability methods comprise of local test, synchronous remote test, tactile paper prototyping and computer-based prototyping. In terms of effectiveness of evaluation and the experience of participants and the facilitator, local tests were compared with synchronous remote tests and tactile paper prototyping with computer-based prototyping. Through the comparison of local and synchronous remote tests, it has been found that the number of usability problems uncovered in different categories with both approaches was comparable. In terms of task completion time, there is a significant difference for blind participants, but not for the visually impaired and sighted. Most of the blind and visually impaired participants prefer the local test. As for the comparison of tactile paper prototyping and computer-based prototyping, it has been revealed that tactile paper prototyping provides a better overview of an application while the interaction with computer-based prototypes is closer to reality. Problems regarding the planning and conducting of these methods as they arise in particular with blind people were also discussed. Based on the authors’ experiences, recommendations were provided for dealing with these problems from both the technical and the organization perspectives.     

Tracking the effectiveness of usability evaluation methods

. We present a case study that tracks usability problems predicted with six usability evaluation methods (claims analysis, cognitive walkthrough, GOMS, heuristic evaluation, user action notation, and simply reading the specification) through a development process. We assess the method's predictive power by comparing the predictions to the results of user tests. We assess the method's persuasive power by seeing how many problems led to changes in the implemented code. We assess design-change effectiveness by user testing the resulting new versions of the system. We concludethatpredictivemethodsarenot as effective as the HCI field would like and discuss directions for future research.     

The user action framework: a reliable foundation for usability engineering support tools

Although various methods exist for performing usability evaluation, they lack a systematic framework for guiding and structuring the assessment and reporting activities. Consequently, analysis and reporting of usability data are ad hoc and do not live up to their potential in cost effectiveness, and usability engineering support tools are not well integrated. We developed the User Action Framework, a structured knowledge base of usability concepts and issues, as a framework on which to build a broad suite of usability engineering support tools. The User Action Framework helps to guide the development of each tool and to integrate the set of tools in the practitioner's working environment. An important characteristic of the User Action Framework is its own reliability in term of consistent use by practitioners. Consistent understanding and reporting of the underlying causes of usability problems are requirements for cost-effective analysis and redesign. Thus, high reliability in terms of agreement by users on what the User Action Framework means and how it is used is essential for its role as a common foundation for the tools. Here we describe how we achieved high reliability in the User Action Framework, and we support the claim with strongly positive results of a summative reliability study conducted to measure agreement among 10 usability experts in classifying 15 different usability problems. Reliability data from the User Action Framework are also compared to data collected from nine of the same usability experts using a classic heuristic evaluation technique.     

Web系统可用性自动化评估系统的设计与实现

为了从用户真实行为中挖掘可能存在的可用性问题,避免启发式评估不可靠的问题,提出了基于大量用户行为数据,面向Web系统的可用性自动化评估系统.通过自动化匹配大量用户行为数据与需求分析阶段产生的用例,发现不一致的任务和任务前后置关系进而发现可用性问题.真实项目实践结果表明,输出的异常任务和异常任务前后置关系对可用性评估人员有指导价值,可以为进一步改进Web系统提供参考.     

Systematic usability evaluation on two harnesses for a wearable chairless exoskeleton

Harnesses are used in wearable robots in safety, sports, medical, and industrial environments. Wearable robot harnesses for lower extremities should be designed to prevent musculoskeletal disorders caused by excessive strain on the spine and lower extremities when users wear these robots for prolonged periods. Therefore, this study compares and analyzes the usability of the two harnesses—the former (old design) and the latter (new design)—for the Hyundai chairless exoskeleton (CEX). In total, 27 participants took part in this study. The wearability, stability, and overall wearing satisfaction of the former and the latter harnesses were compared. We conducted an experiment where the participants wore the former and the latter harnesses while completing a series of tasks that resembled the ones performed by the workers on a front-end module assembling line of an automobile factory. After the experiment, the participants took part in a survey and semi-structured interviews. The data was analyses by t-test, correlation, and regression analyses. The survey results demonstrated that for the latter harness the cushioning sensation (thigh r = 0.539; calf r = 0.516) had the strongest effect on the overall wearing satisfaction. In addition, the latter harness scored higher in overall wear satisfaction than the previous harness. This study was able to verify the difference between the new and old designs according to the usability evaluation of the harness. The results of this study are considered to be helpful in design to improve the usability of the harness in the future.     

A usability evaluation toolkit for In-Vehicle Information Systems (IVISs)

Usability must be defined specifically for the context of use of the particular system under investigation. This specific context of use should also be used to guide the definition of specific usability criteria and the selection of appropriate evaluation methods. There are four principles which can guide the selection of evaluation methods, relating to the information required in the evaluation, the stage at which to apply methods, the resources required and the people involved in the evaluation. This paper presents a framework for the evaluation of usability in the context of In-Vehicle Information Systems (IVISs). This framework guides designers through defining usability criteria for an evaluation, selecting appropriate evaluation methods and applying those methods. These stages form an iterative process of design–evaluation–redesign with the overall aim of improving the usability of IVISs and enhancing the driving experience, without compromising the safety of the driver.