Evaluating expert systems

Abstract: Evaluation is crucial for improving expert system design and performance. This paper stresses the need for considering system evaluation throughout the development process. It highlights the importance of evaluating system usability and discusses key usability issues. A number of basic evaluation methods are described, including interviews, questionnaires, observation, system logging, user diaries, laboratory experiments and field trials. Finally, the paper looks at evaluating systems within organisations, and assessing other long term effects of expert systems.     

User-adapted travel planning system for personalized schedule recommendation

Recently, the Internet has made a lot of services and products appear online provided by many tourism sectors. By this way, many information such as timetables, routes, accommodations, and restaurants are easily available to help travelers plan their travels. However, how to plan the most appropriate travel schedule under simultaneously considering several factors such as tourist attractions visiting, local hotels selecting, and travel budget calculation is a challenge. This gives rise to our interest in exploring the recommendation systems with relation to schedule recommendation. Additionally, the personalized concept is not implemented completely in most of travel recommendation systems. One notable problem is that they simply recommended the most popular travel routes or projects, and cannot plan the travel schedule. Moreover, the existing travel planning systems have limits in their capabilities to adapt to the changes based on users’ requirements and planning results. To tackle these problems, we develop a personalized travel planning system that simultaneously considers all categories of user requirements and provides users with a travel schedule planning service that approximates automation. A novel travel schedule planning algorithm is embedded to plan travel schedules based on users’ need. Through the user-adapted interface and adjustable results design, users can replace any unsatisfied travel unit to specific one. The feedback mechanism provides a better accuracy rate for next travel schedule to new users. An experiment was conducted to examine the satisfaction and use intention of the system. The results showed that participants who used the system with schedule planning have statistical significant on user satisfaction and use intention. We also analyzed the validity of applying the proposed algorithm to a user preference travel schedule through a number of practical system tests. In addition, comparing with other travel recommendation systems, our system had better performance on the schedule adjustment, personalization, and feedback giving.     

Development and evaluation of the problem identification picture cards method

In this paper the development and assessment of a new formative evaluation method called the problem identification picture cards (PIPC) method is described. This method enables young children to express both usability and fun problems while playing a computer game. The method combines the traditional thinking-aloud method with picture cards that children can place in a box to indicate that there is a certain type of problem. An experiment to assess this method shows that children may express more problems (verbally, or with a picture card, or with a combination of a picture card and a verbalisation) with the PIPC method than without this method (in which they can only indicate problems verbally). Children in the experiment did not just replace verbalisations by using the provided picture cards and some children preferred to use the PIPC method during the test instead of the standard thinking-aloud method. The PIPC method or some aspects of the method could be a good instrument to increase the amount of information expressed by young children during an evaluation.     

DEEP: Design-Oriented Evaluation of Perceived Usability

Existing perceived usability questionnaires detect the appearance of usability issues rather than the underlying design generating those issues. This limits the capability of existing instruments to directly inform design recommendations. To address this problem, a usability questionnaire structured around the analytical composition of the design was created and validated. A four-stage process was followed. First, 3 usability experts refined 54 questions from highly cited usability questionnaires and structured them around 6 design dimensions. Second, 12 raters scored the questions by their relevance to assess usability. Third, questions and dimensions were then improved through exploratory and confirmatory factor analysis (N?=?196) and, fourth, further enhanced through confirmatory factor analysis (N?=?362). The result is DEEP, a 19-question usability questionnaire based on 5 main design dimensions (content, information architecture, navigation, layout, and visual guidance). DEEP can be used to capture detailed usability feedback that more directly relates to specific aspects of design requirements.     

Perspective-based Usability Inspection: An Empirical Validation of Efficacy

Inspection is a fundamental means of achieving software usability. Past research showed that the current usability inspection techniques were rather ineffective. We developed perspective-based usability inspection, which divides the large variety of usability issues along different perspectives and focuses each inspection session on one perspective. We conducted a controlled experiment to study its effectiveness, using a post-test only control group experimental design, with 24 professionals as subjects. The control group used heuristic evaluation, which is the most popular technique for usability inspection. The experimental design and the results are presented, which show that inspectors applying perspective-based inspection not only found more usability problems related to their assigned perspectives, but also found more overall problems. Perspective-based inspection was shown to be more effective for the aggregated results of multiple inspectors, finding about 30% more usability problems for 3 inspectors. A management implication of this study is that assigning inspectors more specific responsibilities leads to higher performance. Internal and external threats to validity are discussed to help better interpret the results and to guide future empirical studies.     

Program transformations for information personalization

Personalization constitutes the mechanisms necessary to automatically customize information content, structure, and presentation to the end-user to reduce information overload. Unlike traditional approaches to personalization, the central theme of our approach is to model a website as a program and conduct website transformation for personalization by program transformation (e.g., partial evaluation, program slicing). The goal of this paper is study personalization through a program transformation lens, and develop a formal model, based on program transformations, for personalized interaction with hierarchical hypermedia. The specific research issues addressed involve identifying and developing program representations and transformations suitable for classes of hierarchical hypermedia, and providing supplemental interactions for improving the personalized experience. The primary form of personalization discussed is out-of-turn interaction—a technique which empowers a user navigating a hierarchical website to postpone clicking on any of the hyperlinks presented on the current page and, instead, communicate the label of a hyperlink nested deeper in the hierarchy. When the user supplies out-of-turn input we personalize the hierarchy to reflect the user's informational need. While viewing a website as a program and site transformation as program transformation is non-traditional, it offers a new way of thinking about personalized interaction, especially with hierarchical hypermedia. Our use of program transformations casts personalization in a formal setting and provides a systematic and implementation-neutral approach to designing systems. Moreover, this approach helped connect our work to human-computer dialog management and, in particular, mixed-initiative interaction. Putting personalized web interaction on a fundamentally different landscape gave birth to this new line of research. Relating concepts in the web domain (e.g., sites, interactions) to notions in the program-theoretic domain (e.g., programs, transformations) constitutes the creativity in this work.     

An empirical investigation of two electronic mail systems

Two electronic mail systems were studied within an organization: an executive system for upper management and a general system for all employees. The results from 130 interviews indicated that both systems were perceived to contribute to productivity, a reduction in use of memos and a reduction in telephone calls. However, the executive system was perceived to be more satisfactory and more effective than the general system. The ratio of positive comments to negative comments was used as an independent measure of satisfaction with each electronic mail system. The user comments were also classified and discussed in terms of types of perceived advantages and problems with these systems.     

Utilizing contextual ontological user profiles for personalized recommendations

As users may have different needs in different situations and contexts, it is increasingly important to consider user context data when filtering information. In the field of web personalization and recommender systems, most of the studies have focused on the process of modelling user profiles and the personalization process in order to provide personalized services to the user, but not on contextualized services. Rather limited attention has been paid to investigate how to discover, model, exploit and integrate context information in personalization systems in a generic way. In this paper, we aim at providing a novel model to build, exploit and integrate context information with a web personalization system. A context-aware personalization system (CAPS) is developed which is able to model and build contextual and personalized ontological user profiles based on the user’s interests and context information. These profiles are then exploited in order to infer and provide contextual recommendations to users. The methods and system developed are evaluated through a user study which shows that considering context information in web personalization systems can provide more effective personalization services and offer better recommendations to users.     

Usability inspection of digital libraries: a case study

This paper reports a case study about lessons learned and usability issues encountered in a usability inspection of a digital library system called the Networked Computer Science Technical Reference Library (NCSTRL). Using a co-discovery technique with a team of three expert usability inspectors (the authors), we performed a usability inspection driven by a broad set of anticipated user tasks. We found many good design features in NCSTRL, but the primary result of a usability inspection is a list of usability problems as candidates for fixing. The resulting problems are organized by usability problem type and by system functionality, with emphasis on the details of problems specific to digital library functions. The resulting usability problem list was used to illustrate a cost/importance analysis technique that trades off importance to fix against cost to fix. The problems are sorted by the ratio of importance to cost, producing a priority ranking for resolution.     

Exploring the Value of Usability Feedback Formats

The format used to present feedback from usability evaluations to developers affects whether problems are understood, accepted, and fixed. Yet, little research has investigated which formats are the most effective. We describe an explorative study where three developers assess 40 usability findings presented using five feedback formats. Our usability findings comprise 35 problems and 5 positive comments. Data suggest that feedback serves multiple purposes. Initially, feedback must convince developers about the relevance of a problem and convey an understanding of this. Feedback must next be easy to use and finally serve as a reminder of the problem. Prior to working with the feedback, developers rated redesign proposals, multimedia reports, and annotated screen dumps as more valuable than lists of problems, all of which were rated as more valuable than scenarios. After having spent some time working with the feedback to address the usability problems, there were no significant differences among the developers' ratings of the value of the different formats. This suggests that all of the formats may serve equally well as reminders in later stages of working with usability problems, but that redesign proposals, multimedia reports, and annotated screen dumps best address the initial feedback goals convincing developers that a usability problem exists and of conveying an understanding of the problem.     

Why usability gets lost or usability in in-house software development

This study tries to shed some light on what happens to usability and occupational health issues in a bespoke software development project. Usability is an essential quality in software, in particular in a work context where poor usability and other risk factors related to the software and computers may cause health problems. We have interviewed a number of software developers, usability people and users about their attitudes to and practices for integrating usability and users' health concerns in software development. The interviews were conducted in two Swedish organisations with in-house development of bespoke software. Our main conclusion is that several factors combine to push usability and occupational health matters aside, some of which are attitudes to usability and users' health issues, unclear responsibilities, poor support for user-centeredness and usability in software development models, ineffective user participation and usability and users' health being ignored or forgotten in decisions about the software, its use and its design.     

基于多级客户模型的个性化推荐机制

个性化是未来Web智能系统的一大特征．为了实现商品的个性化推荐，提出了一种新的基于多级客户模型的推荐系统机制，它由数据准备、模型学习、推荐集的生成和智能过滤四个子过程构成．该机制借助于多级客户模型从客户的购物需求、偏爱特征和消费能力三方面捕获客户的实际需求，从而实现了一种深层次的个性化推荐，改善了推荐效果．     

Personalized e-learning system using Item Response Theory

Personalized service is important on the Internet, especially in Web-based learning. Generally, most personalized systems consider learner preferences, interests, and browsing behaviors in providing personalized services. However, learner ability usually is neglected as an important factor in implementing personalization mechanisms. Besides, too many hyperlink structures in Web-based learning systems place a large information burden on learners. Consequently, in Web-based learning, disorientation (losing in hyperspace), cognitive overload, lack of an adaptive mechanism, and information overload are the main research issues. This study proposes a personalized e-learning system based on Item Response Theory (PEL-IRT) which considers both course material difficulty and learner ability to provide individual learning paths for learners. The item characteristic function proposed by Rasch with a single difficulty parameter is used to model the course materials. To obtain more precise estimation of learner ability, the maximum likelihood estimation (MLE) is applied to estimate learner ability based on explicit learner feedback. Moreover, to determine an appropriate level of difficulty parameter for the course materials, this study also proposes a collaborative voting approach for adjusting course material difficulty. Experiment results show that applying Item Response Theory (IRT) to Web-based learning can achieve personalized learning and help learners to learn more effectively and efficiently.     

The Bitwise Hashing Trick for Personalized Search

Many real world problems require fast and efficient lexical comparison of large numbers of short text strings. Search personalization is one such domain. We introduce the use of feature bit vectors using the hashing trick for improving relevance in personalized search and other personalization applications. We present results of several lexical hashing and comparison methods. These methods are applied to a user’s historical behavior and are used to predict future behavior. Using a single bit per dimension instead of floating point results in an order of magnitude decrease in data structure size, while preserving or even improving quality. We use real data to simulate a search personalization task. A simple method for combining bit vectors demonstrates an order of magnitude improvement in compute time on the task with only a small decrease in accuracy.     

A qualitative study of smartwatch usage and its usability

This study aims to explore usability issues of watch‐type wearable devices and to suggest guidelines for improved operation of smartwatches. To do so, we conducted a series of surveys, interviews, and task performance experiments. Thirty smartwatch users from ages 20 to 43 years were recruited. Users’ experiences of smartwatches were collected via a weeklong, online‐based diary study, which consisted of various tasks to be completed while smartwatches were in use. Our study assessed usability problems associated with those tasks, concurrent tasks conducted while interacting with smartwatches, pain points/discomfort that users had while interacting with their devices, and requirements/requests of the smartwatch users. During the week of tracking, participants were asked to complete the usability evaluation three times a day using usability principles we designed for the study: information display, control, learnability, interoperability, and preference. In addition, task performance tests were conducted for the tasks most frequently conducted on touch‐based displays: number entry, swiping, and scrolling. Specific usability issues of smartwatches were identified and summarized for each usability principle by triangulating survey, interview, and task performance evaluation results. Based on the insights from the results of the study, we conclude by suggesting guidelines for further enhancing users’ experience of future smartwatches.     

When does web-based personalization really work? The distinction between actual personalization and perceived personalization

A number of prior studies have suggested that personalization is more efficacious than non-personalization. However, the existing literature is somewhat ambiguous on whether the test of personalization effects should be based on a message sender's actual personalization process or a message recipient's perception of the message. It is argued in this article that an actual personalization process does not automatically yield more favorable effects because people's perceptions of personalized messages tend to be biased. Through three experiments, it is demonstrated that testing personalization effects based on a message sender's actual personalization process can be problematic and produce misleading results. Specifically, a personalized message can be perceived as non-personalized and a non-personalized message can be perceived as personalized. The key finding is that perceived personalization, instead of actual personalization, is the underlying psychological mechanism of message effectiveness. A message will show superior effects when it is perceived to be personalized by a message recipient, regardless of whether it is actually personalized or not.     

Current practice in measuring usability: Challenges to usability studies and research

How to measure usability is an important question in HCI research and user interface evaluation. We review current practice in measuring usability by categorizing and discussing usability measures from 180 studies published in core HCI journals and proceedings. The discussion distinguish several problems with the measures, including whether they actually measure usability, if they cover usability broadly, how they are reasoned about, and if they meet recommendations on how to measure usability. In many studies, the choice of and reasoning about usability measures fall short of a valid and reliable account of usability as quality-in-use of the user interface being studied. Based on the review, we discuss challenges for studies of usability and for research into how to measure usability. The challenges are to distinguish and empirically compare subjective and objective measures of usability; to focus on developing and employing measures of learning and retention; to study long-term use and usability; to extend measures of satisfaction beyond post-use questionnaires; to validate and standardize the host of subjective satisfaction questionnaires used; to study correlations between usability measures as a means for validation; and to use both micro and macro tasks and corresponding measures of usability. In conclusion, we argue that increased attention to the problems identified and challenges discussed may strengthen studies of usability and usability research.     

Usability laboratories at Philips: supporting research, development, and design for consumer and professional products

This paper describes two of the usability laboratories at Philips, discusses practical issues arising from our experience using the facilities, gives an example of a typical usability evaluation, and briefly outlines our vision for the future of the laboratories. Usability tests at Philips can involve any product from a portfolio ranging from Compact Disc Interactive (CD-I) to electron microscopes. Performing usability tests for consumer electronic products poses a number of specific problems: our user group is broad and diverse, the context in which our products are used is highly variable, and it is difficult to determine the importance of usability relative to other design goals. In the further development of our facilities, the efficient planning and the execution of usability test is of particular concern since we are driven by demanding time schedules. In the future, we expect a shift in focus towards testing more products and product concepts in their actual context of use.     

IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use

This article describes recent research in subjective usability measurement at IBM, focused on evaluating the psychometric properties of questionnaires designed for use in scenario‐based usability evaluation. The questionnaires address evaluation at both a global overall system level and at a more detailed scenario level. The primary goals of this article are to (a) discuss the psychometric characteristics of IBM questionnaires that measure user satisfaction with computer system usability, and (b) provide the questionnaires, with administration and scoring instructions. For scenario‐level measurement, the 3‐item After‐Scenario Questionnaire (ASQ) has excellent internal consistency, with coefficient alphas across a set of scenarios ranging from .90 to .96. For more global assessment, the Post‐Study System Usability Questionnaire (PSSUQ) also has excellent internal consistency, with an overall coefficient alpha of .97. Preliminary principal factor analysis of 48 PSSUQ questionnaires suggested the presence of three factors named, after varimax rotation, System Usefulness, Information Quality, and Interface Quality, with corresponding coefficient alphas of .96, .91, and .91. Evaluation of 377 PSSUQ questionnaires (modified to allow mailing to respondents in their offices and referred to as the Computer System Usability Questionnaire, or CSUQ) confirmed the structure of the preliminary principal factor analysis. Consequently, usability practitioners can use these questionnaires to help them measure users’ satisfaction with the usability of computer systems in the context of scenario‐based usability studies.     

Usability laboratories at Philips: supporting research,development, and design for consumer and professional products

Abstract

This paper describes two of the usability laboratories at Philips, discusses practical issues arising from our experience using the facilities, gives an example of a typical usability evaluation, and briefly outlines our vision for the future of the laboratories. Usability tests at Philips can involve any product from a portfolio ranging from Compact Disc Interactive (CD-I) to electron microscopes. Performing usability tests for consumer electronic products poses a number of specific problems: our user group is broad and diverse, the context in which our products are used is highly variable, and it is difficult to determine the importance of usability relative to other design goals. In the further development of our facilities, the efficient planning and the execution of usability test is of particular concern since we are driven by demanding time schedules. In the future, we expect a shift in focus towards testing more products and product concepts in their actual context of use.