Integration of Usability Evaluation Studies via a Novel Meta-Analytic Approach: What are Significant Attributes for Effective Evaluation?

The overall discovery rates, which are the ratios of sum of unique usability problems detected by all experiment participants against the number of usability problems existed in the evaluated systems, were investigated to find significant factors of usability evaluation through a meta-analytic approach with the n-corrected effect sizes newly defined in this study. Since many studies of usability evaluation have been conducted under specific contexts showing some mixed findings, usability practitioners need holistic and more generalized conclusions. Due to the limited applicability of the traditional meta-analysis to usability evaluation studies, a new meta-analytic approach was established and applied to 38 experiments that reported overall discovery rates of usability problems as a criterion measure. Through the meta-analytic approach with the n-corrected effect sizes, we successfully combined 38 experiments and found evaluator's expertise, report type, and interaction between usability evaluation method and time constraint as significant factors. We suggest that in order to increase overall discovery rates of usability problems, (a) free-style written reports are better than structured written reports; (b) when heuristic evaluation or cognitive walkthrough is used, the usability evaluation experiments should be conducted without time constraint, but when think-aloud is used, time constraint is not an important experimental condition; (c) usability practitioners do not need to be concerned about unit of evaluation, fidelity of evaluated systems, and task type; and (d) HCI experts are better than novice users or evaluators. Our conclusions can guide usability practitioners when determining evaluation contexts, and the meta-analytic approach of this study provides an alternative way to combine the empirical results of usability evaluation besides the traditional meta-analysis.     

Product design and apparent usability. The influence of novelty in product appearance

This research enhances our understanding of the relationship between aesthetics and usability by investigating the effects of novelty in product appearance on the apparent usability of a product. In two experimental studies using washing machines and digital cameras as stimuli, we systematically manipulated the level of novelty (low vs. high) in the product appearance by changing the product's color or shape. Participants were presented with one of these product appearances and a list of the product's technical specifications. Next, participants indicated how difficult or easy they expected the usage of the product to be. Our findings demonstrate that because people associate a high level of novelty with technological advancement, novelty in a product appearance negatively affects their expectations of a product's usability at the point of sale. Furthermore, novices are more likely to use the level of novelty as a cue for a product's apparent usability than experts.     

A usability evaluation of medical software at an expert conference setting

Introduction

A usability test was employed to evaluate two medical software applications at an expert conference setting. One software application is a medical diagnostic tool (electrocardiogram [ECG] viewer) and the other is a medical research tool (electrode misplacement simulator [EMS]). These novel applications have yet to be adopted by the healthcare domain, thus, (1) we wanted to determine the potential user acceptance of these applications and (2) we wanted to determine the feasibility of evaluating medical diagnostic and medical research software at a conference setting as opposed to the conventional laboratory setting.

Methods

The medical diagnostic tool (ECG viewer) was evaluated using seven delegates and the medical research tool (EMS) was evaluated using 17 delegates that were recruited at the 2010 International Conference on Computing in Cardiology. Each delegate/participant was required to use the software and undertake a set of predefined tasks during the session breaks at the conference. User interactions with the software were recorded using screen-recording software. The ‘think-aloud’ protocol was also used to elicit verbal feedback from the participants whilst they attempted the pre-defined tasks. Before and after each session, participants completed a pre-test and a post-test questionnaire respectively.

Results

The average duration of a usability session at the conference was 34.69 min (SD = 10.28). However, taking into account that 10 min was dedicated to the pre-test and post-test questionnaires, the average time dedication to user interaction of the medical software was 24.69 min (SD = 10.28). Given we have shown that usability data can be collected at conferences, this paper details the advantages of conference-based usability studies over the laboratory-based approach. For example, given delegates gather at one geographical location, a conference-based usability evaluation facilitates recruitment of a convenient sample of international subject experts. This would otherwise be very expensive to arrange. A conference-based approach also allows for data to be collected over a few days as opposed to months by avoiding administration duties normally involved in laboratory based approach, e.g. mailing invitation letters as part of a recruitment campaign.Following analysis of the user video recordings, 41 (previously unknown) use errors were identified in the advanced ECG viewer and 29 were identified in the EMS application. All use errors were given a consensus severity rating from two independent usability experts. Out of a rating scale of 4 (where 1 = cosmetic and 4 = critical), the average severity rating for the ECG viewer was 2.24 (SD = 1.09) and the average severity rating for the EMS application was 2.34 (SD = 0.97). We were also able to extract task completion rates and times from the video recordings to determine the effectiveness of the software applications. For example, six out of seven tasks were completed by all participants when using both applications. This statistic alone suggests both applications already have a high degree of usability. As well as extracting data from the video recordings, we were also able to extract data from the questionnaires. Using a semantic differential scale (where 1 = poor and 5 = excellent), delegates highly rated the ‘responsiveness’, ‘usefulness’, ‘learnability’ and the ‘look and feel’ of both applications.

Conclusion

This study has shown the potential user acceptance and user-friendliness of the novel EMS and the ECG viewer applications within the healthcare domain. It has also shown that both medical diagnostic software and medical research software can be evaluated for their usability at an expert conference setting. The primary advantage of a conference-based usability evaluation over a laboratory-based evaluation is the high concentration of experts at one location, which is convenient, less time consuming and less expensive.     

The influence of prototype fidelity and aesthetics of design in usability tests: effects on user behaviour, subjective evaluation and emotion

An empirical study examined the impact of prototype fidelity on user behaviour, subjective user evaluation and emotion. The independent factors of prototype fidelity (paper prototype, computer prototype, fully operational appliance) and aesthetics of design (high vs. moderate) were varied in a between-subjects design. The 60 participants of the experiment were asked to complete two typical tasks of mobile phone usage: sending a text message and suppressing a phone number. Both performance data and a number of subjective measures were recorded. The results suggested that task completion time may be overestimated when a computer prototype is being used. Furthermore, users appeared to compensate for deficiencies in aesthetic design by overrating the aesthetic qualities of reduced fidelity prototypes. Finally, user emotions were more positively affected by the operation of the more attractive mobile phone than by the less appealing one.     

The effects of prototype medium on usability testing

Inconsistencies among testing methods and results in previous research prompted this study that builds upon a systematic usability testing research framework to better understand how interface medium influences users' abilities to detect usability flaws in applications. Interface medium was tested to identify its effects on users' perceptions of usability and abilities to detect usability problems and severe usability problems. Results indicated that medium has no effect on users' abilities to detect usability problems or perceptions of usability. However, results did indicate an interaction between the medium and the tested application in which users were able to identify significantly more usability problems on a higher fidelity medium using a particular application. Results also indicated that as users' perceptions of an application's usability increases, the users are less able to detect usability problems in that application. Usability testing should begin early in the design process, even if low fidelity mediums will be used.     

Attitudinal and experiential predictors of technological expertise

This study sought to clarify which computer and Internet skills and experiences differentiate technological expertise and to identify predictors of this expertise. Two hundred and fifty-eight incoming university students were surveyed on Internet and computer experiences, skills, and attitudes. Six specific Internet and computer uses that differentiate technological expertise were identified based on frequency of use. Males and those who own computers had greater technological expertise. Factor analyses identified two skill factors (Internet skills, Computer/Math skills), three experience factors (Internet, Computer, Recreational Use of Internet), five attitude factors (Importance of Internet and computer knowledge, Computer aversive, Internet and computer comfort/competency, Internet and computer overuse, Technology aversive). These factors as well as gender, computer ownership, and weekly computer and Internet use were entered as independent variables in a general linear model (GLM) which was then used to determine which factors affected technological expertise. The overall GLM was significant (R² = .414, F = 5.85, p < .001). Internet and computer skills and computer abuse were the strongest predictors of technological expertise. Weekly Internet use, Internet and computer comfort/competency also were predictors, and gender was almost significant (p = .056) as a predictor. Neither computer ownership nor other Internet and computer attitudes were significant predictors.     

Expert vs novice collaborative heuristic evaluation (CHE) of a smartphone app for cultural heritage sites

This study was conducted to compare CHE between Human-Computer Interaction (HCI) experts and novices in evaluating the Smartphone app for the cultural heritage site. It uses the Smartphone Mobile Application heuRisTics (SMART), focusing on smartphone applications and traditional Nielsen heuristics, focusing on a wider range of interactive systems. Six experts and six novices used the severity rating scale to categorise the severity of the usability issues. These issues were mapped to both heuristics. The study found that experts and novice evaluators identified 19 and 14 usability issues, respectively, with ten as the same usability issues. However, these same usability issues have been rated differently. Although the t-test indicates no significant differences between experts and novices in their ratings for usability issues, these results nevertheless indicate the need for both evaluators in CHE to provide a more comprehensive perspective on the severity of the usability issues. Furthermore, the mapping of the usability issues for Nielsen and SMART heuristics concluded that more issues with the smartphone app could be addressed through smartphone-specific heuristics than general heuristics, indicating a better tool for heuristic evaluation of the smartphone app. This study also provides new insight into the required number of evaluators needed for CHE.

     

Utility of the RT3 triaxial accelerometer in free living: An investigation of adherence and data loss

There is strong evidence for the protective effects of physical activity on chronic health problems. Activity monitors can objectively measure free living occupational and leisure time physical activity. Utility is an important consideration when determining the most appropriate monitor for specific populations and environments. Hours of activity data collected, the reasons for activity hours not being recorded, and how these two factors might change over time when using an activity monitor in free living are rarely reported. This study investigated user perceptions, adherence to minimal wear time and loss of data when using the RT3 activity monitor in 21 healthy adults, in a variety of occupations, over three (7 day) repeated weeks of measurement in free living. An activity diary verified each day of monitoring and a utility questionnaire explored participant perceptions on the usability of the RT3. The RT3 was worn for an average of 14 h daily with 90% of participants having complete data sets. In total 6535.8 and 6092.5 h of activity data were collected from the activity diary and the RT3 respectively. An estimated 443.3 h (6.7%) of activity data were not recorded by the RT3. Data loss was primarily due to battery malfunction (45.2%). Non-adherence to wear time accounted for 169.5 h (38.2%) of data loss, of which 14 h were due to occupational factors. The RT3 demonstrates good utility for free living activity measurement, however, technical issues and strategies to manage participant adherence require consideration with longitudinal and repeated measures studies.     

Some surprising differences between novice and expert errors in computerized office work

Abstract

This paper investigates the impact of different levels of expertise on errors in human-computer interaction. In a field study 174 clerical workers from 12 different companies were observed during their normal office work and were questioned on their expertise with computers. The level of expertise was determined by (a) the length of time an employee had worked with a computer (computer expertise); (b) the number of programs she knew (program expertise); and (c) the daily time s/he spent working with the computer (daily work-time expertise). These different operationalizations of novices and experts led to different results. In contrast to widespread assumptions, experts did not make fewer errors than novices (except in knowledge errors). On the other hand, experts spent less time handling the errors than novices. A cluster analysis produced four groups in the workforce: occasional users, frequent users, beginners, and general users.     

Interacting with notebook input devices: an analysis of motor performance and users' expertise

In the present study the usability of two different types of notebook input devices was examined. The independent variables were input device (touchpad vs. mini-joystick) and user expertise (expert vs. novice state). There were 30 participants, of whom 15 were touchpad experts and the other 15 were mini-joystick experts. The experimental tasks were a point-click task (Experiment 1) and a point-drag-drop task (Experiment 2). Dependent variables were the time and accuracy of cursor control. To assess carryover effects, we had the participants complete both experiments, using not only the input device for which they were experts but also the device for which they were novices. Results showed the touchpad performance to be clearly superior to mini-joystick performance. Overall, experts showed better performance than did novices. The significant interaction of input device and expertise showed that the use of an unknown device is difficult, but only for touchpad experts, who were remarkably slower and less accurate when using a mini-joystick. Actual and potential applications of this research include an evaluation of current notebook input devices. The outcomes allow ergonomic guidelines to be derived for optimized usage and design of the mini-joystick and touchpad devices.     

Wireless Face Interface: Using voluntary gaze direction and facial muscle activations for human-computer interaction

The present aim was to investigate the functionality of a new wireless prototype called Face Interface. The prototype combines the use of voluntary gaze direction and facial muscle activations, for pointing and selecting objects on a computer screen, respectively. The subjective and objective functionality of the prototype was evaluated with a series of pointing tasks using either frowning (i.e., frowning technique) or raising the eyebrows (i.e., raising technique) as the selection technique. Pointing task times and accuracies were measured using three target diameters (i.e., 25, 30, 40 mm), seven pointing distances (i.e., 60, 120, 180, 240, 260, 450, and 520 mm), and eight pointing angles (0°, 45°, 90°, 135°, 180°, 225°, 270°, and 315°). The results showed that the raising technique was faster selection technique than the frowning technique for the objects that were presented in the pointing distances from 60 mm to 260 mm. For those pointing distances the overall pointing task times were 2.4 s for the frowning technique, and 1.6 s for the raising technique. Fitts’ law computations showed that the correlations for the Fitts’ law model were r = 0.77 for the frowning technique and r = 0.51 for the raising technique. Further, the index of performance (IP) value was 1.9 bits/s for the frowning technique and 5.4 bits/s for raising the eyebrows technique. Based on the results, the prototype functioned well and was adjustable so that two different facial activations can be used in combination with gaze direction for pointing and selecting objects on a computer screen.     

Affective impact of navigational and signaling aids to e-learning

College students had 30 min to study a 17-frame online lesson on distance learning that included navigational aids (for showing the learner’s location in the lesson), signaling aids (for highlighting the important content), both aids, or no aids. On a 30-item usability survey consisting of 8 usability scales, students who received navigational aids produced significantly higher mean ratings on each of the 8 usability scales—ease of use, satisfaction of use, awareness of lesson structure, awareness of lesson length, awareness of location, ease of navigation, lesson comprehension, and lesson learning—with effect sizes ranging from d = 0.50 to d = 1.35. Students who received signaling aids produced significantly higher ratings on 4 of the 8 usability scales—ease of use, satisfaction of use, lesson comprehension, and lesson learning with effect sizes ranging from d = 0.39 to d = 2.15. Results help to clarify the mechanism underlying previous findings showing that students learned more from e-lessons that contained navigational aids. In the present study, there was a significant positive correlation between usability rating and recall test score for 5 of the 8 usability scales (particularly for ease of use), indicating partial support for the prediction that learners’ satisfaction with an e-learning system is related to their learning outcome. Results support the predictions of the emotional design hypothesis and have implications for the design of e-learning interfaces.     

Some surprising differences between novice and expert errors in computerized office work

This paper investigates the impact of different levels of expertise on errors in human-computer interaction. In a field study 174 clerical workers from 12 different companies were observed during their normal office work and were questioned on their expertise with computers. The level of expertise was determined by (a) the length of time an employee had worked with a computer (computer expertise); (b) the number of programs she knew (program expertise); and (c) the daily time s/he spent working with the computer (daily work-time expertise). These different operationalizations of novices and experts led to different results. In contrast to widespread assumptions, experts did not make fewer errors than novices (except in knowledge errors). On the other hand, experts spent less time handling the errors than novices. A cluster analysis produced four groups in the workforce: occasional users, frequent users, beginners, and general users.     

Intensive use of ICT in school: Developing differences in students’ ICT expertise

The purpose of the study was to examine the development of students’ high-level computer skills and competence (student expertise) in information and communication technology (ICT), and to examine the characteristics of such expertise. Eighteen lower secondary school students, selected to represent both genders and all school achievement levels, were given laptops for three years to be used both at school and at home. The data of the longitudinal study consisted of a test on ICT skills; self-evaluation questionnaires on ICT competence, thoughts about their own expertise, ICT-related activities and task, interest in ICT, and classroom observations. The results of the study indicated that the intensive use of ICT and the process-oriented learning environment supported the development of student expertise. In the analysis, three groups were identified, which oriented themselves somewhat differently in relation to ICT: student experts (n = 6), advanced users (n = 8) and non-interested users (n = 4). The experts differed from the other groups in respect of especially focusing on advancing their ICT skills; they had undertaken ICT-related tasks outside the school, they had metacognitive consciousness about their competence, and they had future plans concerning ICT in their further education and profession.     

Understanding internet recruitment via signaling theory and the elaboration likelihood model

A detailed model specifying the linkages between Internet recruitment websites and organizational attraction was examined. Participants (N = 581) viewed Fortune 500 company websites and responded to questions about the content and design of these websites and their resulting attitudes, fit perceptions, and organizational attraction. Results showed that recruitment website content and design influence attitudes toward the recruitment websites, organizational attitudes, and subsequently organizational attraction. The moderating effects of person-organization (P-O) and person-job (P-J) fit were examined. Two sets of hypotheses based on signaling theory (  and ) and the elaboration likelihood model ( Petty & Cacioppo, 1981) were largely supported. Consistent with signaling theory, the amount of job and organizational information on a recruitment website interacted with website usability, such that when less job information was presented, website usability played a greater role in predicting favorable attitudes towards the organization. Consistent with the elaboration likelihood model, when P-J fit was high, website aesthetics were less important in predicting attitudes towards the organization.     

Evaluation of the MODIS (MOD10A1) daily snow albedo product over the Greenland ice sheet

This study evaluates the performance of the beta-test MODIS (MOD10A1) daily albedo product using in situ data collected in Greenland during summer 2004. Results indicate the beta-test product tracks the general seasonal variability in albedo but exhibits significant more temporal variability than observed at the stations. This may indicate problems with the cloud detection algorithm, and/or failure of the BRDF model to adequately model the bidirectional reflectance of snow. Comparisons with in situ observations at five automatic weather stations in Greenland indicate an overall RMSE of 0.067 for the Terra instrument and an RMSE of 0.075 on Aqua. The Terra-retrieved-albedo are slightly better correlated with the in situ data than the Aqua retrievals (r = 0.79 versus r = 0.77). Comparisons were also made between the MODIS daily albedo product and the MODIS 16-day albedo product (MOD43B3). Results indicate general correspondence between the two products, with better agreement found using the Terra-retrieved-albedo than the Aqua-retrieved albedo. The reason for the differences in albedo between the Aqua and Terra satellites remains unclear. At the stations examined, both the Terra and Aqua retrievals were made at nearly the same time of the day and therefore the differences in albedo between the satellites cannot be explained by differences in solar illumination. Finally, the albedo derived using MODIS data and the direct estimation algorithm (DEA) was also compared with 2004 Greenland in situ data. Results from this comparison suggest that the DEA performs well as long as the solar zenith angle of the observation is not greater than 70°.     

In-store consumer behavior: How mobile recommendation agents influence usage intentions,product purchases,and store preferences

Product information given in purchase situations influences purchase behavior. In online purchase situations, the use of recommendation agents increases the value of product information as information becomes adaptive and thus more relevant to consumers’ information needs. Correspondingly, mobile recommendation agents (MRAs) may also increase the value of product information in bricks-and-mortar stores. In this sense, product information is not only adaptive but can also be requested at any place such as in front of products consumers are interested in. Because unprecedented, we investigate the use of a MRA that is virtually bound to a physical product via an RFID-enabled mobile device and provides product information. Based on Theory of Planned Behavior, Innovation Diffusion Theory, and Technology Acceptance Model, we develop a model to better understand the impact of MRAs on usage intentions, product purchases and store preferences of consumers. This model is then tested in a lab experiment (n = 47). Among high usability scores, results indicate that perceived usefulness of a MRA influences product purchases, predicts usage intentions and store preferences of consumers. Thus, new business models for retail stores can be considered in which MRAs satisfy both the information needs of consumers and the communication needs of retailers.     

A portable device for the assessment of dynamic visual acuity

Dynamic visual acuity (DVA) thresholds are among the few visual functions predictive of automobile crashes. DVA is also sensitive to alcohol and aging. However, measuring DVA is awkward because there is no standardized, efficient, flexible apparatus for DVA assessment. In this project, we developed a prototype of an automated, portable DVA system using a low-energy laser, and we compared this laser DVA with the traditional device in two within-subjects, repeated measures designs. The two studies included 48 participants (22 males and 26 females with an average age of 18.33 years). The most important findings were that: (1) retest reliabilities of the two DVA devices were comparable and higher with the laser; (2) average correlations between the two devices were r = 0.62 (p < 0.01) and r = 0.65 (p < 0.01) for the two designs respectively; and (3) after correction for reliability attenuation these improved to r = 0.92 and r = 0.78. These findings indicate that a flexible DVA laser device can be developed to measure the same construct as the more traditional bulky DVA device.     

Beyond the User Preferences: Aligning the Prototype Design to the Users’ Expectations

It is important for practitioners to conceptualize and tailor a prototype in tune with the users’ expectations in the early stages of the design life cycle so the modifications of the product design in advanced phases are kept to a minimum. According to user preference studies, the aesthetic and the usability of a system play an important role in the user appraisal and selection of a product. However, user preferences are just a part of the equation. The fact that a user prefers one product over the other does not mean that he or she would necessarily buy it. To understand the factors affecting the user's assessment of a product before the actual use of the product and the user's intention to purchase the product we conducted a study, reported in this article. Our study, a modification of a well‐known protocol, considers the users’ preferences of six simulated smartphones each with different combination of attributes. A sample consisting of 365 participants was involved in our analysis. Our results confirm that the main basis for the users’ pre‐use preferences is the aesthetics of the product, whereas our results suggest that the main basis for the user's intention to purchase are the expected usability of the product. Moreover, our analysis reveals that the personal characteristics of the users have different effects on both the users’ preferences and their intention to purchase a product. These results suggest that the designers should carefully balance the aesthetics and usability features of a prototype in tune with the users expectations. If the conceptualization of a product is done properly the redesign cycles after the usability testing can be reduced and speed up the process for releasing the product on the market.