An approach to usability evaluation of e-learning applications

Despite recent advances of electronic technologies in e-learning, a consolidated evaluation methodology for e-learning applications is not available. The evaluation of educational software must consider its usability and more in general its accessibility, as well as its didactic effectiveness. This work is a first step towards the definition of a methodology for evaluating e-learning applications. Specific usability attributes capturing the peculiar features of these applications are identified. A preliminary user study involving a group of e-students, observed during their interaction with an e-learning application in a real situation, is reported. Then, the proposal is put forward to adapt to the e-learning domain a methodology for systematic usability evaluation, called SUE. Specifically, evaluation patterns are proposed that are able to drive the evaluators in the analysis of an e-learning application.     

Exploring user satisfaction for e-learning systems via usage-based metrics and system usability scale analysis

The use of e-learning technology is incontestably recognized as an important and integral part of the educational process. Considerable research studies are carried out in order to apprehend how effective and usable e-learning systems. In this paper, an empirical-based study is conducted to explore how lecturers interact with an e-learning environment based on a predefined task model describing low-level interactions. Client-side log data is collected from university lecturers from the Electrical and Computer Science departments. Subsequently, data analysis is conducted to infer the usability degree from the estimated usage metrics together with further exploratory analysis from user feedback via System Usability Scale. Experimental results reveal that the System Usability Scale score is not a sufficient measure to express the true acceptance and satisfaction level of lecturers for using the e-learning systems. The evaluation must be fulfilled in tandem with analyzing the usage metrics derived from interaction traces in a non-intrusive fashion. The proposed approach is a milestone towards usability evaluation to improve the acceptance and user experience for academic staff and students.     

Usability, quality, value and e-learning continuance decisions

Previous research suggests that an eventual information technology (IT) success depend on both its initial adoption (acceptance) and subsequent continued usage (continuance). Expectancy disconfirmation theory (EDT) has been successfully used to predict users’ intention to continue using information technologies. This study proposed a decomposed EDT model to examine cognitive beliefs and affect that influence users’ continuance decision in the context of e-learning service. The proposed model extended EDT by decomposing the perceived performance component into usability, quality, and value. Research hypotheses derived from this model are empirically validated using the responses to a survey on e-learning usage among 183 users. The results suggest that users’ continuance intention is determined by satisfaction, which in turn is jointly determined by perceived usability, perceived quality, perceived value, and usability disconfirmation.     

Experience with usability evaluation of e-learning systems

Progress in the field of e-learning has been slow, with related problems mainly associated with the poor design of e-learning systems. Moreover, because of a depreciated importance of usability, usability studies are not very frequent. This paper reports the experience with the usability assessment of intelligent learning and teaching systems which are based on TEx-Sys model and are intended to enhance the process of knowledge acquisition in daily classroom settings. The applied scenario-based usability evaluation, as a combination of behaviour and opinion based measurements, enabled to quantify usability in terms of users’ (teachers’ and students’) performance and satisfaction. According to the achieved results, the main directions for interface redesign are offered. The acquired experience indicates that useful usability assessments with a significant identification of interface limitations can be performed quite easily and quickly. On the other hand, it raised a series of questions which, in order to be clarified, require further comprehensive research, the more so if the employment of universal design within e-learning context is considered.     

A proposal toward the development of accessible e-learning content by human involvement

Most of the existing efforts for supporting the design, preparation, and deployment of accessible e-learning applications propose guidelines that primarily address technical accessibility issues. Little, if any, consideration is given to the real actors involved in the learning experience, such as didactical experts and disabled learners. Moreover, implementing artifacts addressed to the e-learning world requires a wide range of particular skills which are related not only to technical but also to didactical, pedagogical, usability, and accessibility aspects of the produced material. This paper argues that the know-how of a number of stakeholders should be blended into a joint design activity, and that it should be possible to determine the role of each participant in the successive phases of the development lifecycle of e-learning applications. The paper sketches the methodological guidelines of a design framework based on involving the users with disabilities, as well as pedagogical experts, in the development process. The novelty of this proposal mainly stems from being built up around the core of strategies and choices specifically bound to accessibility requirements. Characteristic elements of learner-centered design are then further integrated into processes and methodologies which are typical of participatory and contextual design approaches. Following such guidelines, it will be possible to gain a deeper understanding of the requirements and of the operational context of people needing accessible material, either as learners or educators. The underlying objective is to increase the potential to realize learning systems that better meet different user needs and that provide a more satisfying learning experience. Moreover, when people get involved in the development process, they gain a sense of ownership of the system and are therefore more likely to accept and “promote” it.     

Affective responses to system messages in human–computer-interaction: Effects of modality and message type

Affective responses of users to system messages in human–computer interaction are a key to study user satisfaction. However, little is known about the particular affective patterns elicited by various types of system messages. In this experimental study we examined if and how different system messages, presented in different modalities, influence users’ affective responses. Three types of messages, input requests, status notifications, and error messages, were presented either as text or speech, and either alone or in combination with icons or sounds, while users worked on several typical computer tasks. Affective responses following system messages were assessed employing a multi-modal approach, using subjective rating scales as well as physiological measures. Results show that affective responses vary systematically depending on the type of message, and that spoken messages generally elicit more positive affect than written messages. Implications on how to enhance user satisfaction by appropriate message design are discussed.     

Towards a learning networked organisation: human capital, compatibility and usability in e-learning systems

In all parts of organisations there flourish developments of different new subsystems in areas of knowledge and learning. Over recent decades, new systems for classification of jobs have emerged both at the level of organisations and at a macro-labour market level. Recent developments in job evaluation systems make it possible to cope with the new demands for equity at work (between, for example, genders, races, physical abilities). Other systems have emerged to describe job requirements in terms of skills, knowledge and competence. Systems for learning at work and web-based learning have created a demand for new ways to classify and to understand the process of learning. Often these new systems have been taken from other areas of the organisation not directly concerned with facilitating workplace learning. All these new systems are of course closely interrelated but, in most organisations, a major problem is the severe lack of cohesion and compatibility between the different subsystems. The aim of this paper is to propose a basis for how different human resource systems can be integrated into the business development of an organisation. We discuss this problem and develop proposals alternative to integrated macro-systems. A key element in our proposition is a structure for classification of knowledge and skill to be used in all parts of the process. This structure should be used as an added dimension or an overlay on all other subsystems of the total process. This will facilitate a continued use of all existing systems within different organisations. We develop Burge's (personal communication) model for learning to show that learning is not a successive linear process, but rather an iterative process. In this way we emphasise the need for greater involvement of learners in the development of learning systems towards increased usability in a networked system. This paper is divided into two parts which are closely related. The first part gives an overview of the lack of compatibility between the different subsystems. In this first part we note two paradoxes which impact learning and for which we propose solutions. The second part deals with 'usability' aspects of these competency-related systems; in particular, usability in e-learning systems. In this second part we describe an example of a new organisational structure. We conclude by discussing four key concepts that are necessary conditions for organisations to address when developing their human capital. Establishing these conditions helps ensure compatibility and usability in e-learning systems.     

Development and evaluation of a Web 2.0 annotation system as a learning tool in an e-learning environment

The emergence of Web 2.0 technology provides more opportunities to foster online communication and sharing in an e-learning environment. The purpose of this study was to develop a Web 2.0 annotation system, MyNote, based on the Web 2.0 core concepts which emphasize ease of access and active sharing and then to gain an understanding about people’s perceptions of MyNote from a usability perspective. MyNote was employed on multimedia learning objects in a Learning Management System (LMS), and out of the LMS as well in this study. The evaluation results showed that, with factor analysis, interactivity, usefulness, helpfulness, and willingness for future use were categorized to represent the perceptions of MyNote. It was also found that the factors of interactivity and helpfulness were statistically significant to predict the future use of MyNote. Lastly, the habit of taking notes also affected learners’ perceptions of using MyNote.     

Assessing the usability of a visual tool for the definition of e-learning processes

In this paper, we present a usability study aiming at assessing a visual language-based tool for developing adaptive e-learning processes. The tool implements the adaptive self-consistent learning object SET (ASCLO-S) visual language, a special case of flow diagrams, to be used by instructional designers to define classes of learners through stereotypes and to specify the more suited adaptive learning process for each class of learners. The usability study is based on the combined use of two techniques: a questionnaire-based survey and an empirical analysis. The survey has been used to achieve feedbacks from the subjects’ point of view. In particular, it has been useful to capture the perceived usability of the subjects. The outcomes show that both the proposed visual notation and the system prototype are suitable for instructional designers with or without experience on the computer usage and on tools for defining e-learning processes. This result is further confirmed by the empirical analysis we carried out by analysing the correlation between the effort to develop adaptive e-learning processes and some measures suitable defined for those processes. Indeed, the empirical analysis revealed that the effort required to model e-learning processes is not influenced by the experience of the instructional designer with the use of e-learning tools, but it only depends on the size of the developed process.     

Adjunct aids and signals support online learning from multiple representations

To achieve deeper understanding when learning from multiple representations, learners should actively select, organize and integrate the relevant information from text and graphics within a coherent mental representation. However, as learners often fail to select and integrate all relevant information, especially from graphics, they need specific instructional support. The current study investigated the effects of instructional support in the form of adjunct aids (i.e. fill-in-the blank tasks) with references to the graphics (e.g. “see Figure X”) on retention and understanding. In our study, 106 learners (N) received multimedia instructional materials about the formation of auroras either with or without adjunct aids – the former with references to graphics (signals), or with none. In line with our hypotheses, adjunct aids with signals led to deeper understanding, as reflected by higher scores in the comprehension test. In contrast, adjunct aids with signals did not lead to higher scores in the retention test. Thus, our results are in line with previous research, showing that instructional support for integrating text and graphics specifically fosters deeper understanding. Possible boundary conditions and implications for future research are discussed.     

Context enabled decision aids to support the situation awareness and performance of risk engineers carrying out loss prevention surveys

Routine inspection by insurance companies at their clients’ facility, also known as loss prevention survey, help identify the best strategies to minimize damages when there is a high-speed wind event. More specifically, wind vulnerabilities associated with a building are evaluated using a process known as windstorm risk inspection. This routine inspection helps clients reduce the extent of damages caused by high-speed wind events including hurricane and tornado. Risk engineers make use of their subjective and analytical deduction skills to successfully carry out the inspection tasks. In this research the researchers investigated the effect of context-based visualization strategies on situation awareness and their understanding of the situation. The study examined how different types of information contribute towards the three levels of situation awareness. Following a between-subjects study design, 65 participants completed the study. Each session lasted 90–120 min. A checklist based and predictive display-based decision aids were tested and found to be effective in supporting the situation awareness requirements as well as performance of risk engineers. However, the predictive display only helped with certain tasks such as understanding the interaction among different components on the rooftop. For remaining tasks such as perceiving obvious issues like membrane tear, clogged drains and vegetation growth, checklist alone was sufficient. This study helped the understanding of the advantages and disadvantages of the decision aids tested. More specifically, these decision aids can improve the mental model of novice risk engineers. Additionally, this study provided insights that could help design training materials for infrastructure inspectors.     

The effect of positive mood on intention to use computerized decision aids

While psychology research has indicated that positive mood enhances cognition and behavior, MIS researchers have largely ignored such potential effects on user acceptance of new IT. Using two cognitive theories on mood and memory, positive mood theory and the affect infusion model (AIM), we developed hypotheses about the contribution of these mood conditions on user acceptance of new IT with two levels of uncertainty. These hypotheses were investigated via an experiment using a computerized decision making aid. We found that positive mood increased acceptance, under both levels of uncertainty. These results held for both induced and naturally occurring positive mood. The results were consistent with positive mood theory but not with the AIM.     

Human-factors aspects of designing computer voice output systems as communication aids for the disabled

Possible ways of meeting the user's needs when designing communication aids with synthetic voice output are discussed. The points raised evolved from a case study of the development of microcomputer based systems which the blind people can use to read page-oriented text. The ideas discussed in this paper may also enhance the performance of blind computer programmers. Essential facilities in communication aids for the non-vocal are also inferred.     

The impact of distractions on the usability and intention to use mobile devices for wireless data services

Mobile technology has quickly become ingrained in society due to the flexibility of anywhere/anytime usage. However, factors associated with and that impact mobility, mobile users, and mobile use of products and services are still poorly understood. For example, even though distractions are ever present during everyday use of mobile devices, the nature and extent to which user perceptions and performance are affected by their presence is unknown. An empirical study was undertaken to investigate the impact of distractions and confirmation of pre-trial expectations on usability and its subsequent effect on consumers’ behavioral intention toward using a mobile device for wireless data services. Distractions were simulated in this study in the form of either user motion or environmental noise (i.e. background auditory and visual stimuli). A Structural Equation Modelling (SEM) analysis confirmed the impacts of distractions on efficiency and effectiveness, and in turn the users’ satisfaction and behavioral intention to use a mobile device for wireless data services. Support was also obtained for a mediating effect of post-trial confirmation of expectations between perceived performance and satisfaction. Implications of these findings for theory, practice, and future research are outlined.     

Using sketches and storyboards to assess impact of age difference in user experience

We compared two versions of a touch-screen digital thermostat using a framework encompassing several user experience (UX) characteristics, and here describe how the implementation of certain design factors (specialists, praises, tooltips and increased interactivity) was done on mixed-fidelity prototypes of the user interface. We illustrate how the experimental comparison, involving 20 university students and 20 older adults, revealed important differences in UX, including perceived ease of use, behavioral intentions, enjoyment, quality, satisfaction, trust and usability, measured mainly through established questionnaires.Analysis revealed that using that kind of artifacts is a very cost effective way to elicit interesting and useful results; many UX variables are significantly affected by design factors and by age differences, as expected; effects of design factors go well beyond usability and therefore could not be caught by running an investigation focused only on usability.Age difference matters: older adults do not respond to addition of specialists, praises and tooltips as younger users do. We argue that potential benefits of these design choices are outweigh by the increase in complexity of the user interface.From a methodological viewpoint we suggest using a particular array of UX characteristics and metrics when testing mixed-fidelity prototypes. Not all the metrics that we adopted were equally useful, and in particular perceived usability, subjective mental effort, and emotions did not help us highlighting differences.     

基于Haptic技术的盲人路径诱导辅具应用研究现状

触觉(Haptic)技术信息表达是一种有效的人机交互模式和信息传递方式,并作为信息传递的重要通道用于盲人导航/路径诱导辅具领域,弥补了在特定条件下声音的缺点与不足。对盲人导航/路径诱导辅具总体情况进行了概述;结合空间认知介绍了基于Haptic技术的盲人导航/路径诱导辅具研究现状与应用;分析了基于Haptic技术的盲人导航/路径诱导辅具在今后的发展趋势和存在的问题,并讨论了应用前景。     

The impact of intrinsic motivation on e-learning in authentic computer tasks

Abstract Students with high intrinsic motivation often outperform students with low intrinsic motivation. However, little is known about the processes that lead to these differences. In education based on simulations or authentic electronic learning environments, this lack of insight is even more clear. The present study investigated what students actually did in an electronic learning environment that was designed as a game‐like realistic simulation in which students had to play the role of a junior consultant. The results show that students with high intrinsic motivation did not do more, rather they tended to do different things. Analysis of log files showed that the increased curiosity that students with high intrinsic motivation have, resulted in proportionally more explorative study behaviour. However, the learning outcomes of students with high intrinsic motivation were not better.     

The contribution of apparent and inherent usability to a user's satisfaction in a searching and browsing task on the Web

Previous research found that apparent and inherent usability had big influences on user's acceptance of computer systems. This research extended these findings to the area of user's satisfaction. The study investigated the impact of inherent and apparent usability on user's satisfaction of Web page designs while the two main tasks on the World Wide Web, searching and browsing, were performed. The results indicated that inherent usability was the main factor contributing to user's satisfaction in both searching and browsing. General principles and specific recommendations for the design and evaluation of Web page designs are derived from these findings.