Examining the designs of computer-based assessment and its impact on student engagement,satisfaction, and pass rates

Many researchers who study the impact of computer-based assessment (CBA) focus on the affordances or complexities of CBA approaches in comparison to traditional assessment methods. This study examines how CBA approaches were configured within and between modules, and the impact of assessment design on students' engagement, satisfaction, and pass rates. The analysis was conducted using a combination of longitudinal visualisations, correlational analysis, and fixed-effect models on 74 undergraduate modules and their 72,377 students. Our findings indicate that educators designed very different assessment strategies, which significantly influenced student engagement as measured by time spent in the virtual learning environment (VLE). Weekly analyses indicated that assessment activities were balanced with other learning activities, which suggests that educators tended to aim for a consistent workload when designing assessment strategies. Since most of the assessments were computer-based, students spent more time on the VLE during assessment weeks. By controlling for heterogeneity within and between modules, learning design could explain up to 69% of the variability in students' time spent on the VLE. Furthermore, assessment activities were significantly related to pass rates, but no clear relation with satisfaction was found. Our findings highlight the importance of CBA and learning design to how students learn online.     

The effects of time-compressed audio and verbal redundancy on learner performance and satisfaction

Digital audio is becoming increasingly popular in higher education with faculty digitally recording and broadcasting lectures for students to learn-on-demand. Students have discovered accelerated playback features in popular computer software and use it to reduce the amount of time spent listening to audio-enhanced instruction. In the current study, 183 undergraduates were randomly assigned to one of three audio-enhanced multimedia presentations that were recorded at three speeds (1.0, 1.4, and 1.8). Results show no significant difference on performance across treatments and a significant difference on satisfaction in favor of 1.4 times the normal audio speed. The results also indicate statistical differences in favor of verbal redundancy, in which the same verbal information was presented on both an auditory and visual channel.     

Learning with hypermedia: The influence of representational formats and different levels of learner control on performance and learning behavior

In this paper, two experiments on the use of hypermedia environments for learning about probability theory are reported. In Experiment 1a it was tested whether multimedia design principles (multimedia principle, modality principle, redundancy principle) are valid in hypermedia environments, despite the fact that hypermedia offers more learner control than multimedia. The results showed only little evidence for this validity, although the hypermedia environment entailed only a rather low level of learner control. In Experiment 1b it was investigated how learner control affects performance and how its possible impact is moderated by learners’ prior knowledge. A high level of learner control positively affected the effectiveness of instruction only with regard to intuitive knowledge, but was at the same time accompanied by large increases in learning time, thereby rendering the instruction inefficient. Unexpectedly, effects of learner control were not moderated by students’ prior knowledge. The results imply that the idea to use multimedia design principles for hypermedia learning is too simple and that the benefits and drawbacks of learner control depend heavily on learning objectives and time constraints.     

A model of (en)action to approach embodiment: a cornerstone for the design of virtual environments for learning

This paper presents a model of (en)action from a conceptual and theoretical point of view. This model is used to provide solid bases to overcome the complexity of designing virtual environments for learning (VEL). It provides a common grounding for trans-disciplinary collaborations where embodiment can be perceived as the cornerstone of the project. Where virtual environments are concerned, both computer scientists and educationalists have to deal with the learner/user’s body; therefore the model provides tools with which to approach both human actions and learning processes within a threefold model. It is mainly based on neuroscientific research, including enaction and the neurophysiology of action.     

Technology use,self-directed learning,student engagement and academic performance: Examining the interrelations

The widespread technology use among current college and university students has made higher educational institutions worldwide acknowledge the need of incorporating it in teaching and learning for explicit reasons. But does access and usage of technology enhance academic performance and foster student engagement in reality? Researches in the last over two decades have conjectured both the positive and negative outcomes of the students’ continuous interface with technology. Student engagement and self-directed learning (SDL) are the two other themes that have independently attracted considerable interest of researchers, ascribable to the explicit and implicit assertions that both are related to the academic success. Additionally, the relationship of technology use with these two academic behaviors have also been investigated although not very extensively. The current study aimed to inspect a path model with technology use, student engagement, self-directed learning and academic performance among undergraduate students. 761 students responded to an online survey comprising three scales: Media and Technology Usage and Attitude Scale (MTUAS), Self-Rating Scale of Self-Directed Learning (SRSSDL), and student version of Utrecht’s Work Engagement Scale (UWES-S). The results showed that use of technology has a direct positive relationship with students’ engagement and self-directed learning, however, no significant direct effect was found between technology use and academic performance. The findings point towards the complex interchange of relationships of the students’ technology use with student engagement, self-directed learning and academic performance. The implications and future research directions are discussed.     

Computer-supported team-based learning: The impact of motivation,enjoyment and team contributions on learning outcomes

The benefits of teamwork and collaboration have long been advocated by many educational theories, such as constructivist and social learning models. Among the various applications of collaborative learning, the iterative team-based learning (TBL) process proposed by Michaelsen, Fink, and Knight (2002) has been successfully used in the classroom without computer support. This paper describes the implementation and evaluation results of a classroom application of the TBL process, which was modified to include computer mediation. We call this process computer-supported team-based learning (CS-TBL). This work extends learning in small teams from the traditional classroom to the hybrid classroom where students meet both face-to-face and online by emphasizing the importance of online team interactions. The outcomes are assessed through an evaluation model that considers the impact of motivation, enjoyment and team contributions on learning outcomes. The study results indicate that motivation influences the relationship between team interactions and perceived learning. Enjoyment is affected by motivation and perceptions of team members’ contributions, with the implication that students who perceive that the team interactions are adding value to their education will better enjoy learning and will experience higher-level learning outcomes.     

The impact of on-demand metacognitive help on effortful behaviour: A longitudinal study using task-related visual analytics

This longitudinal study investigates the differences in learners' effortful behaviour over time due to receiving metacognitive help—in the form of on-demand task-related visual analytics. Specifically, learners' interactions (N = 67) with the tasks were tracked during four self-assessment activities, conducted at four discrete points in time, over a period of 8 weeks. The considered and coded time points were: (a) prior to providing the metacognitive help; (b) while the task-related visual analytics were available (treatment); (c) after the removal of the treatment; and (d) while the option to receive metacognitive help was available again. To measure learners' effortful behaviour across the self-assessment activities, this study utilized learners' response-times to correctly/wrongly complete the tasks and on-task effort expenditure. The panel data analysis shown that the usage of metacognitive help caused statistically significant changes in learners' effortful behaviour, mostly in the third and fourth phase. Statistically significant changes were detected also in the usage of metacognitive help. These results provide empirical evidence on the benefits of task-related visual analytics to support learners' on-task engagement, and suggest relevant cues on how metacognitive help could be designed and prompted by focusing on the “task”, instead of the “self”.     

Associations between technological scaffolding and micro-level processes of self-regulated learning: A workplace study

This paper reports the results of a study of the association of a set of technological scaffolding interventions – implemented in a learning software called Learn-B – with micro-level processes of self-regulated learning in the workplace. The study was conducted in the context of two European organizations, in which the study participants (N = 53) used the learning software Learn-B for two months in their regular activities. Data about perceived usefulness of the technological scaffolding interventions and traces about the events of the actual use of the technological interventions and micro-level processes of self-regulated learning were collected and analyzed. The analysis of the data showed that when directly asked about their own perspectives, the participants tended to rely on their organizational context in their planning phase. The results also indicate that the participants did consider the social context of their organization when planning their learning goals, but it was not found as the most influential factor. The analysis of the trace data revealed a moderate balance between reliance on both social and organizational contexts. The analysis of the theorized contingencies via trace data showed that the participants’ usefulness perceptions of the proposed interventions for their self-regulated processes, poorly matched with their actual learning actions.     

The role of student learning styles, gender, attitudes and perceptions on information and communication technology assisted learning

The present study adopts an illuminative approach to evaluate students’ initial attitudes towards the use of information and communication technology (ICT). Ninety-nine undergraduate science students participated in this study and their learning styles were classified according to Honey and Mumford (1986) (Honey, P., Mumford, A., 1986. The Manual of Learning styles. Peter Honey, 10 Linden Avenue, Maidenhead) learning style questionnaire. Student learning styles were classified as activist, reflector, theorist, or pragmatist. No significant difference in learning styles was observed between genders and between student cohorts. Six dimensions to student attitudes toward ICT were identified as follows; ‘comfort’, ‘interactivity’, ‘self-satisfaction’, ‘value new technology’, ‘experience’ and ‘context’. Students exhibited low scores in the attitude dimensions of ‘value new technology’, ‘interactivity’ and ‘context’ indicating that they were uncomfortable with computers, were unhappy about the lack of personal contact and would prefer to learn in a more traditional mode. A significant, though weak, negative correlation between the ‘theorist’ and the ‘interactivity’ and ‘context’ attitude dimensions was also observed. In addition, based on the results of this study it appears that first year students exhibit a more positive perception of ICT supported learning than second and third year students. Though the use of ICT in higher education is becoming more widespread based on the results of this study student use of the technology may be limited by a negative attitude toward a style of teaching which is not consistent with their past learning experiences.     

Learner outcomes and satisfaction: A comparison of live video-streamed instruction,satellite broadcast instruction,and face-to-face instruction

This study examined the final grade and satisfaction level differences among students taking specific courses using three different methods: face-to-face in class, via satellite broadcasting at remote sites, and via live video-streaming at home or at work. In each case, the same course was taught by the same instructor in all three delivery methods, and an attempt was made to survey students taking the course via the three different delivery methods. MANOVA results indicated no grade or satisfaction level differences among the three populations. Self-reported computer literacy skills revealed a slight fit between the chosen delivery mode and the reported computer literacy skills. These results provide additional evidence to support both the “no significant difference” phenomenon and the use of distance education as a viable, convenient and flexible alternative delivery mode capable of extending learning opportunities to non-traditional students.     

Multi-criteria evaluation of the web-based e-learning system: A methodology based on learner satisfaction and its applications

The web-based e-learning system (WELS) has emerged as a new means of skill training and knowledge acquisition, encouraging both academia and industry to invest resources in the adoption of this system. Traditionally, most pre- and post-adoption tasks related to evaluation are carried out from the viewpoints of technology. Since users have been widely recognized as being a key group of stakeholders in influencing the adoption of information systems, their attitudes toward this system are pivotal. Therefore, based on the theory of multi-criteria decision making and the research products of user satisfaction from the fields of human–computer interaction and information systems, this study proposed a multi-criteria methodology from the perspective of learner satisfaction to support those evaluation-based activities taking place at the pre- and post-adoption phases of the WELS life cycle. In addition, by following this methodology, this study empirically investigated learners’ perceptions of the relative importance of decision criteria. This investigation carried out a survey of college students, and the data thus obtained was then analyzed by analytic hierarchy process in order to derive an integrated preference structure of learners as a ground for evaluation. We found that learners regarded the learner interface as being the most important dimension of decision criteria. Future applications of these results are recommended and the implications are discussed.     

The impact of learning styles on student grouping for collaborative learning: a case study

Learning style models constitute a valuable tool for improving individual learning by the use of adaptation techniques based on them. In this paper, we present how the benefit of considering learning styles with adaptation purposes, as part of the user model, can be extended to the context of collaborative learning as a key feature for group formation. We explore the effects that the combination of students with different learning styles in specific groups may have in the final results of the tasks accomplished by them collaboratively. With this aim, a case study with 166 students of computer science has been carried out, from which conclusions are drawn. We also describe how an existing web-based system can take advantage of learning style information in order to form more productive groups. Our ongoing work concerning the automatic extraction of grouping rules starting from data about previous interactions within the system is also outlined. Finally, we present our challenges, related to the continuous improvement of collaboration by the use and dynamic modification of automatic grouping rules.     

Evaluating learning, design, and engagement in web-based learning tools (WBLTs): The WBLT Evaluation Scale

Web-based learning tools (WBLTs), also known as learning objects, are online, interactive tools that support the learning of specific concepts by enhancing, amplifying, and/or guiding the cognitive processes of learners. Research examining the effectiveness of WBLTs is somewhat limited because sound, reliable, valid evaluation metrics are sparse, particularly in the K-12 environment. The purpose of the following study was to re-examine the Learning Object Evaluation Scale for students (LOES-S), originally developed by Kay and Knaack (2009), to assess three key constructs: learning, design, and engagement. Over 800 middle and secondary schools students participated in high quality, pre-designed lessons intended to accentuate the use of WBLTs. Data collected from the new WBLT Evaluation Scale demonstrated good internal reliability, construct validity, convergent validity and predictive validity.     

Assessing the effects of gamification in the classroom: A longitudinal study on intrinsic motivation,social comparison,satisfaction, effort,and academic performance

Gamification, the application of game elements to non-game settings, continues to grow in popularity as a method to increase student engagement in the classroom. We tested students across two courses, measuring their motivation, social comparison, effort, satisfaction, learner empowerment, and academic performance at four points during a 16-week semester. One course received a gamified curriculum, featuring a leaderboard and badges, whereas the other course received the same curriculum without the gamified elements. Our results found that students in the gamified course showed less motivation, satisfaction, and empowerment over time than those in the non-gamified class. The effect of course type on students' final exam scores was mediated by students' levels of intrinsic motivation, with students in the gamified course showing less motivation and lower final exam scores than the non-gamified class. This suggests that some care should be taken when applying certain gamification mechanics to educational settings.     

An implementation of design-based learning through creating educational computer games: A case study on mathematics learning during design and computing

This mixed-method case study examined the potential of computer-assisted, math game making activities in facilitating design-based math learning for school children. Sixty-four middle school children participated in Scratch-based, math game making activities. Data were collected via activity and conversation observation, artifact analysis, interviewing, and survey. The study findings indicated that participants developed significantly more positive dispositions toward mathematics after computer game making. The study also found that experience-driven game design processes helped to activate children's reflection on everyday mathematical experiences. Mathematical thinking and content experience were intertwined within the process of computer game authoring. On the other hand, children designers were involved in game-world and story crafting more than mathematical representation. And it was still challenging for them to perform computer game coding with abstract reasoning.     

The correlates of the digital divide and their impact on college student learning

By focusing on two dimensions of the digital divide—computer use and computer knowledge, this study explores four research questions: (1) What are the undergraduates doing with the computers they use at colleges? (2) How do undergraduates perform in regard to computer knowledge and skills? (3) With what is the digital divide among college students correlated? (4) What consequences does the digital divide have for student academic performance? In order to answer these research questions, a national survey was conducted. The survey investigated 3083 first-year college students of 12 4-year universities in Taiwan. A total of 2719 of them completed the questionnaires resulting in a response rate of 88.2%. In this study, the digital divide is measured in terms of computer use, which includes a variety of purposes for using computers and academic-related work as a proportion of total computer hours, and computer knowledge. Multiple regressions and a generalized ordered logit, i.e. a partial proportional odds model, are employed. The main findings include the following: (1) Undergraduates use computers not only for fulfilling their academic requirements and searching for information, but also for entertainment. On average, undergraduates spend about 19 h per week using computers, of which 5 h are academic-related. (2) Most undergraduates perform at the middle average level in terms of computer knowledge. (3) No significant differences among correlates in relating to demographic and socioeconomic family background were found in predicting the various purposes in using computers. (4) Students who are female, whose fathers and/or whose mothers are from minorities, whose fathers are blue-collar workers or unemployed, who study in the fields of the humanities and social sciences, and who enter private universities are at a disadvantage in terms of computer skills and knowledge. However, female students, students whose mothers were less educated and students who enroll in private universities are more focused computer users in terms of allocating time to academic-related work. (5) Computer knowledge and devotion to using computers for academic-related work have a moderate effect on college student learning, while the various other uses of computers do not. Of the different kinds of computer knowledge, it is the knowledge of software that helps students to learn the most.     

Engaging online learners: The impact of Web-based learning technology on college student engagement

Widespread use of the Web and other Internet technologies in postsecondary education has exploded in the last 15 years. Using a set of items developed by the National Survey of Student Engagement (NSSE), the researchers utilized the hierarchical linear model (HLM) and multiple regressions to investigate the impact of Web-based learning technology on student engagement and self-reported learning outcomes in face-to-face and online learning environments. The results show a general positive relationship between the use the learning technology and student engagement and learning outcomes. We also discuss the possible impact on minority and part-time students as they are more likely to enroll in online courses.     

Learning through playing Virtual Age: Exploring the interactions among student concept learning,gaming performance,in-game behaviors,and the use of in-game characters

Video games possess many unique features that facilitate learning. Meanwhile, teaching about evolution is never an easy task due to the existence of some barriers to its learning. Virtual Age, therefore, has been developed in an attempt to harness the power of gaming to increase student understanding of biological evolution. The aim of this study was to examine whether Virtual Age is effective for learning about evolution and to further explore the interplay of student concept learning, gaming performance, and in-game behaviors. A total of 62 7th graders took part in the study, and significant findings were revealed. The students did learn by playing Virtual Age, and their long-term knowledge retention was promising. The in-game behaviors, such as times and duration of viewing the relevant information embedded in Virtual Age, were significantly related to gaming performance (game score), which subsequently influenced learning outcomes. Moreover, the results of cluster analysis indicated that three clusters of low learning outcomes/low gaming performance, high learning outcomes, and high gaming performance emerged. Overall, Virtual Age is an effective game for learning about evolution based on its sound and sophisticated design. Implications derived from the study and suggestions for future work are proposed.     

The role of collective efficacy, cognitive quality, and task cohesion in computer-supported collaborative learning (CSCL)

Research has suggested that CSCL environments contain fewer social context clues, resulting in various group processes, performance or motivation. This study thus attempts to explore the relationship among collective efficacy, group processes (i.e. task cohesion, cognitive quality) and collaborative performance in a CSCL environment. A total of 75 Taiwanese college students (divided into 25 groups) participated in the study. Both quantitative and qualitative methods were applied for data analysis. The results indicate that collective efficacy significantly predicted task cohesion but not cognitive quality in the CSCL environment. For the role of group processes in performance, both task cohesion and cognitive quality significantly predicted group performance, but cognitive quality predicted better than task cohesion. In addition, for the predictive capability of prior performance, task cohesion, and cognitive quality in collective efficacy, the results showed that only task cohesion predicted subsequent collective efficacy significantly in the CSCL environment.