Web-based dynamic assessment: Taking assessment as teaching and learning strategy for improving students’ e-Learning effectiveness

This research combines the idea of cake format dynamic assessment defined by Sternberg and Grigorenko (2001) and the ‘graduated prompt approach’ proposed by 9 and 10 to develop a multiple-choice Web-based dynamic assessment system. This research adopts a quasi-experimental design to investigate the effectiveness of this Web-based dynamic assessment system (GPAM-WATA) and normal Web-based test (N-WBT). One hundred and sixteen sixth grade elementary students from four classes participated in this research. These four classes were randomly divided into the GPAM-WATA group and N-WBT group. Before e-Learning instruction, all the students took the prior knowledge assessment and the pre-test of the summative assessment. After 2-week e-Learning instruction, all the students took the post-test of the summative assessment. The research findings show that students in the GPAM-WATA group experience better e-Learning effectiveness than those in the N-WBT group. GPAM-WATA is also found to be effective in improving the e-Learning effectiveness of students with low-level prior knowledge.     

Developing Web-based assessment strategies for facilitating junior high school students to perform self-regulated learning in an e-Learning environment

This research refers to the self-regulated learning strategies proposed by Pintrich (1999) in developing a multiple-choice Web-based assessment system, the Peer-Driven Assessment Module of the Web-based Assessment and Test Analysis system (PDA-WATA). The major purpose of PDA-WATA is to facilitate learner use of self-regulatory learning behaviors to perform self-regulated learning and in turn improve e-Learning effectiveness. PDA-WATA includes five main strategies: ‘Adding Answer Notes,’ ‘Stating Confidence,’ ‘Reading Peer Answer Notes,’ ‘Recommending Peer Answer Notes’ and ‘Querying Peers’ Recommendation on Personal Answer Notes’. Using these strategies, examinees are allowed to add answer notes to explain why they chose a certain option as the correct answer and state their confidence in their own answer and answer notes, for peers’ reference. In addition to reading peer answer notes, examinees can also recommend peer answer notes as valuable references. The recommendation information can also be queried by all examinees. Quasi-experimental design was adopted to understand the effectiveness of PDA-WATA in facilitating learner use of self-regulatory learning behaviors to perform self-regulated learning and in improving learner e-Learning effectiveness. Participants were 123 seventh-grade junior high school students from four classes. These four classes were randomly divided into the PDA-WATA group (n = 63) and the N-WBT group (n = 60). Before e-Learning instruction, all students took the pre-test of the Learning Process Inventory (LPI), used to understand how often learners use self-regulatory learning behaviors in the learning process, and the pre-test of the summative assessment. After a two-week e-Learning instruction, the students all took the post-test of the LPI and the summative assessment. Results indicate that students in the PDA-WATA group appear to be more willing to take the Web-based formative assessment than students in the N-WBT group. In addition, PDA-WATA appears to be significantly more effective than N-WBT in facilitating learner use of self-regulatory learning behaviors to perform self-regulated learning and in improving their e-Learning effectiveness. Moreover, this research also finds that in the PDA-WATA group, there is no significant difference between the learning effectiveness of students with a low level of self-regulated learning and students with a high level of self-regulated learning, but similar result cannot be found in the N-WBT group.     

The evaluation of different gaming modes and feedback types on game-based formative assessment in an online learning environment

This study proposed an online learning system for energy education, modifying the typical rules of tic-tac-toe and incorporating multiple choice tests into the game in order to develop a game-based formative assessment tool for an online learning course. In order to explore how different gaming modes and feedback types in this game-based formative assessment affect knowledge acquisition effectiveness and participation perceptions, a tic-tac-toe quiz game (TRIS-Q) with two gaming modes: single-player online game (SOG) and multi-player online game (MOG), and two feedback types: immediate elaborated feedback (IEF) and no immediate elaborated feedback (no IEF), were developed. A 2(SOG vs. MOG) × 2(IEF vs. no IEF) between-subject experiment was also conducted to investigate the effects on 109 ninth-grade students from four junior high school classes. The research findings indicated that different gaming modes of TRIS-Q did not affect the effectiveness of knowledge acquisition; providing IEF for each question answered in the game facilitated the enhancement of both energy knowledge acquisition and student tic-tac-toe ability when comparing it with the no IEF type. Additionally, the different gaming modes and feedback types did not affect participation perceptions.     

How flipped learning based on the cognitive theory of multimedia learning affects students' academic achievements

The study aims to identify the effect of a flipped classroom approach designed according to the cognitive theory of multimedia learning on the academic achievements of eighth‐grade students (aged 14 years) in Saudi Arabia in computer science. To this end, a quasi‐experimental design was used, with a sample of 67 students; 33 students were assigned to the experimental group, whereas 34 comprised the control group. The experimental group was subjected to the flipped classroom approach, whereas the control group was given direct instruction. To measure student achievements, an instrument that measures cognitive skills based on Revised Bloom's taxonomy levels was designed. Findings revealed a positive effect on the experimental group's achievement levels with respect to Bloom's higher order thinking skills, that is, applying, analysing, and evaluating. No difference was found between the two groups in terms of academic achievements at the remembering and understanding levels of Bloom's taxonomy. Moreover, learners with low prior knowledge showed a higher improvement in academic achievements compared with those with high prior knowledge. This corresponds to the assumption of the cognitive theory of multimedia learning that learners with low prior knowledge would benefit from its principles more than learners with high prior knowledge.     

Developing an assessment-centered e-Learning system for improving student learning effectiveness

This research used Web-based two-tier diagnostic assessment and Web-based dynamic assessment to develop an assessment-centered e-Learning system, named the ‘GPAM-WATA e-Learning system.’ This system consists of two major designs: (1) personalized dynamic assessment, meaning that the system automatically generates dynamic assessment for each learner based on the results of the pre-test of the two-tier diagnostic assessment; (2) personalized e-Learning material adaptive annotation, meaning that the system annotates the e-Learning materials each learner needs to enhance learning based on the results of the pre-test of the two-tier diagnostic assessment and dynamic assessment. This research adopts a quasi-experimental design, applying GPAM-WATA e-Learning system to remedial Mathematics teaching of the ‘Speed’ unit in an elementary school Mathematics course. 107 sixth-graders from four classes in an elementary school participated in this research (55 male and 52 female). With each class as a unit, they were divided into four different e-Learning models: (1) the personalized dynamic assessment and personalized e-Learning material adaptive annotation group (n = 26); (2) the personalized dynamic assessment and non-personalized e-Learning material adaptive annotation group (n = 28); (3) the non-personalized dynamic assessment and personalized e-Learning material adaptive annotation group (n = 26); and (4) the non-personalized dynamic assessment and non-personalized e-Learning material adaptive annotation group (n = 27). Before remedial teaching, all students took the prior knowledge assessment and the pre-test of the summative assessment and two-tier diagnostic assessment. Students then received remedial teaching and completed all teaching activities. After remedial teaching, all students took the post-test of the summative assessment and two-tier diagnostic assessment. It is found that compared to the e-Learning models without personalized dynamic assessment, e-Learning models with personalized dynamic assessment are significantly more effective in facilitating student learning achievement and improvement of misconceptions, especially for students with low-level prior knowledge. This research also finds that personalized e-Learning material adaptive annotation significantly affects the percentage of reading time students spend on the e-Learning materials they need to enhance learning. However, it does not appear to predict student learning achievement and improvement of misconceptions.     

Supervised machine learning in multimodal learning analytics for estimating success in project‐based learning

Multimodal learning analytics provides researchers new tools and techniques to capture different types of data from complex learning activities in dynamic learning environments. This paper investigates the use of diverse sensors, including computer vision, user‐generated content, and data from the learning objects (physical computing components), to record high‐fidelity synchronised multimodal recordings of small groups of learners interacting. We processed and extracted different aspects of the students' interactions to answer the following question: Which features of student group work are good predictors of team success in open‐ended tasks with physical computing? To answer this question, we have explored different supervised machine learning approaches (traditional and deep learning techniques) to analyse the data coming from multiple sources. The results illustrate that state‐of‐the‐art computational techniques can be used to generate insights into the "black box" of learning in students' project‐based activities. The features identified from the analysis show that distance between learners' hands and faces is a strong predictor of students' artefact quality, which can indicate the value of student collaboration. Our research shows that new and promising approaches such as neural networks, and more traditional regression approaches can both be used to classify multimodal learning analytics data, and both have advantages and disadvantages depending on the research questions and contexts being investigated. The work presented here is a significant contribution towards developing techniques to automatically identify the key aspects of students success in project‐based learning environments, and to ultimately help teachers provide appropriate and timely support to students in these fundamental aspects.     

Analytics of learning tactics and strategies in an online learnersourcing environment

Background

The use of crowdsourcing in a pedagogically supported form to partner with learners in developing novel content is emerging as a viable approach for engaging students in higher-order learning at scale. However, how students behave in this form of crowdsourcing, referred to as learnersourcing, is still insufficiently explored.

Objectives

To contribute to filling this gap, this study explores how students engage with learnersourcing tasks across a range of course and assessment designs.

Methods

We conducted an exploratory study on trace data of 1279 students across three courses, originating from the use of a learnersourcing environment under different assessment designs. We employed a new methodology from the learning analytics (LA) field that aims to represent students' behaviour through two theoretically-derived latent constructs: learning tactics and the learning strategies built upon them.

Results

The study's results demonstrate students use different tactics and strategies, highlight the association of learnersourcing contexts with the identified learning tactics and strategies, indicate a significant association between the strategies and performance and contribute to the employed method's generalisability by applying it to a new context.

Implications

This study provides an example of how learning analytics methods can be employed towards the development of effective learnersourcing systems and, more broadly, technological educational solutions that support learner-centred and data-driven learning at scale. Findings should inform best practices for integrating learnersourcing activities into course design and shed light on the relevance of tactics and strategies to support teachers in making informed pedagogical decisions.     

Predicting students' knowledge after playing a serious game based on learning analytics data: A case study

Serious games have proven to be a powerful tool in education to engage, motivate, and help students learn. However, the change in student knowledge after playing games is usually measured with traditional (paper) prequestionnaires–postquestionnaires. We propose a combination of game learning analytics and data mining techniques to predict knowledge change based on in-game student interactions. We have tested this approach in a case study for which we have conducted preexperiments–postexperiments with 227 students playing a previously validated serious game on first aid techniques. We collected student interaction data while students played, using a game learning analytics infrastructure and the standard data format Experience API for Serious Games. After data collection, we developed and tested prediction models to determine whether knowledge, given as posttest results, can be accurately predicted. Additionally, we compared models both with and without pretest information to determine the importance of previous knowledge when predicting postgame knowledge. The high accuracy of the obtained prediction models suggests that serious games can be used not only to teach but also to measure knowledge acquisition after playing. This will simplify serious games application for educational settings and especially in the classroom easing teachers' evaluation tasks.     

The comparisons of the influences of prior knowledge on two game-based learning systems

Game-based learning provides many benefits, such as enhancing the interaction with students and stimulating their learning motivation. Thus, it is popular to learners who have diverse characteristics. To this end, individual differences play an essential role. Among various individual differences, previous studies demonstrated that prior knowledge has great effects on game-based learning. However, such studies mainly considered a single game-based learning system. To address this issue, this study examined how prior knowledge affects students' reactions to two different types of game based learning systems, i.e., the Machinarium and the CSI: Web Adventures. The former delivers procedural knowledge while the latter provides declarative knowledge. The results from this study indicate that prior knowledge has positive impacts in the CSI group while it has negative impacts in the Machinarium group. These findings imply that prior knowledge is useful for the context of declarative knowledge while it may not be helpful for the context of procedural knowledge.     

Improving the effectiveness of English vocabulary review by integrating ARCS with mobile game‐based learning

Memorizing English vocabulary is often considered uninteresting, and a lack of motivation exists during learning activities. Moreover, most vocabulary practice systems automatically select words from articles and do not provide integrated model methods for students. Therefore, this study constructed a mobile game‐based English vocabulary practice system, which imitates the popular block elimination game and combines test items of article, difficulty, and teacher model in accordance with curriculum objectives and demands. The participants were first‐year students and classified into traditional group (Group T) and game‐based group (Group G), and learning effectiveness and motivation were analysed in this study. According to statistical analysis results, students who used the game‐based vocabulary practice system, which could captivate their attention and interest, had higher learning effectiveness and provided positive feedback of learning motivation. Furthermore, English teachers could select teaching materials that are consistent with the features, knowledge, and cultural background of learners to improve the relevance dimension of learning motivation.     

Enhancing community of inquiry and reflective thinking skills of undergraduates through using learning analytics-based process feedback

This research aims to investigate into the effect of using learning analytics (LA)-based process feedback on students' perceptions of community of inquiry (teaching, social and cognitive presence) and their reflective thinking skills. By using a mixed-method research approach (QUAN + qual), this study was conducted as an experimental design with the pretest–posttest control group. A total of 104 university students who were randomly assigned to the experiment group (EG) and control group (CG) were recruited in this study. The procedure was conducted within the scope of the computing course based on the flipped classroom (FC) model. While the participants in the EG received LA-based process feedback which shows their LA results in a weekly manner, those in the CG did not get any LA-based process feedback. The data were collected through the Community of Inquiry Scale, the Reflective Thinking Scale and a semi-structured student opinion form. The findings indicated that sending feedback including the students' LA results had a statistically significant effect on the students' perceptions of community of inquiry and reflective thinking skills. Based on the findings of the study, several recommendations for teachers, instructional designers and researchers have been made.     

Facilitating application of language skills in authentic environments with a mobile learning system

We uncovered two critical issues in earlier studies: (a) some studies have shown that mobile learning technology is not beneficial for all students due to complexity of learning environments and student prior knowledge, skills, and experience and (b) familiarity of students with the authentic environments in which they learn using mobile technology did not receive much attention in earlier studies. To address these issues, we designed three learning tasks for a class of 26 junior high school students. The students applied language skills by completing the tasks in authentic environments individually in a first task, loosely collaborating with peers in a second task, and tightly collaborating with peers in a third task. A mobile learning system was also designed in this study to support students to accomplish the tasks. The aim of this study was to explore students' learning experiences using the learning system, their perceptions towards the system, and to assess how differently the students perform on the three tasks. According to our design, in the first task, the students took pictures of objects and described them orally or in writing using the mobile learning system. In the second task, after the students completed assignments, each student received comments from a partner through the system. In comments, the partner indicated flaws in student assignments and suggested how to fix them. In the third task, the students completed assignments, shared them using the system, and then exchanged comments with their partners face to face regarding issues related to their completed assignments and suggested how to improve them. Such learning behaviours in the three tasks enabled the students to practise writing and speaking skills. Our results demonstrate that most of the students highly valued our learning system and intend to use it in the future. Furthermore, the results show that the students performed best when they collaborated; namely, student performance was enhanced the most after the third task that required tight collaboration. Based on our results, we learned that students' familiarity with authentic environments is very important and beneficial for their learning. In addition, we learned that even in complex environment, less skilled and experienced students with low prior knowledge can perform well when they tightly collaborate with more skilled and experienced students with high prior knowledge, and our learning system can facilitate such collaboration.     

Effects of prior knowledge on learning from different compositions of representations in a mobile learning environment

Two experiments examined the effects of prior knowledge on learning from different compositions of multiple representations in a mobile learning environment on plant leaf morphology for primary school students. Experiment 1 compared the learning effects of a mobile learning environment presenting text and photos of plants on a tablet PC, either in combination with or without real plants in the physical environment. Results indicated that there were no interactions between prior knowledge and experimental condition. Students who learned with tablet PCs only outperformed students who additionally learned with real plants on a comprehension and an application test. In addition, high prior knowledge students outperformed low prior knowledge students on both tests. To investigate whether these effects were caused by the specific characteristics of the combination of photos of real plants and real plants, Experiment 2 compared the differential effects of prior knowledge on learning with the combination of texts, photos and real plants to a combination in which the photos were replaced by schematic hand drawings. Results indicated that both low and high prior knowledge students, who learned with the combination of texts, schematic hand drawings and real plants performed better on a comprehension and an application test. High prior knowledge students performed better on both tests. It is concluded that the number and type of representations used is critical for the effectiveness of mobile learning environments.     

Effects of abstract and concrete simulation elements on science learning

Contemporary evidence on the effectiveness of concrete and abstract representations in science education is based solely on studies conducted in college context. There it has been found that learning with abstract representations produces predominantly better outcomes than learning with concrete representations and combining the representations can be even more productive. In the present study, 52 elementary school students used a computer simulation to discover the basic principles of electric circuits. The perceptual concreteness of simulation elements either remained concrete or switched from concrete to abstract during the learning. In order to compare the relative effectiveness of the learning conditions and assess the changes in students' conceptual understanding, the students completed a subject knowledge assessment questionnaire before and after learning with the simulation. According to the results, the students gained a better understanding of the circuits by learning with constantly concrete simulation elements than switching from concrete to abstract elements during the learning. Constantly concrete elements also enhanced the learning process by reducing the learning times. The outcomes of the study suggest that the effects of concrete and abstract representations in science education are notably different in elementary school context as compared with college contexts.     

Teacher regulation of cognitive activities during student collaboration: Effects of learning analytics

By collaboratively solving a task, students are challenged to share ideas, express their thoughts, and engage in discussion. Collaborating groups of students may encounter problems concerning cognitive activities (such as a misunderstanding of the task material). If these problems are not addressed and resolved in time, the collaborative process is hindered. The teacher plays an important role in monitoring and solving the occurrence of problems. To provide adaptive support, teachers continuously have to be aware of students' activities in order to identify relevant events, including those that require intervention. Because the amount of available information is high, teachers may be supported by learning analytics. The present experimental study (n = 40) explored the effect of two learning analytics tools (the Concept Trail and Progress Statistics) that give information about students' cognitive activities. The results showed that when teachers had access to learning analytics, they were not better at detecting problematic groups, but they did offer more support in general, and more specifically targeted groups that experienced problems. This could indicate that learning analytics increase teachers' confidence to act, which in turn means students could benefit more from the teacher's presence.     

Powering up flipped learning: An online learning environment with a concept map-guided problem-posing strategy

Scholars have suggested that flipped learning can be enhanced by designing effective in-class learning activities to improve students' higher order thinking skills. Problem posing has been recognized as such a learning strategy that has great potential for improving students' higher order thinking skills. However, it has been reported as a challenging task for most students. Research evidence has indicated the potential of engaging students in group learning with step-by-step hints when conducting problem-posing activities. Therefore, this study proposed a multilevel concept mapping-based problem-posing strategy to assist students in completing the problem-posing tasks in flipped learning. To understand the effectiveness of the proposed strategy, a flipped learning system was developed and adopted in the science class of an elementary school. Seventy-five fifth graders from three classes were divided into experimental group A, experimental Group B and a control group. Students in Group A used the concept map-guided problem-posing strategy for flipped learning (CMPP-FL), students in Group B used the problem-posing strategy-based flipped learning, while students in the control group used the conventional flipped learning strategy. The results showed that the CMPP-FL was effective in terms of improving students' learning performance, especially for students with higher levels of critical thinking tendency.     

An interactive peer-assessment criteria development approach to improving students' art design performance using handheld devices

Engaging students in higher order thinking such as evaluation and analysis has been recognized as being an important strategy for helping them develop knowledge and skills, in particular, in creativeness-oriented learning activities such as artwork design. In this paper, an interactive peer-assessment criteria development approach is proposed to help students develop assessment criteria, learn from viewing peers' work, and make reflections in artwork design activities using mobile devices. To evaluate the effectiveness of the proposed approach, an experiment was conducted in an elementary school art course. A total of 103 students participating in the experiment were assigned to an experimental group and a control group. The students in the experimental group learned with the proposed approach, while those in the control group learned with the conventional peer assessment approach. From the experimental results, it was found that the proposed approach significantly improved the students' learning achievement, learning motivation and meta-cognitive awareness, suggesting the effectiveness of engaging students in assessment criteria development in an interactive manner.     

Micro-persistence and difficulty in a game-based learning environment for computational thinking acquisition

Persistence has been identified as a crucial quality of learning. However, it is hard to attain in online game-based environments as the drive to progress in the game may influence the ability to achieve the learning goals. This study aimed to examine the associations between micro-persistence, that is, the tendency to complete an individual task successfully, and task difficulty while acquiring computational thinking (CT). We further explored whether contextual or personal attributes better explain micro-persistence. We analysed data of 111 school students who used the CodeMonkey platform. We took a learning analytics approach for analysing the platform's log files. We found that micro-persistence is associated with task difficulty and that students who demonstrated an aptitude to learn new material are motivated to achieve the best solution. We also found that contextual variables better-explained micro-persistence than personal attributes. Encouraging micro-persistence can improve CT acquisition and the learning processes involved.     

An investigation into effectiveness of different reflective learning strategies for learning operational software

Skill certification promotion is one of the main policies facilitated by the technological and vocational education, where application software instruction is regarded as the core curriculum to foster skill certification. With its close connection with problem-solving learning, application software instruction relies heavily on hands-on operation incorporating information technology to adequately unravel the challenges where living or working application is simulated as problem situations. According to Dewey (1963) and Edwards (1996), the process of reflection is characterized by the inference course where learners attempt to analyze and solve the problems. However, more evidence is needed to decide what reflective learning strategies are effective for students' learning. Application software operation is categorized as procedural knowledge. Repeated drills are requisite to reach the ultimate goal of spontaneous reaction without thinking. Features of CAL system offer a well-rounded environment to meet the demands. The purposes of this study were 1) to investigate how different reflective learning strategies can affect learning effectiveness of operational application software acquisition, 2) to identify effective learning strategies and to incorporate the CAL approach with instructional practices to foster learning performance. Aiming at characteristics of operational software, this study proposed operational software learning strategy theory model based on reflective learning and Adaptive Character of Thought-Rational (ACT-R) model theories. The proposed model modified the reflective learning theory and added cyclical loop into CAL to fit for operational software instruction. The CAL system is developed and incorporated into learning activities of reflective learning theory strategy model by collecting frequent operation errors made in the first-year experiment as the source drill items. This study is conducted in a two-year sequence. A total of 172 second-grade students was recruited from a vocational high school. Different reflective learning strategies, individual and group reflective learning strategy, are implemented on two experimental groups in the first year. CAL strategy is later added into the experimental groups in the second year. The results suggest that group reflective learning strategy can enhance learning effectiveness of the holistic and medium-score group students. When reflective learning strategy is incorporated with CAL, in addition to maintaining the first-year learning effectiveness, learning effectiveness of the holistic and low-score group students can be benefited by individual reflective learning strategy. Furthermore, reflective learning incorporating with CAL has greater learning effectiveness than the learning without CAL.     

On-line synchronous scientific argumentation learning: Nurturing students' argumentation ability and conceptual change in science context

The purpose of this study is to examine the difference in effectiveness between two on-line scientific learning programs – one with an argumentation component and one without an argumentation component – on students' scientific argumentation ability and conceptual change. A quasi-experimental design was used in this study. Two classes of 8th grade students (the experimental group) received the on-line scientific argumentation learning program about chemical reaction, and the other two classes of 8th grade students (the control group) received the same on-line scientific learning program about chemical reaction, but without argumentation, for two weeks. All 140 students were administered the scientific conception test, conceptual change test, and argumentation test before, one week after, and eight weeks after learning. In addition, the experimental group students' on-line argumentation process was collected. Results showed that the students of the experimental group significantly outperformed the control group, regardless of scientific conceptions, conceptual change, and argumentation. Regression results indicated that hold of scientific conceptions is the best predicator for students' conceptual change, followed by argumentation ability. The quantity and quality of scientific arguments that students generated in a series of argumentation questions improved across the four topics. In addition, students also successfully changed their conceptions from pre- to post-driving questions across four topics. This clearly demonstrates that students' argumentation ability and conceptual change were both facilitated through receiving the on-line Synchronous Argumentation science learning program.