Effects of feedback in a computer-based assessment for learning

The effects of written feedback in a computer-based assessment for learning on students’ learning outcomes were investigated in an experiment at a Higher Education institute in the Netherlands. Students were randomly assigned to three groups, and were subjected to an assessment for learning with different kinds of feedback. These are immediate knowledge of correct response (KCR) + elaborated feedback (EF), delayed KCR + EF, and delayed knowledge of results (KR). A summative assessment was used as a post-test. No significant effect was found for the feedback condition on student achievement on the post-test. Results suggest that students paid more attention to immediate than to delayed feedback. Furthermore, the time spent reading feedback was positively influenced by students’ attitude and motivation. Students perceived immediate KCR + EF feedback to be more useful for learning than KR. Students also had a more positive attitude towards feedback in a CBA when they received KCR + EF rather than KR only.     

Providing feedback on computer-based algebra homework in middle-school classrooms

Homework is transforming at a rapid rate with continuous advances in educational technology. Computer-based homework, in particular, is gaining popularity across a range of schools, with little empirical evidence on how to optimize student learning. The current aim was to test the effects of different types of feedback on computer-based homework. In the study, middle school students completed a computer-based pretest, homework assignment, and posttest containing challenging algebraic problems. On the homework assignment, students were assigned to different feedback conditions. In Experiment 1 (N = 103), students received no feedback or correct-answer feedback after each problem. In Experiment 2 (N = 143), students received (1) no feedback, (2) correct-answer feedback, (3) try-again feedback, or (4) explanation feedback after each problem. For students with low prior knowledge, feedback resulted in better posttest performance than no feedback. However, students with high prior knowledge learned just as much whether they received feedback or not. Results suggest the provision of basic feedback on computer-based homework can benefit novice students’ mathematics learning.     

Learning through computer-based concept mapping with scaffolding aid

Abstract Concept mapping has been applied in a variety of fields, including instruction, learning, curriculum development, and assessment. Because many empirical studies have proven the validity of concept mapping, a computer-based concept mapping system has been developed. The system provides two learning environments. In the 'construct-by-self' environment, the system provides students with the evaluation results and corresponding hints for feedback. The students construct concept maps by themselves with only the assistance of the feedback. In the 'construct-on-scaffold' environment, in addition to the feedback, the students receive an incomplete concept map, within which some nodes and links were set as blanks for the scaffold. A study comparing the effectiveness of the 'construct-by-self', 'construct-on-scaffold', and 'construct by paper-and-pencil' concept mapping showed that the 'construct-on-scaffold' had better effect for learning on biology. Both of the two computer-based procedures are helpful for students in completing their concept maps.     

Effects of feedback type and modality on motor skill learning and retention

ABSTRACT

Two types of feedback, including knowledge of results (KR) and knowledge of performance (KP) are typically delivered as part of motor training using different sensory modalities. Twenty-four right-handed individuals performed a computer-based psychomotor training task using a contemporary haptic interface. Results revealed that KP was superior to KR for overall task performance retention, whereas KR was good for training specific aspects of performance. A combination of KP and KR through the same modality did not produce an additive positive learning effect for task performance. Auditory and haptic feedback improved retention over visual, but only with the use of multiple information cues. Design guidelines are provided for training systems for motor skill development and/or rehabilitation.     

Concept maps: A useful and usable tool for computer-based knowledge assessment? A literature review with a focus on usability

The concept map is now widely accepted as an instrument for the assessment of conceptual knowledge and is increasingly being embedded into technology-based environments. Usability addresses how appropriate (for a particular use) or how user-friendly a computer-based assessment instrument is. As we know from human-computer interaction research, if the interface is not user-friendly, a computer-based assessment can result in decreased test performance and reduced validity. This means that the usability of the interface affects the assessment in such a way that if the test is not user-friendly, then the test taker will not be able to fully demonstrate his/her level of proficiency and will instead be scored according to his/her information and communication technology (ICT) literacy skills. The guidelines of the International Test Commission (2006) require usability testing for such instruments and suggest that design standards be implemented. However, we do not know whether computer-conducted concept map assessments fulfill these standards. The present paper addresses this aspect. We conducted a systematic research review to examine whether and how researchers have studied and considered usability when conducting computer-based concept map assessments. Only 24 out of 119 journal articles that assessed computer-based concept maps discussed the usability issue in some way. Nevertheless, our review brings to light the idea that the impact of usability on computer-based concept map assessments is an issue that has received insufficient attention. In addition, usability ensures a suitable interaction between test taker and test device; thus, the training effort required for test use can be reduced if a test's usability is straight forward. Our literature review, however, illustrates that the interplay between usability and test use training has mostly been neglected in current studies.     

Transitional feedback schedules during computer-based problem-solving practice

Feedback has a strong influence on effective learning from computer-based instruction. Prior research on feedback in computer-based instruction has mainly focused on static feedback schedules that employ the same feedback schedule throughout an instructional session. This study examined transitional feedback schedules in computer-based multimedia instruction on procedural problem-solving in electrical circuit analysis. Specifically, we compared two transitional feedback schedules: the TFS-P schedule switched from initial feedback after each problem step to feedback after a complete problem at later learning states; the TFP-S schedule transitioned from feedback after a complete problem to feedback after each problem step. As control conditions, we also considered two static feedback schedules, namely providing feedback after each practice problem-solving step (SFS) or providing feedback after attempting a complete multi-step practice problem (SFP). Results indicate that the static stepwise (SFS) and transitional stepwise to problem (TFS-P) feedback produce higher problem solving near-transfer post-test performance than static problem (SFP) and transitional problem to step (TFP-S) feedback. Also, TFS-P resulted in higher ratings of program liking and feedback helpfulness than TFP-S. Overall, the study results indicate benefits of maintaining high feedback frequency (SFS) and reducing feedback frequency (TFS-P) compared to low feedback frequency (SFP) or increasing feedback frequency (TFP-S) as novice learners acquire engineering problem solving skills.     

基于领域概念网络的自动批改技术*

随着网络教学的飞速发展和推广应用,作为网络教学核心技术的自动批改技术是近年来讨论的热点问题。针对主观题自动批改技术的难点,提出一种基于带权概念网络进行分层语义相似度计算的主观题自动批改方案,并且通过实际应用验证了该解决方案的可行性,从而在一定程度上解决了自动批改问题。     

On-line peer assessment and the role of the peer feedback: A study of high school computer course

The purposes of this study were to explore the effects and the validity of on-line peer assessment in high schools and to analyze effects of various types of peer feedback on students. The participating students, a total of 184 10th graders, developed their individual course projects by involving the on-line peer assessment learning activities in a computer course. The peer assessment activities consisted of three rounds, and each of the students acted as an author and a reviewer. Research data as evaluated from peers and experts indicated that students significantly improved their projects as involving the peer assessment activities. The scores determined by the learning peers were highly correlated with those marked by the experts, indicating that peer assessment in high school could be perceived as a valid assessment method. Moreover, this study also examined the relationships between the types of peer feedbacks in which students obtained from peer assessment and the subsequent performance of their projects. We categorized peer feedbacks into four types: Reinforcing, Didactic, Corrective and Suggestive. It was found that Reinforcing peer feedback was useful in helping students’ development of better projects; however, Didactic feedback and perhaps Corrective feedback provided by peers might play an unfavorable role for subsequent improvement of students’ projects. The Suggestive feedback may be helpful in the beginning of peer assessment activities; however, in the later parts of peer assessment, the effect of this type of feedback on learning might not be significant.     

Effect of different score reports of Web-based formative test on students' self-regulated learning

A new score report based on a mechanism of formative assessment and feedback is developed to offer individual testees not only their final scores but also their sub-scale scores, their percentile position, as well as corresponding feedback on self-regulation strategies. Structural equation modeling is adopted in the confirmatory factor analysis to validate the questionnaire data as well as the Web-based test. In designing and developing the new score report, multiple regression analysis is employed to quantitatively analyze the relationship between sub-scale scores and testee use of cognitive regulation. The experiment is then qualitatively complemented by the one-to-one interview. Results indicate that the new score report is more efficient to students' self-regulated learning than the traditional one.     

Cognitive load and instructionally supported learning with provided and learner-generated visualizations

This study investigated, whether learning from science texts can be enhanced by providing learners with different forms of visualizations (pictures) in addition to text. One-hundred-two 9th and 10th graders read a computer-based text on chemical processes of washing and answered questions on cognitive load (mental effort, perceived difficulty) and comprehension (retention, transfer, drawing). Instruction varied according to a 2 × 2-factorial design with ‘learner-generated pictures’ (yes, no) and ‘provided pictures’ (yes, no) as factors. Results indicate positive main effects of provided pictures on all three comprehension measures and negative main effects on both cognitive load measures. Additional analyses revealed a mediation effect of perceived difficulty on retention and transfer, that is learning with provided pictures decreased cognitive load and enhanced comprehension. Furthermore, results show a positive main effect of learner-generated pictures on drawing and mental effort, but no mediation effect. Taken together, computer-based learning with provided pictures enhances comprehension as it seems to promote active processing while reducing extraneous cognitive processing. Learners, generating pictures, however, seem to have less cognitive resources available for essential and generative processing, resulting in reduced comprehension. These results are in line with cognitive load theory, cognitive theories of multimedia learning, and generative theories of learning.     

On-line peer assessment and the role of the peer feedback: A study of high school computer course

The purposes of this study were to explore the effects and the validity of on-line peer assessment in high schools and to analyze effects of various types of peer feedback on students. The participating students, a total of 184 10th graders, developed their individual course projects by involving the on-line peer assessment learning activities in a computer course. The peer assessment activities consisted of three rounds, and each of the students acted as an author and a reviewer. Research data as evaluated from peers and experts indicated that students significantly improved their projects as involving the peer assessment activities. The scores determined by the learning peers were highly correlated with those marked by the experts, indicating that peer assessment in high school could be perceived as a valid assessment method. Moreover, this study also examined the relationships between the types of peer feedbacks in which students obtained from peer assessment and the subsequent performance of their projects. We categorized peer feedbacks into four types: Reinforcing, Didactic, Corrective and Suggestive. It was found that Reinforcing peer feedback was useful in helping students’ development of better projects; however, Didactic feedback and perhaps Corrective feedback provided by peers might play an unfavorable role for subsequent improvement of students’ projects. The Suggestive feedback may be helpful in the beginning of peer assessment activities; however, in the later parts of peer assessment, the effect of this type of feedback on learning might not be significant.     

Learning to teach with worked-out examples: a computer-based learning environment for teachers

Abstract   Learning from worked-out examples has been shown to be very effective in initial cognitive skill acquisition. In order to fully exploit the potential of example-based learning, teachers should, however, know how to effectively employ such examples in classroom instruction. Therefore, we have designed a computer-based learning environment in which mathematics and science teachers learn how to effectively employ worked-out examples. The learning environment was developed according to approved design criteria. The topics that are addressed in the learning environment were chosen in response to the results of a needs assessment that analysed schoolbooks and classroom videos, and took interviews with teachers into account. In experiments as well as in several teacher trainings on example-based learning, the computer-based learning environment was evaluated and improved accordingly. On a more general level, our research and development project showed that a computer-based learning programme could be a sensible tool that supports follow-up work in teacher training contexts.     

Enhancing computer-supported writing of learning protocols by adaptive prompts

Cognitive and metacognitive prompts are a central support procedure in eHELp a computer-based environment that supports the writing of learning protocols. In order to investigate the effectiveness of adapting prompts, 79 students revised a learning protocol in eHELp either supported by prompts that were adapted on the basis of the results of an integrated learning-strategy questionnaire or a meta-knowledge test, respectively, by randomly selected prompts, or without any support. Adaptive prompts improved the quality of the learning protocols and fostered the acquisition of declarative knowledge and deep understanding, irrespective of the applied diagnostic instrument. In conclusion, open-ended learning tasks like the writing of learning protocols can be made more effective by adaptive support based on prior strategy assessment.     

高职院校“三维能力”考核系统开发

With the deepening of higher vocational education reform, more and more attention has been paid to the cultivation of so- cial ability and method ability in addition to technical ability in teaching. For the "three-dimensional ability" assessment mode gener- ated by this reform idea, more attention has been paid to the assessment of the state and performance of students in the learning pro- cess. At present, most of the domestic performance management systems are based on the functions of final performance statistics, teachers' performance and learning query. There are few records for students in the learning process. The development of informa- tion technology also makes it possible to record students' learning state in class. The "three-dimensional ability" assessment system of higher vocational colleges is the result of the development of both.     

高职院校“三维能力”考核系统开发

随着高职院校教育改革的深入,教学中除了技术能力之外的社会能力和方法能力的培养越来越受到重视,针对这种改革思路产生的“三维能力”考核模式更重视考核学生在学习过程中的状态和表现。目前国内的考核成绩管理系统多是以期末成绩统计、教师上传成绩、学生查询成绩等功能为主。对于平时学生学习过程中的记录较少。信息技术的发展也使得课堂记录学生学习状态成为可能。高职院校“三维能力”考核系统是两者发展的结果。     

Positive technological and negative pre-test-score effects in a four-year assessment of low socioeconomic status K-8 student learning in computer-based Math and Language Arts courses

Motivated by the Federal Title I program to improve the Math and Language Arts learning of underachieving students of low socioeconomic status, the Education Program for Gifted Youth (EPGY) at Stanford University has developed computer-based online Math and Language Arts courses for such students in elementary and middle schools. Using several large student samples, the four-year statistical assessment of state test performance is the focus of this report. The main statistical conclusion is that sustained and careful computer-based work, guided by motivated teachers, can be done by many, when taught on an individualized basis, at their current level of competence. The gains made by individual students are, to a large extent, monotonically increasing in their amount of net correct computer-based course work, and to an even larger extent monotonically decreasing as pre-test scores rise, a result that favors technological support of the more underachieving students.     

Multimedia recipe reading: Predicting learning outcomes and diagnosing cooking interest using eye-tracking measures

This study aimed to explore the relationships between students' visual behaviors and learning outcomes, and between visual behaviors and prior cooking interest in multimedia recipe learning. An eye-tracking experiment, including pretest, recall test, and retention test, was conducted with a sample of 29 volunteer hospitality majors in Taiwan. The multimedia recipe included a static page showing the ingredients in a text-and-picture representation and a dynamic page showing the knife skills in a text-and-video representation. Total reading time, total fixation duration, number of fixations and inter-scanning count were used to explore the students' visual attention distributions among the different representation elements and their visual strategies for learning the recipe. The results showed that all students paid more visual attention to the text than to the picture information for the static recipe, and paid more visual attention to the video than to the text on the dynamic page. In addition, the visual attention paid to the text on the dynamic page was negatively correlated with the retention of the episodic knowledge of knife skills. In contrast, the visual attention paid to the text on the static ingredient page was positively correlated with students' prior cooking interest. Finally, the inter-scanning count between text and video on the dynamic page was the best index to negatively predict students' learning retention. Total fixation duration on the text information on the static page was the best index to positively predict students' prior cooking interest. Future studies and applications are discussed.     

Measuring learning styles with questionnaires versus direct observation of preferential choice behavior in authentic learning situations: the visualizer/verbalizer behavior observation scale (VV-BOS)

Research on learning styles and their assessment has been the focus of growing interest due to rapid advances in educational multimedia applications. As an alternative to conventional questionnaires, the VV-BOS, a computer-based instrument for direct observation of students' preference for visual or verbal learning material in an authentic learning situation, was developed. A study with 103 second language learners was conducted to assess the reliability and validity of the scale scores provided by this instrument. The visualizer/verbalizer scale scores were highly reliable. They correlated moderately with corresponding VVQ and ELSIE questionnaire scale scores. Students' learning outcome could successfully be predicted using the VV-BOS scale, but not using the questionnaire scales. The results indicate that the direct observation of students' preferential choice behavior is a promising alternative to conventional questionnaires for the assessment of individual differences in multimedia training and instruction.     

Computer-based testing: An alternative for the assessment of Turkish undergraduate students

Virtually errorless high speed data processing feature has made computers popular assessment tools in education. An important concern in developing countries considering integrating computers as an educational assessment tool before making substantial investment is the effects of computer-based testing on students’ test scores as compared to paper-and-pencil tests. This study investigated whether test scores of Turkish students were different in the computer-based test and in the paper-and-pencil test, with forty-seven undergraduate students studying at a public university located in the Blacksea region of Turkey. Findings of this study showed that test scores of undergraduate students were not different in the computer-based test and in the paper-and-pencil test which led us to reach the conclusion that computer-based testing can be considered as a promising alternative technique for the undergraduate students in Turkey.     

Attention-gated reinforcement learning of internal representations for classification

Animal learning is associated with changes in the efficacy of connections between neurons. The rules that govern this plasticity can be tested in neural networks. Rules that train neural networks to map stimuli onto outputs are given by supervised learning and reinforcement learning theories. Supervised learning is efficient but biologically implausible. In contrast, reinforcement learning is biologically plausible but comparatively inefficient. It lacks a mechanism that can identify units at early processing levels that play a decisive role in the stimulus-response mapping. Here we show that this so-called credit assignment problem can be solved by a new role for attention in learning. There are two factors in our new learning scheme that determine synaptic plasticity: (1) a reinforcement signal that is homogeneous across the network and depends on the amount of reward obtained after a trial, and (2) an attentional feedback signal from the output layer that limits plasticity to those units at earlier processing levels that are crucial for the stimulus-response mapping. The new scheme is called attention-gated reinforcement learning (AGREL). We show that it is as efficient as supervised learning in classification tasks. AGREL is biologically realistic and integrates the role of feedback connections, attention effects, synaptic plasticity, and reinforcement learning signals into a coherent framework.