Developing Web-based assessment strategies for facilitating junior high school students to perform self-regulated learning in an e-Learning environment

This research refers to the self-regulated learning strategies proposed by Pintrich (1999) in developing a multiple-choice Web-based assessment system, the Peer-Driven Assessment Module of the Web-based Assessment and Test Analysis system (PDA-WATA). The major purpose of PDA-WATA is to facilitate learner use of self-regulatory learning behaviors to perform self-regulated learning and in turn improve e-Learning effectiveness. PDA-WATA includes five main strategies: ‘Adding Answer Notes,’ ‘Stating Confidence,’ ‘Reading Peer Answer Notes,’ ‘Recommending Peer Answer Notes’ and ‘Querying Peers’ Recommendation on Personal Answer Notes’. Using these strategies, examinees are allowed to add answer notes to explain why they chose a certain option as the correct answer and state their confidence in their own answer and answer notes, for peers’ reference. In addition to reading peer answer notes, examinees can also recommend peer answer notes as valuable references. The recommendation information can also be queried by all examinees. Quasi-experimental design was adopted to understand the effectiveness of PDA-WATA in facilitating learner use of self-regulatory learning behaviors to perform self-regulated learning and in improving learner e-Learning effectiveness. Participants were 123 seventh-grade junior high school students from four classes. These four classes were randomly divided into the PDA-WATA group (n = 63) and the N-WBT group (n = 60). Before e-Learning instruction, all students took the pre-test of the Learning Process Inventory (LPI), used to understand how often learners use self-regulatory learning behaviors in the learning process, and the pre-test of the summative assessment. After a two-week e-Learning instruction, the students all took the post-test of the LPI and the summative assessment. Results indicate that students in the PDA-WATA group appear to be more willing to take the Web-based formative assessment than students in the N-WBT group. In addition, PDA-WATA appears to be significantly more effective than N-WBT in facilitating learner use of self-regulatory learning behaviors to perform self-regulated learning and in improving their e-Learning effectiveness. Moreover, this research also finds that in the PDA-WATA group, there is no significant difference between the learning effectiveness of students with a low level of self-regulated learning and students with a high level of self-regulated learning, but similar result cannot be found in the N-WBT group.     

Implementation of Web-based dynamic assessment in facilitating junior high school students to learn mathematics

This research adopts the Graduated Prompting Assessment Module of the WATA system (GPAM-WATA) and applies it to the remedial teaching of junior high school mathematics. The theoretical basis of the development of GPAM-WATA is the idea of ‘graduated prompt approach’ proposed by Campione and Brown, 1985, Campione and Brown, 1987. In GPAM-WATA, when examinees fail to answer items correctly, they obtain instructional prompts (IPs) in a graduated way. This research developed the contents of IPs based on the mathematical problem-solving theory of Mayer (1992, pp. 458–460). A quasi-experimental design was adopted. Ninety-six junior high school seventh graders from three different classes participated in this research. The three classes were randomly divided into the GPAM-WATA group (n = 31), the N-WBT group (n = 31) and the PPT group (n = 34). All students received traditional mathematics instruction from the same teacher. After traditional mathematics instruction, all students took the pre-test of the summative assessment. The students in the three different groups then respectively received remedial teaching in the form of GPAM-WATA, normal Web-based test (N-WBT), and paper-and-pencil test (PPT). After the remedial teaching, all students took the post-test of the summative assessment. The results indicate that compared with other groups, performing remedial teaching using GPAM-WATA has significantly better effectiveness. Moreover, it is found that the IPs in GPAM-WATA are effective in remedial teaching for not only those students most lacking in different types of mathematical problem-solving knowledge but also all the other students.     

Developing an assessment-centered e-Learning system for improving student learning effectiveness

This research used Web-based two-tier diagnostic assessment and Web-based dynamic assessment to develop an assessment-centered e-Learning system, named the ‘GPAM-WATA e-Learning system.’ This system consists of two major designs: (1) personalized dynamic assessment, meaning that the system automatically generates dynamic assessment for each learner based on the results of the pre-test of the two-tier diagnostic assessment; (2) personalized e-Learning material adaptive annotation, meaning that the system annotates the e-Learning materials each learner needs to enhance learning based on the results of the pre-test of the two-tier diagnostic assessment and dynamic assessment. This research adopts a quasi-experimental design, applying GPAM-WATA e-Learning system to remedial Mathematics teaching of the ‘Speed’ unit in an elementary school Mathematics course. 107 sixth-graders from four classes in an elementary school participated in this research (55 male and 52 female). With each class as a unit, they were divided into four different e-Learning models: (1) the personalized dynamic assessment and personalized e-Learning material adaptive annotation group (n = 26); (2) the personalized dynamic assessment and non-personalized e-Learning material adaptive annotation group (n = 28); (3) the non-personalized dynamic assessment and personalized e-Learning material adaptive annotation group (n = 26); and (4) the non-personalized dynamic assessment and non-personalized e-Learning material adaptive annotation group (n = 27). Before remedial teaching, all students took the prior knowledge assessment and the pre-test of the summative assessment and two-tier diagnostic assessment. Students then received remedial teaching and completed all teaching activities. After remedial teaching, all students took the post-test of the summative assessment and two-tier diagnostic assessment. It is found that compared to the e-Learning models without personalized dynamic assessment, e-Learning models with personalized dynamic assessment are significantly more effective in facilitating student learning achievement and improvement of misconceptions, especially for students with low-level prior knowledge. This research also finds that personalized e-Learning material adaptive annotation significantly affects the percentage of reading time students spend on the e-Learning materials they need to enhance learning. However, it does not appear to predict student learning achievement and improvement of misconceptions.     

Web-based quiz-game-like formative assessment: Development and evaluation

This research aims to develop a multiple-choice Web-based quiz-game-like formative assessment system, named GAM-WATA. The unique design of ‘Ask-Hint Strategy’ turns the Web-based formative assessment into an online quiz game. ‘Ask-Hint Strategy’ is composed of ‘Prune Strategy’ and ‘Call-in Strategy’. ‘Prune Strategy’ removes one incorrect option and turns the original 4-option item into a 3-option one. ‘Call-in Strategy’ provides the rate at which other test takers choose each option when answering a question. This research also compares the effectiveness of three different types of formative assessment in an e-Learning environment: paper-and-pencil test (PPT), normal Web-based test (N-WBT) and GAM-WATA. In total, 165 fifth grade elementary students (from six classes) in central Taiwan participated in this research. The six classes of students were then divided into three groups and each group was randomly assigned one type of formative assessment. Overall results indicate that different types of formative assessment have significant impacts on e-Learning effectiveness and that the e-Learning effectiveness of the students in the GAM-WATA group appears to be better. Students in the GAM-WATA group more actively participate in Web-based formative assessment to do self-assessment than students in the N-WBT group. The effectiveness of formative assessment will not be significantly improved only by replacing the paper-and-pencil test with Web-based test. The strategies included in GAM-WATA are recommended to be taken into consideration when researchers design Web-based formative assessment systems in the future.     

Applying learning analytics dashboards based on process-oriented feedback to improve students' learning effectiveness

With the development of a technology-supported environment, it is plausible to provide rich process-oriented feedback in a timely manner. In this paper, we developed a learning analytics dashboard (LAD) based on process-oriented feedback in iTutor to offer learners their final scores, sub-scale reports, and corresponding suggestions on further learning content. We adopted a quasi-experimental design to investigate the effectiveness of the report on students' learning. Ninety-four freshman from two classes participated in this research. The two classes were divided into the LAD group and the original analytics report (OAR) based on a product-oriented feedback group. Before the experiment, all the students took the prior knowledge assessment. After a semester's instruction, all the students took the post-test of the summative assessment. Results indicated that students in the LAD group experienced better learning effectiveness than students in the OAR group. LAD based on process-oriented feedback was also effective in improving the skill learning effectiveness of the students with low-level prior knowledge.     

Developing a web‐based multimedia assessment system for facilitating science laboratory instruction

This study developed a web‐based multimedia assessment system (WMA system) and applied it to science laboratory instruction. The goal was to improve students' knowledge acquisition under science laboratory instruction. The developed system enabled learners to perform self‐assessments by responding to multimedia technology test items online. The system recorded each learner's complete answer history and provided the students with personalized learning resources. This study adopted a quasi‐experimental research design. The learning content was an “experiment on separating mixtures.” Students participating in the research were divided into a typical science laboratory instruction group (TI group; n = 25) and a group that received instruction through the WMA system (WMA group; n = 26). Before instruction, all the students completed the conceptual knowledge and experimental knowledge pretests. During instruction, the TI group watched the teacher's demonstration experiment, and the students then performed the experiment in a real laboratory. In the WMA group, after learning through the WMA system, the students also performed the experiment in a real laboratory. After instruction, all the students completed the conceptual knowledge and experimental knowledge posttests. The findings indicated that the students in the WMA group showed significantly higher improvements in their scientific conceptual knowledge and experimental knowledge.     

Learning styles and formative assessment strategy: enhancing student achievement in Web-based learning

Abstract The purpose of this research was to investigate the effects of formative assessment and learning style on student achievement in a Web-based learning environment. A quasi-experimental research design was used. Participants were 455 seventh grade students from 12 classes of six junior high schools. A Web-based course, named BioCAL, combining three different formative assessment strategies was developed. The formative assessment strategies included Formative Assessment Module of the Web-Based Assessment and Test Analysis system (FAM-WATA) (with six Web-based formative assessment strategies), Normal Module of Web-Based Assessment and Test Analysis system (N-WATA) (only with partial Web-based formative assessment strategy) and Paper and Pencil Test (PPT) (without Web-based formative assessment strategy). Subjects were tested using Kolb's Learning Style Inventory, and assigned randomly by class into three groups. Each group took Web-based courses using one of the formative assessment strategies. Pre- and post-achievement testing was carried out. A one-way ANCOVA analysis showed that both learning style and formative assessment strategy are significant factors affecting student achievement in a Web-based learning environment. However, there is no interaction between these two factors. A post hoc comparison showed that performances of the FAM-WATA group are higher than the N-WATA and PPT groups. Learners with a 'Diverger' learning style performed best followed by, 'Assimilator', 'Accommodator', and 'Converger', respectively. Finally, FAM-WATA group students are satisfied with six strategies of the FAM-WATA.     

A Web-based model for developing assessment literacy of secondary in-service teachers

This research investigates the effect of a web-based model, named ‘Practicing, Reflecting, and Revising with Web-based Assessment and Test Analysis system (P2R-WATA) Assessment Literacy Development Model,’ on enhancing assessment knowledge and perspectives of secondary in-service teachers, and adopts a single group experimental research design. The WATA system provides teachers with personalized learning resources and situated environment to practice assembling, administering tests on-line, and appraising test-related statistical information to reflect and revise test items. The sample consisted of forty-seven secondary in-service mathematics and science teachers in a summer program for 36 h within six weeks. This research collects and analyzes quantitative data, including the pre-test and post-test of teachers’ knowledge and perspectives on assessment. The major results generally confirm the effectiveness of P2R-WATA model. Firstly, the assessment knowledge of the participants has improved after training, especially for teachers with low-level prior knowledge. Secondly, the findings also reveal that there is a significant improvement on teachers’ assessment perspectives.     

The evaluation of different gaming modes and feedback types on game-based formative assessment in an online learning environment

This study proposed an online learning system for energy education, modifying the typical rules of tic-tac-toe and incorporating multiple choice tests into the game in order to develop a game-based formative assessment tool for an online learning course. In order to explore how different gaming modes and feedback types in this game-based formative assessment affect knowledge acquisition effectiveness and participation perceptions, a tic-tac-toe quiz game (TRIS-Q) with two gaming modes: single-player online game (SOG) and multi-player online game (MOG), and two feedback types: immediate elaborated feedback (IEF) and no immediate elaborated feedback (no IEF), were developed. A 2(SOG vs. MOG) × 2(IEF vs. no IEF) between-subject experiment was also conducted to investigate the effects on 109 ninth-grade students from four junior high school classes. The research findings indicated that different gaming modes of TRIS-Q did not affect the effectiveness of knowledge acquisition; providing IEF for each question answered in the game facilitated the enhancement of both energy knowledge acquisition and student tic-tac-toe ability when comparing it with the no IEF type. Additionally, the different gaming modes and feedback types did not affect participation perceptions.     

基于网络环境的学生发展性评价系统的设计

针对学生目前的终结性评价方式所存在的不足,本文提出一种基于网络的、动态的发展性评价系统,并给出该系统的设计思想、所使用的评价方法、体系结构和具体的功能设计。该系统是一个关注学生发展过程的持续性评价系统,所发挥的综合评价功效有利于学生的进步和健康成长。     

The validity of a game-based assessment of persistence

In this study, 154 students individually played a challenging physics video game for roughly 4 h. Based on time data for both solved and unsolved problems derived from log files, we created a game-based assessment of persistence that was validated against an existing measure of persistence. We found that the game-based assessment of persistence predicted learning of qualitative physics after controlling for gender, video game experience, pretest knowledge and enjoyment of the game. These findings support the implementation of a real-time formative assessment of persistence to be used to dynamically change gameplay.     

Implementation of Web‐based argumentation in facilitating elementary school students to learn environmental issues

This research develops a Web‐based argumentation system named the Web‐based Interactive Argumentation System (WIAS). WIAS can provide teachers with the scaffolding for argumentation instruction. Students can propose their statements, collect supporting evidence and share and discuss with peers online. This research adopts a quasi‐experimental design, applying WIAS to the teaching of environmental issues, including mudslides, global warming and nuclear power. Fifty‐seven elementary school fifth graders from two classes participated in this research. With each class as a unit, they were divided into the WIAS group (n = 30) and the traditional argumentation instruction (TAI) group (n = 27). Before research, all students took the pre‐test of the ‘achievement test for environmental issues (ATEI)’ and the ‘environmental literacy scale (ELS).’ Then all students received argumentation training and six classes of argumentation instruction. Students in the WIAS group performed argumentation in the WIAS, while those in the TAI group performed argumentation in a traditional classroom. After the six‐class argumentation instruction, all students took the post‐test of the ATEI and ELS. The results show that students in the WIAS group have significantly better learning effectiveness than those in the TAI group. Students in the WIAS group also exhibited significantly better improvement in their environmental literacy.     

What strategies are effective for formative assessment in an e‐learning environment?

Abstract  The web-based formative assessment developed in this research is named Formative Assessment Module of the Web-based Assessment and Test Analysis System (FAM-WATA). FAM-WATA is a multiple-choice web-based formative assessment module containing six effective strategies: 'repeat the test', 'correct answers are not given', 'query scores', 'ask questions', 'monitor answering history', and 'all pass and then reward'. This research explored the effectiveness of FAM-WATA, cognitive styles and e-learning, and student attitudes towards the six strategies of FAM-WATA. A total of 503 seventh-grade students in central Taiwan were valid in this research. Overall results indicated that students displayed a positive attitude towards the six strategies of FAM-WATA. In addition, results also showed that students in an e-learning environment equipped with FAM-WATA achieved better learning effectiveness, and that field independent students appeared to make better use of FAM-WATA strategies than field dependent students. This research concluded that FAM-WATA benefited student learning in an e-learning environment.     

Effects of a collaborative annotation method on students’ learning and learning-related motivation and affect

Two studies tested the effectiveness of a web-based collaborative annotation system (Hy-Lighter) for learning comprehension, and learning-related affect and motivation. In an undergraduate course setting, students (N = 27) in study 1, (1) highlighted and annotated selected articles, and (2) highlighted and annotated selected articles and reviewed peer highlights and annotations. In a graduate course setting, students (N = 40) in study 2, (1) highlighted and annotated selected articles, and (2) highlighted and annotated selected articles and reviewed peer highlights and annotations. Control groups in both studies read a hard copy of the articles -without using HyLighter and engaging in its associated annotation practices. The main dependent variables included: (a) performance on quizzes, and (b) a number of affective and motivational variables related to reading assignments and academic success. Although not statistically significant, summative assessment scores were higher for students using HyLigther relative to the ones exposed to conventional instruction. HyLighter use also seemed to be associated with more positive affect in undergraduate students relative to their graduate counterparts. Somewhat equivocal findings between the two studies were attributed to the differential implementation of the software in and outside of the classroom. Recommendations for optimal use and desired outcomes were advanced.     

A knowledge‐structure‐based adaptive dynamic assessment system for calculus learning

The purpose of this study was to investigate the effect of a calculus system that was designed using an adaptive dynamic assessment (DA) framework on performance in the “finding an area using an integral”. In this study, adaptive testing and dynamic assessment were combined to provide different test items depending on students’ abilities. Prompts were provided from among various options. Two hundred fifty‐seven freshmen from one public university in Taiwan participated in the study. The pre‐test was held within one week after the examinees learned how to find an area using an integral. Remedial instruction was completed within two weeks, and the students were administered the post‐ test. Two weeks after the administration of the post‐test, a delayed post‐test was administered to evaluate the students’ retention. When an examinee responded to an item correctly, he received a score. Otherwise, the student was given a prompt. In this study, five experimental groups were compared: three different DA groups, one self‐study group and one remedial group instruction. The results of the study revealed that the instructive effect of the adaptive dynamic assessment approach (the third DA group) was the best and that the proposed methods helped students improve their learning performance.     

A methodology for applying students' interactive task performance scores from a multimedia-based performance assessment in a Bayesian Network

Computer-based simulations are increasingly being used in educational assessment. In most cases, the simulation-based assessment (SBA) is used for formative assessment, which can be defined as assessment for learning, but as research on the topic continues to grow, possibilities for summative assessment, which can be defined as assessment of learning, are also emerging. The current study contributes to research on the latter category of assessment. In this article, we present a methodology for scoring the interactive and complex behavior of students in a specific type of SBA, namely, a Multimedia-based Performance Assessment (MBPA), which is used for a summative assessment purpose. The MBPA is used to assess the knowledge, skills, and abilities of confined space guard (CSG) students. A CSG supervises operations that are carried out in a confined space (e.g., a tank or silo). We address two specific challenges in this article: the evidence identification challenge (i.e., scoring interactive task performance), and the evidence accumulation challenge (i.e., accumulating scores in a psychometric model). Using expert ratings on the essence and difficulty of actions in the MBPA, we answer the first challenge by demonstrating that interactive task performance in MBPA can be scored. Furthermore, we answer the second challenge by recoding the expert ratings in conditional probability tables that can be used in a Bayesian Network (a psychometric model for reasoning under uncertainty and complexity). Finally, we validate and illustrate the presented methodology through the analysis of the response data of 57 confined space guard students who performed in the MBPA.     

Studying the effectiveness of an online argumentation model for improving undergraduate students' argumentation ability

This research develops a web-based model, entitled the “intuitive claim, peer-assessment, discussion, and elaborate claim argumentation training” (IPadE) model, and embeds with a Web-based Interactive Argumentation System to enhance undergraduate students' socioscientific argumentation abilities. This research adopts a quasi-experimental research design; the sample comprised 131 undergraduate students from two classes (69 in the experimental group and 62 in the control group). The socioscientific issue discussed were related to global health. This study collected and analysed quantitative and qualitative data, including the pretest and posttest of students' knowledge test scores and argumentation abilities questionnaire. The results generally confirmed the effectiveness of the IPadE model. First, in a comparison of the content knowledge and argumentation skills, the experimental group have statistically significantly improved than the control group. Second, regarding the number of reasoning modes proposed, the experimental group could propose multiple reasoning modes and reasoning levels on rebuttals increased after training.     

Designing a Web-based assessment environment for improving pre-service teacher assessment literacy

Teacher assessment literacy is a key factor in the success of teaching, but some studies concluded that teachers lack it. The aim of this research is to propose the “Practicing, Reflecting and Revising with WATA system (P2R-WATA) Assessment Literacy Development Model” for improving pre-service teacher assessment literacy. WATA system offers personalized learning resources and opportunities for pre-service teachers to assemble tests and administer them to students on-line. Furthermore, WATA system facilitates performance of test analysis and item analysis, and enables pre-service teachers to review statistical information from the test and item analyses to revise test items. Sixty pre-service teachers participated in this research. The research results indicate that pre-service teachers using P2R-WATA Assessment Literacy Development Model have better effectiveness in improving their assessment knowledge and assessment perspectives.     

动静态特征结合的漏洞风险评估及缓解方法

针对如何提高漏洞风险评估的准确性进行了研究,提出一种动静态特征结合的漏洞风险评估及缓解方法。通过将传统风险评估方法中常用的来源于通用漏洞评分系统（CVSS）的攻击复杂度、影响程度、攻击向量等固定属性作为静态特征,将防御能力、漏洞修复情况、攻击者的攻击能力等随时间推移可能发生变化的属性作为动态特征,两者结合对漏洞的风险程度进行更加全面的评估。给出了在实际应用中各特征的量化计算方法,以及漏洞修复策略的推荐方法。以单个漏洞的风险评估过程和多个漏洞的风险评估结果为例,将评估结果与CVSS评分进行对比实验。结果表明该方法能结合具体的网络环境给出更加准确的漏洞风险评估结果及合理的漏洞修复策略,验证了该方法的可行性和有效性。