Web-based dynamic assessment: Taking assessment as teaching and learning strategy for improving students’ e-Learning effectiveness

This research combines the idea of cake format dynamic assessment defined by Sternberg and Grigorenko (2001) and the ‘graduated prompt approach’ proposed by 9 and 10 to develop a multiple-choice Web-based dynamic assessment system. This research adopts a quasi-experimental design to investigate the effectiveness of this Web-based dynamic assessment system (GPAM-WATA) and normal Web-based test (N-WBT). One hundred and sixteen sixth grade elementary students from four classes participated in this research. These four classes were randomly divided into the GPAM-WATA group and N-WBT group. Before e-Learning instruction, all the students took the prior knowledge assessment and the pre-test of the summative assessment. After 2-week e-Learning instruction, all the students took the post-test of the summative assessment. The research findings show that students in the GPAM-WATA group experience better e-Learning effectiveness than those in the N-WBT group. GPAM-WATA is also found to be effective in improving the e-Learning effectiveness of students with low-level prior knowledge.     

Developing Web-based assessment strategies for facilitating junior high school students to perform self-regulated learning in an e-Learning environment

This research refers to the self-regulated learning strategies proposed by Pintrich (1999) in developing a multiple-choice Web-based assessment system, the Peer-Driven Assessment Module of the Web-based Assessment and Test Analysis system (PDA-WATA). The major purpose of PDA-WATA is to facilitate learner use of self-regulatory learning behaviors to perform self-regulated learning and in turn improve e-Learning effectiveness. PDA-WATA includes five main strategies: ‘Adding Answer Notes,’ ‘Stating Confidence,’ ‘Reading Peer Answer Notes,’ ‘Recommending Peer Answer Notes’ and ‘Querying Peers’ Recommendation on Personal Answer Notes’. Using these strategies, examinees are allowed to add answer notes to explain why they chose a certain option as the correct answer and state their confidence in their own answer and answer notes, for peers’ reference. In addition to reading peer answer notes, examinees can also recommend peer answer notes as valuable references. The recommendation information can also be queried by all examinees. Quasi-experimental design was adopted to understand the effectiveness of PDA-WATA in facilitating learner use of self-regulatory learning behaviors to perform self-regulated learning and in improving learner e-Learning effectiveness. Participants were 123 seventh-grade junior high school students from four classes. These four classes were randomly divided into the PDA-WATA group (n = 63) and the N-WBT group (n = 60). Before e-Learning instruction, all students took the pre-test of the Learning Process Inventory (LPI), used to understand how often learners use self-regulatory learning behaviors in the learning process, and the pre-test of the summative assessment. After a two-week e-Learning instruction, the students all took the post-test of the LPI and the summative assessment. Results indicate that students in the PDA-WATA group appear to be more willing to take the Web-based formative assessment than students in the N-WBT group. In addition, PDA-WATA appears to be significantly more effective than N-WBT in facilitating learner use of self-regulatory learning behaviors to perform self-regulated learning and in improving their e-Learning effectiveness. Moreover, this research also finds that in the PDA-WATA group, there is no significant difference between the learning effectiveness of students with a low level of self-regulated learning and students with a high level of self-regulated learning, but similar result cannot be found in the N-WBT group.     

Implementation of Web-based dynamic assessment in facilitating junior high school students to learn mathematics

This research adopts the Graduated Prompting Assessment Module of the WATA system (GPAM-WATA) and applies it to the remedial teaching of junior high school mathematics. The theoretical basis of the development of GPAM-WATA is the idea of ‘graduated prompt approach’ proposed by Campione and Brown, 1985, Campione and Brown, 1987. In GPAM-WATA, when examinees fail to answer items correctly, they obtain instructional prompts (IPs) in a graduated way. This research developed the contents of IPs based on the mathematical problem-solving theory of Mayer (1992, pp. 458–460). A quasi-experimental design was adopted. Ninety-six junior high school seventh graders from three different classes participated in this research. The three classes were randomly divided into the GPAM-WATA group (n = 31), the N-WBT group (n = 31) and the PPT group (n = 34). All students received traditional mathematics instruction from the same teacher. After traditional mathematics instruction, all students took the pre-test of the summative assessment. The students in the three different groups then respectively received remedial teaching in the form of GPAM-WATA, normal Web-based test (N-WBT), and paper-and-pencil test (PPT). After the remedial teaching, all students took the post-test of the summative assessment. The results indicate that compared with other groups, performing remedial teaching using GPAM-WATA has significantly better effectiveness. Moreover, it is found that the IPs in GPAM-WATA are effective in remedial teaching for not only those students most lacking in different types of mathematical problem-solving knowledge but also all the other students.     

Developing an assessment-centered e-Learning system for improving student learning effectiveness

This research used Web-based two-tier diagnostic assessment and Web-based dynamic assessment to develop an assessment-centered e-Learning system, named the ‘GPAM-WATA e-Learning system.’ This system consists of two major designs: (1) personalized dynamic assessment, meaning that the system automatically generates dynamic assessment for each learner based on the results of the pre-test of the two-tier diagnostic assessment; (2) personalized e-Learning material adaptive annotation, meaning that the system annotates the e-Learning materials each learner needs to enhance learning based on the results of the pre-test of the two-tier diagnostic assessment and dynamic assessment. This research adopts a quasi-experimental design, applying GPAM-WATA e-Learning system to remedial Mathematics teaching of the ‘Speed’ unit in an elementary school Mathematics course. 107 sixth-graders from four classes in an elementary school participated in this research (55 male and 52 female). With each class as a unit, they were divided into four different e-Learning models: (1) the personalized dynamic assessment and personalized e-Learning material adaptive annotation group (n = 26); (2) the personalized dynamic assessment and non-personalized e-Learning material adaptive annotation group (n = 28); (3) the non-personalized dynamic assessment and personalized e-Learning material adaptive annotation group (n = 26); and (4) the non-personalized dynamic assessment and non-personalized e-Learning material adaptive annotation group (n = 27). Before remedial teaching, all students took the prior knowledge assessment and the pre-test of the summative assessment and two-tier diagnostic assessment. Students then received remedial teaching and completed all teaching activities. After remedial teaching, all students took the post-test of the summative assessment and two-tier diagnostic assessment. It is found that compared to the e-Learning models without personalized dynamic assessment, e-Learning models with personalized dynamic assessment are significantly more effective in facilitating student learning achievement and improvement of misconceptions, especially for students with low-level prior knowledge. This research also finds that personalized e-Learning material adaptive annotation significantly affects the percentage of reading time students spend on the e-Learning materials they need to enhance learning. However, it does not appear to predict student learning achievement and improvement of misconceptions.     

Learning styles and formative assessment strategy: enhancing student achievement in Web-based learning

Abstract The purpose of this research was to investigate the effects of formative assessment and learning style on student achievement in a Web-based learning environment. A quasi-experimental research design was used. Participants were 455 seventh grade students from 12 classes of six junior high schools. A Web-based course, named BioCAL, combining three different formative assessment strategies was developed. The formative assessment strategies included Formative Assessment Module of the Web-Based Assessment and Test Analysis system (FAM-WATA) (with six Web-based formative assessment strategies), Normal Module of Web-Based Assessment and Test Analysis system (N-WATA) (only with partial Web-based formative assessment strategy) and Paper and Pencil Test (PPT) (without Web-based formative assessment strategy). Subjects were tested using Kolb's Learning Style Inventory, and assigned randomly by class into three groups. Each group took Web-based courses using one of the formative assessment strategies. Pre- and post-achievement testing was carried out. A one-way ANCOVA analysis showed that both learning style and formative assessment strategy are significant factors affecting student achievement in a Web-based learning environment. However, there is no interaction between these two factors. A post hoc comparison showed that performances of the FAM-WATA group are higher than the N-WATA and PPT groups. Learners with a 'Diverger' learning style performed best followed by, 'Assimilator', 'Accommodator', and 'Converger', respectively. Finally, FAM-WATA group students are satisfied with six strategies of the FAM-WATA.     

Web-based Personalised System of Instruction: An effective approach for diverse cohorts with virtual learning environments?

The Personalised System of Instruction is a form of mastery learning which, though it has been proven to be educationally effective, has never seriously challenged the dominant lecture-tutorial teaching method in higher education and has largely fallen into disuse. An information and communications technology assisted version of the Personalised System of Instruction using a virtual learning environment is promoted here based on the authors’ longitudinal design research into this pedagogy. The particular elements of the virtual learning environment which are promoted are short video clips, online formative tests and an assessment management system. The authors present their experiences of developing and deploying this pedagogy for the teaching of introductory discrete mathematics to large classes of Computer Science students at two UK higher education institutions both with whole cohorts and ‘at risk’ groups of students. In particular, this method is promoted as particularly helpful to students who do not adopt a deep approach to learning as many students fail to do. Moreover ‘at risk’ students using this method (n = 71) demonstrated an average Glass effect size of 0.83 compared with other ‘at risk’ students who did not (n = 35). Based on these experiences, this pedagogy is promoted as an effective approach to teaching in higher education, especially the teaching of cognitive skills to diverse cohorts of students on foundation level modules.     

Computer-based testing: An alternative for the assessment of Turkish undergraduate students

Virtually errorless high speed data processing feature has made computers popular assessment tools in education. An important concern in developing countries considering integrating computers as an educational assessment tool before making substantial investment is the effects of computer-based testing on students’ test scores as compared to paper-and-pencil tests. This study investigated whether test scores of Turkish students were different in the computer-based test and in the paper-and-pencil test, with forty-seven undergraduate students studying at a public university located in the Blacksea region of Turkey. Findings of this study showed that test scores of undergraduate students were not different in the computer-based test and in the paper-and-pencil test which led us to reach the conclusion that computer-based testing can be considered as a promising alternative technique for the undergraduate students in Turkey.     

Formative e-assessment: Practitioner cases

This paper reports on one aspect of the Joint Information Systems Committee (JISC)-funded project ‘Scoping a vision of formative e-assessment’, namely on cases of formative e-assessment developed iteratively with the UK education practitioner community. The project, which took place from June 2008 to January 2009, aimed to identify current theories and practices relating to formative assessment of learning where technologies play a key role. The project aimed to scope the ‘domain’ of formative e-assessment, by developing cases of practice and identifying key formative processes within them, which are affected by the use of technologies. From this analysis, patterns were extracted to inform future software design. A discussion of the key issues emerging from the review of the literature on formative e-assessment, a full account of the project methodology – the design pattern methodology – as well as a critical discussion of the findings – namely the patterns and the role of technology – are the focus of a separate paper (see Daly, Pachler, Mor, and Mellar (in press). This paper documents how cases of formative e-assessment were developed during the project by a collaborative methodology involving practitioners from a range of post-16 education contexts. The cases were analysed with reference to key theoretical perspectives on formative assessment, particularly the work of Black and Wiliam (2009). In addition, Laurillard’s Conversational Framework (2002, 2007) was used to locate practices of formative assessment within a wider concept of learning and teaching involving technologies, although a detailed discussion of the latter is not within the scope of this paper¹.     

What strategies are effective for formative assessment in an e‐learning environment?

Abstract  The web-based formative assessment developed in this research is named Formative Assessment Module of the Web-based Assessment and Test Analysis System (FAM-WATA). FAM-WATA is a multiple-choice web-based formative assessment module containing six effective strategies: 'repeat the test', 'correct answers are not given', 'query scores', 'ask questions', 'monitor answering history', and 'all pass and then reward'. This research explored the effectiveness of FAM-WATA, cognitive styles and e-learning, and student attitudes towards the six strategies of FAM-WATA. A total of 503 seventh-grade students in central Taiwan were valid in this research. Overall results indicated that students displayed a positive attitude towards the six strategies of FAM-WATA. In addition, results also showed that students in an e-learning environment equipped with FAM-WATA achieved better learning effectiveness, and that field independent students appeared to make better use of FAM-WATA strategies than field dependent students. This research concluded that FAM-WATA benefited student learning in an e-learning environment.     

Student engagement with assessment and feedback: Some lessons from short-answer free-text e-assessment questions

Students were observed directly, in a usability laboratory, and indirectly, by means of an extensive evaluation of responses, as they attempted interactive computer-marked assessment questions that required free-text responses of up to 20 words and as they amended their responses after receiving feedback. This provided more general insight into the way in which students actually engage with assessment and feedback, which is not necessarily the same as their self-reported behaviour. Response length and type varied with whether the question was in summative, purely formative, or diagnostic use, with the question itself, and most significantly with students’ interpretation of what the question author was looking for. Feedback was most effective when it was understood by the student, tailored to the mistakes that they had made and when it prompted students rather than giving the answer. On some occasions, students appeared to respond to the computer as if it was a human marker, supporting the ‘computers as social actors’ hypothesis, whilst on other occasions students seemed very aware that they were being marked by a machine.     

The evaluation of different gaming modes and feedback types on game-based formative assessment in an online learning environment

This study proposed an online learning system for energy education, modifying the typical rules of tic-tac-toe and incorporating multiple choice tests into the game in order to develop a game-based formative assessment tool for an online learning course. In order to explore how different gaming modes and feedback types in this game-based formative assessment affect knowledge acquisition effectiveness and participation perceptions, a tic-tac-toe quiz game (TRIS-Q) with two gaming modes: single-player online game (SOG) and multi-player online game (MOG), and two feedback types: immediate elaborated feedback (IEF) and no immediate elaborated feedback (no IEF), were developed. A 2(SOG vs. MOG) × 2(IEF vs. no IEF) between-subject experiment was also conducted to investigate the effects on 109 ninth-grade students from four junior high school classes. The research findings indicated that different gaming modes of TRIS-Q did not affect the effectiveness of knowledge acquisition; providing IEF for each question answered in the game facilitated the enhancement of both energy knowledge acquisition and student tic-tac-toe ability when comparing it with the no IEF type. Additionally, the different gaming modes and feedback types did not affect participation perceptions.     

A usability comparison of PDA-based quizzes and paper-and-pencil quizzes

In the last few years, schools and universities have incorporated personal digital assistants (PDAs) into their teaching curricula in an attempt to enhance students’ learning experience and reduce instructors’ workload. One of the most common uses of PDAs in the classroom is as a test administrator. This study compared the usability effectiveness, efficiency, and satisfaction of a PDA-based quiz application to that of standard paper-and-pencil quizzes in a university course. Effectiveness was measured as students’ quiz scores and through a mental workload questionnaire; efficiency was the time it took students to complete each quiz; and satisfaction was evaluated using a subjective user satisfaction questionnaire. The study showed the PDA-based quiz to be more efficient, that is, students completed it in less time than they needed to complete the paper-and-pencil quiz. No differences in effectiveness and satisfaction were found between the two quiz types. Computer anxiety was not affected by the quiz type. For these reasons, as well as other advantages to both students (e.g., real-time scoring) and teachers (e.g., less time spent on grading), PDAs are an attractive test administration option for schools and universities.     

A usability comparison of PDA-based quizzes and paper-and-pencil quizzes

In the last few years, schools and universities have incorporated personal digital assistants (PDAs) into their teaching curricula in an attempt to enhance students’ learning experience and reduce instructors’ workload. One of the most common uses of PDAs in the classroom is as a test administrator. This study compared the usability effectiveness, efficiency, and satisfaction of a PDA-based quiz application to that of standard paper-and-pencil quizzes in a university course. Effectiveness was measured as students’ quiz scores and through a mental workload questionnaire; efficiency was the time it took students to complete each quiz; and satisfaction was evaluated using a subjective user satisfaction questionnaire. The study showed the PDA-based quiz to be more efficient, that is, students completed it in less time than they needed to complete the paper-and-pencil quiz. No differences in effectiveness and satisfaction were found between the two quiz types. Computer anxiety was not affected by the quiz type. For these reasons, as well as other advantages to both students (e.g., real-time scoring) and teachers (e.g., less time spent on grading), PDAs are an attractive test administration option for schools and universities.     

Teaching–learning-based optimization: A novel method for constrained mechanical design optimization problems

A new efficient optimization method, called ‘Teaching–Learning-Based Optimization (TLBO)’, is proposed in this paper for the optimization of mechanical design problems. This method works on the effect of influence of a teacher on learners. Like other nature-inspired algorithms, TLBO is also a population-based method and uses a population of solutions to proceed to the global solution. The population is considered as a group of learners or a class of learners. The process of TLBO is divided into two parts: the first part consists of the ‘Teacher Phase’ and the second part consists of the ‘Learner Phase’. ‘Teacher Phase’ means learning from the teacher and ‘Learner Phase’ means learning by the interaction between learners. The basic philosophy of the TLBO method is explained in detail. To check the effectiveness of the method it is tested on five different constrained benchmark test functions with different characteristics, four different benchmark mechanical design problems and six mechanical design optimization problems which have real world applications. The effectiveness of the TLBO method is compared with the other population-based optimization algorithms based on the best solution, average solution, convergence rate and computational effort. Results show that TLBO is more effective and efficient than the other optimization methods for the mechanical design optimization problems considered. This novel optimization method can be easily extended to other engineering design optimization problems.     

Exploration of an e-learning model to foster critical thinking on basic science concepts during work placements

We designed an e-learning model to promote critical thinking about basic science topics in online communities of students during work placements in higher education. To determine the effectiveness and efficiency of the model we explored the online discussions in two case studies. We evaluated the quantity of the interactions by looking at quantitative data of the discussion ‘threads’ and we evaluated the quality of the discussion by content analysis of the individual messages. Both the procedural facilitation of the discussion and the instrument for content analysis were based on Garrison’s ‘Practical Inquiry model of Cognitive Presence’. Furthermore, we explored the experiences of the students and moderators by interviewing them and we organised their perceptions using the framework of an activity system. On the basis of the quantitative and qualitative data we conclude that the e-learning model was successful in establishing a dialogue among a group of students and an expert during work placements at different locations. The ‘Practical Inquiry model’ was useful in facilitating a sustained on-topic discourse involving critical thinking. Although the amount of critical thinking was moderate, the results suggest ways to increase integration and resolution activities in the online discussions.     

Comparing e-Learning tools’ success: The case of instructor–student interactive vs. self-paced tools

e-Learning tools have profoundly transformed modern pedagogical approaches. Vendors provide different types of systems, such as self-paced (SP) and instructor–student interactive (ISI) e-Learning tools. Although both types of tools represent promising solutions to facilitate the learning process, it is important to theoretically identify a framework to evaluate the success of these tools and assess whether one type of tool is more effective than another. Toward this end, we (1) propose a model to evaluate e-Learning tools’ success by extending and contextualizing Seddon’s information systems (IS) success model for the e-Learning environment and (2) formulate four hypotheses to predict the differences in the success factors between SP and ISI tools. We test the model and hypotheses using data from 783 students across seven higher education institutions in Hong Kong. The results support the proposed e-Learning tool success model and three of the four hypotheses. ISI tools outperform SP tools in terms of system quality, perceived usefulness, satisfaction, and learning outcome.     

Impacts and preferences study for e-HO as a holistic learning environment complementary to e-Learning

We present in this study the significance and impacts of an innovative e-HO as a holistic and horizontal platform complementary to e-Learning to help realize transdisciplinary learning and foster integration of knowledge in higher education. A comprehensive investigation of a survey conducted among 647 university students for e-HO is presented with detailed studies on students’ preferences toward e-HO with respect to various demographic variables including gender, access frequency, grade level, discipline and access route concerning the five scales extracted in this particular survey. No significant gender difference surfaced, whereas access frequency turned out to be the most influential and decisive variable that has resulted in significant differences on all the scales involved, especially the characteristic ‘Leisure climate’. Further significant differences appeared on different scales with distinct natures for different demographic variables other than gender and access frequency, indicating the multidimensional characteristics of the whole endeavor and that care must be taken whenever a learning environment like this is constructed in higher education.     

The effects of problem-based learning on the classroom community perceptions and achievement of web-based education students

Web-based education as an alternative to face-to-face teaching is now being extended as an application to higher education. Much effort is undertaken to provide multimedia rich, attractive content to learners. While the use of multimedia technologies has a noticeable effect on students’ learning, so does the selected teaching methodology. Underpinned by the Constructivist approach, problem-based learning is one of the methods that could be applied in web-based learning environments. The current study investigates the effects of problem-based learning on students’ classroom community perceptions and their achievement. It focuses on the tertiary level course “Introduction to Computers” offered by the Department of Business Administration of the Distance Education Program at a higher education institution in Turkey. The results indicated that students who worked on problem-based projects felt much more ‘connected’ to other class members when compared to the control group. They achieved higher scores in the post-tests although their online midterm and final examination scores did not indicate any difference between the groups.     

The effect of person–organization fit feedback via recruitment web sites on applicant attraction

We adopted a person–organization (P–O) fit framework to examine applicant attraction to an organization in the context of Web-based recruitment. A total of 121 undergraduate business students participated in a two-stage study first by completing a paper-and-pencil survey and second, by visiting a fictitious recruitment Web site in which the researchers manipulated the fit feedback information participants received. Levels of participants’ subjective P–O fit and P–O fit feedback information were found to be positively related to attraction. That is, participants with high subjective P–O fit perceptions and participants who received high P–O fit feedback information reported higher applicant attraction. Furthermore, the interaction between P–O fit feedback information and whether the P–O fit feedback information was consistent with participants’ subjective P–O fit was significantly related to applicant attraction. Implications of Web-based recruitment with real-time feedback are offered.     

The automatic assessment of free text answers using a modified BLEU algorithm

e-Learning plays an undoubtedly important role in today’s education and assessment is one of the most essential parts of any instruction-based learning process. Assessment is a common way to evaluate a student’s knowledge regarding the concepts related to learning objectives. In this paper, a new method for assessing the free text answers of students based on the BLEU algorithm is presented. We modify the BLEU algorithm so that it is suitable for assessing free text answers and call the new algorithm the modified BLEU (M-BLEU).