Mobile formative assessment tool based on data mining techniques for supporting web-based learning

Current trends clearly indicate that online learning has become an important learning mode. However, no effective assessment mechanism for learning performance yet exists for e-learning systems. Learning performance assessment aims to evaluate what learners learned during the learning process. Traditional summative evaluation only considers final learning outcomes, without concerning the learning processes of learners. With the evolution of learning technology, the use of learning portfolios in a web-based learning environment can be beneficially adopted to record the procedure of the learning, which evaluates the learning performances of learners and produces feedback information to learners in ways that enhance their learning. Accordingly, this study presents a mobile formative assessment tool using data mining, which involves six computational intelligence theories, i.e. statistic correlation analysis, fuzzy clustering analysis, grey relational analysis, K-means clustering, fuzzy association rule mining and fuzzy inference, in order to identify the key formative assessment rules according to the web-based learning portfolios of an individual learner for the performance promotion of web-based learning. Restated, the proposed method can help teachers to precisely assess the learning performance of individual learner utilizing only the learning portfolios in a web-based learning environment. Hence, teachers can devote themselves to teaching and designing courseware, since they save a lot of time in measuring learning performance. More importantly, teachers can understand the main factors influencing learning performance in a web-based learning environment based on the interpretable learning performance assessment rules obtained. Experimental results indicate that the evaluation results of the proposed scheme are very close to those of summative assessment results and the factor analysis provides simple and clear learning performance assessment rules. Furthermore, the proposed learning feedback with formative assessment can clearly promote the learning performances and interests of learners.     

The development of a formative scenario-based computer assisted assessment tool in psychology for teachers: The PePCAA project

Formative computer assisted assessment has become increasingly attractive in Higher Education where providing useful feedback to large numbers of students can be difficult. However, the nature of such assessments has often been limited to objective questions such as multiple-choice. This paper reports on the development and initial trialling of a more innovative, formative use of computer assisted assessment in a Higher Education context. The European funded PePCAA (Pedagogical Psychology Computer Assisted Assessment) project developed a series of scenario-based computer-delivered formative assessments of pedagogical psychology for teachers and trainee teachers, using a range of software features, including the addition of confidence measurement. The project had a two-fold aim: to provide a tool to improve understanding of pedagogical psychology and to explore the potential of more innovative techniques of computer assisted assessment to motivate students and to assess deeper learning. The combination of computer-delivered formative assessment with innovative question styles and confidence ratings is believed to be unique for pedagogical psychology. Scenarios were based on realistic classroom situations and focused on problem solving or on utilising best practice. The PePCAA Learning Assessment Circle (PLAC) provided a framework for indexing the kinds of processes required of users. In the UK, small scale trialling involved a total of 23 teacher trainees such that each assessment was attempted by about seven participants. Participants completed evaluation questionnaires after each assessment. Responses from learners indicated that the UK scenarios were generally very well received and had at least partly achieved the aim of stimulating deeper learning. Transfer of assessments between countries proved more difficult than expected. The next stage of development should be to conduct a larger pilot, thus allowing full investigation of the reliability and validity of the assessments. There is also scope for further development of the PePCAA approach and for its application in other subjects.     

What strategies are effective for formative assessment in an e‐learning environment?

Abstract  The web-based formative assessment developed in this research is named Formative Assessment Module of the Web-based Assessment and Test Analysis System (FAM-WATA). FAM-WATA is a multiple-choice web-based formative assessment module containing six effective strategies: 'repeat the test', 'correct answers are not given', 'query scores', 'ask questions', 'monitor answering history', and 'all pass and then reward'. This research explored the effectiveness of FAM-WATA, cognitive styles and e-learning, and student attitudes towards the six strategies of FAM-WATA. A total of 503 seventh-grade students in central Taiwan were valid in this research. Overall results indicated that students displayed a positive attitude towards the six strategies of FAM-WATA. In addition, results also showed that students in an e-learning environment equipped with FAM-WATA achieved better learning effectiveness, and that field independent students appeared to make better use of FAM-WATA strategies than field dependent students. This research concluded that FAM-WATA benefited student learning in an e-learning environment.     

Implementing computer-based assessment - A web-based mock examination changes attitudes

Interest in the educational use of information and communication technologies (ICT) in higher education is increasing. For successful implementation, it is important to know students’ attitudes and reservations and how they can be positively influenced. The objective of the present study was to examine possible attitudinal changes towards computer-based assessment (CBA) in students, after undergoing one such assessment. A web-based mock examination was provided to all fourth year medical students at Leipzig Medical School in 2008 and 2009. Before and after the web test, students were asked to document their agreement with statements concerning CBA. A large number of students made use of the offered web-based assessment. 383 participants could be analysed for the pre-post comparisons. The majority of the students rated their computer self-efficacy as high. In summary, students’ attitudes towards CBA in higher education tended to be positive. Gender differences seemed to be substantially influenced by differences in computer self-efficacy and were reduced considerably after only one practical experience. The actual experience had a positive influence on attitudes towards CBA. Nevertheless there were strong reservations about technical problems influencing the test performance when used for summative assessment. These concerns should not be ignored when trying to implement CBA. Optional formative CBA, perhaps early in higher education, seems to be a promising possibility of attracting students to computer- or web-based examination and learning methods, and may be a useful component of a successful implementation strategy.     

A blended learning lecture delivery model for large and diverse undergraduate cohorts

A blended learning model was developed to enhance lecture delivery in a large, diverse introductory psychology unit, introducing the use of an online, personalized learning system for lecture preparation and using lecture time to extend students' understanding. Changes to the assessment included diagnostic, formative and summative online quizzes. Using hierarchical multiple regression to account for the variance associated with prior achievement and background knowledge, the results show that students who completed the online formative assessments had significantly higher scores on summative assessment tasks, and that scores were even higher for students who used these resources repeatedly. Changes to future implementations of the model are suggested to improve student engagement in formative assessment, and to facilitate lecturer's use of reports on student progress to focus and improve the quality of discussion in the face-to-face lectures.     

A web-based curriculum-based measurement system for class-wide ongoing assessment

Abstract   Internet technology has offered opportunities to develop ongoing assessment systems for classroom-based evaluation – on a daily basis. In this study, the researcher developed a web-based, curriculum-based measurement system with dynamic features which could generate different types of mathematics probes, track students' progress and provide diagnosed information as well as instructional suggestions for teachers. This paper explores the effects of the system on students' mathematical achievements. A total of 134 third-grade students (9- to 10-year-olds) in four classes participated. The teachers in all groups used the web-based curriculum-based measurement system with different types of curriculum-based measurement probes and growth modeling. The results indicated that the use of class-wide dynamic-growth modeling combined with mixed-type probes enabled the students to perform better than those using single-type probes. This outcome was not seen with the linear-growth modeling groups. The positive findings suggested that applying class-wide dynamic-growth modeling as well as the assessment of integrated mathematics competency in the instructional processes facilitated students' mathematics learning. Therefore, the web-based, curriculum-based measurement system was not only an assessment system, but also a tool for teachers to integrate instructional strategies based on curriculum-based measurement.     

Implementing virtual collaborative inquiry practises in a middle-school context

The aim of the present study was to investigate the challenges that relate to the implementation of virtual inquiry practises in middle school. The case was a school course in which a group of Finnish students (N = 14) and teachers (N = 7) completed group inquiries through virtual collaboration, using a web-based learning environment. The task was to accomplish a cross-disciplinary inquiry into cultural issues. The students worked mainly at home and took much responsibility for their course achievements. The investigators analysed the pedagogical design of the course and the content of the participants' interaction patterns in the web-based environment, using qualitative content analysis and social network analysis. The findings suggest that the students succeeded in producing distinctive cultural products, and both the students and the teachers adopted novel roles during the inquiry. The web-based learning environment was used more as a coordination tool for organizing the collaborative work than as a forum for epistemic inquiry. The tension between the school curriculum and the inquiry practises was manifest in the participants' discussions of the assessment criteria of the course.     

Online formative assessment in higher education: A review of the literature

As online and blended learning has become common place educational strategy in higher education, educators need to reconceptualise fundamental issues of teaching, learning and assessment in non traditional spaces. These issues include concepts such as validity and reliability of assessment in online environments in relation to serving the intended purposes, as well as understanding how formative assessment functions within online and blended learning. This article provides a systematic qualitative review of the research literature on online formative assessment in higher education. As an integrative narrative review, the method applied in this review entailed systematic searching, reviewing, and writing this review of the literature to bring together key themes and findings of research in this field. The authors applied qualitative thematic criteria in selecting and reviewing the available literature from which they focused on identifying and analyzing the core themes that are central to the concept of formative assessment with a key focus on application of formative assessment within blended and online contexts. Various techniques were identified for formative assessment by the individual, peers and the teacher, many of which were linked with online tools such as self-test quiz tools, discussion forums and e-portfolios. The benefits identified include improvement of learner engagement and centrality in the process as key actors, including the development of a learning community. The key findings are that effective online formative assessment can foster a learner and assessment centered focus through formative feedback and enhanced learner engagement with valuable learning experiences. Ongoing authentic assessment activities and interactive formative feedback were identified as important characteristics that can address threats to validity and reliability within the context of online formative assessment.     

Computer support of effective peer assessment in an undergraduate programming class

Abstract Active learning is considered by many academics as an important and effective learning strategy. Assessment is integrated in learning as a tool for learning, but traditional assessment methods often encourage surface learning (passive learning) rather than deep learning (active learning). Peer assessment is a method of motivating students, involving students discussing, marking and providing feedback on other students' work, and is one of the successful approaches which can be used to enhance deep learning. Students are required to think critically about what they are learning during the peer assessment process. Tutors' marking is usually accepted as reliable, but student peers' marking in a peer assessment process is suspect. As part of a study investigating whether peer assessment can be an accurate assessment method in a computer programming course, a novel web-based peer assessment tool has been developed. In this paper, we describe the tool and report the results of evaluating the tool through experiments involving large programming classes. The results suggest that computer-mediated peer assessment is a valuable assessment approach which promotes active learning and is an accurate assessment method in a programming course.     

Collaborative learning as a collective competence when students use the potential of meaning in asynchronous dialogues

The aim of this study is to examine and to describe how student teachers engaged in courses in web-based learning environments over a period of 40-weeks develop a collective competence to collaborate. The collective competence of collaboration is defined as the level of learning ability a group of students express when using dialogues as a tool for their own and other’s learning in a web-based learning environment. The students’ contributions to the course assignments, the group responses and the collaborative discussions and dialogues were analysed and interpreted based on Bakhtin’s and Rommetveit’s theories on dialogic interactions and meaning potentials. The results describe three different levels at which students use dialogues as a tool for learning when they collaborate within the group.     

The design and evaluation of a computerized adaptive test on mobile devices

The use of computerized adaptive testing (CAT) has expanded rapidly over recent years mainly due to the advances in communication and information technology. Availability of advanced mobile technologies provides several benefits to e-learning by creating an additional channel of access with mobile devices such as PDAs and mobile phones. This paper describes the design issues that were considered for the development and the implementation of a CAT on mobile devices, the computerized adaptive test on mobile devices (CAT-MD). Throughout the development of the system, formative evaluation was an integral part of the design methodology. The recommendations, suggestions and the results of the formative evaluation were used to improve the system in order to make the assessment procedure more effective and efficient. These results of the formative evaluation are also presented here.     

Learning styles and formative assessment strategy: enhancing student achievement in Web-based learning

Abstract The purpose of this research was to investigate the effects of formative assessment and learning style on student achievement in a Web-based learning environment. A quasi-experimental research design was used. Participants were 455 seventh grade students from 12 classes of six junior high schools. A Web-based course, named BioCAL, combining three different formative assessment strategies was developed. The formative assessment strategies included Formative Assessment Module of the Web-Based Assessment and Test Analysis system (FAM-WATA) (with six Web-based formative assessment strategies), Normal Module of Web-Based Assessment and Test Analysis system (N-WATA) (only with partial Web-based formative assessment strategy) and Paper and Pencil Test (PPT) (without Web-based formative assessment strategy). Subjects were tested using Kolb's Learning Style Inventory, and assigned randomly by class into three groups. Each group took Web-based courses using one of the formative assessment strategies. Pre- and post-achievement testing was carried out. A one-way ANCOVA analysis showed that both learning style and formative assessment strategy are significant factors affecting student achievement in a Web-based learning environment. However, there is no interaction between these two factors. A post hoc comparison showed that performances of the FAM-WATA group are higher than the N-WATA and PPT groups. Learners with a 'Diverger' learning style performed best followed by, 'Assimilator', 'Accommodator', and 'Converger', respectively. Finally, FAM-WATA group students are satisfied with six strategies of the FAM-WATA.     

Improving student performance in a first-year geography course: Examining the importance of computer-assisted formative assessment

The main objective of this paper is to examine the effectiveness of computer-assisted formative assessment in a large, first-year undergraduate geography course. In particular, the paper evaluates the impact of computer-assisted multiple-choice practice tests on student performance in the course as well as student opinions of this type of formative assessment in two academic years (2008 and 2009). The multiple-choice questions included in the formative assessment vary in their level of difficulty and range from those that focus on knowledge and comprehension to those that focus on application and analysis. While the use of the computer-assisted practice tests is completely voluntary over 50 percent of students used them. Feedback questionnaires from both academic years reveals that students are overwhelmingly positive with over 95 percent indicating that the computer-assisted practice tests assist them in identifying their strengths and weaknesses and help them prepare for in-class midterms and final exams. Statistical analysis of in-class performance on midterms shows that students who use the computer-assisted practice quizzes earn significantly higher grades (i.e., equivalent to three letter grades) than those students who do not. The results of the research demonstrate that computer-assisted formative assessment (in this case practice tests) has a positive impact on student performance.     

The evaluation of different gaming modes and feedback types on game-based formative assessment in an online learning environment

This study proposed an online learning system for energy education, modifying the typical rules of tic-tac-toe and incorporating multiple choice tests into the game in order to develop a game-based formative assessment tool for an online learning course. In order to explore how different gaming modes and feedback types in this game-based formative assessment affect knowledge acquisition effectiveness and participation perceptions, a tic-tac-toe quiz game (TRIS-Q) with two gaming modes: single-player online game (SOG) and multi-player online game (MOG), and two feedback types: immediate elaborated feedback (IEF) and no immediate elaborated feedback (no IEF), were developed. A 2(SOG vs. MOG) × 2(IEF vs. no IEF) between-subject experiment was also conducted to investigate the effects on 109 ninth-grade students from four junior high school classes. The research findings indicated that different gaming modes of TRIS-Q did not affect the effectiveness of knowledge acquisition; providing IEF for each question answered in the game facilitated the enhancement of both energy knowledge acquisition and student tic-tac-toe ability when comparing it with the no IEF type. Additionally, the different gaming modes and feedback types did not affect participation perceptions.     

A case study of Augmented Reality simulation system application in a chemistry course

The comprehension of micro-worlds has always been the focus and the challenge of chemistry learning. Junior high school students’ imaginative abilities are not yet mature. As a result, they are not able to visualize microstructures correctly during the beginning stage of chemistry learning. This study targeted “the composition of substances” segment of junior high school chemistry classes and, furthermore, involved the design and development of a set of inquiry-based Augmented Reality learning tools. Students could control, combine and interact with a 3D model of micro-particles using markers and conduct a series of inquiry-based experiments. The AR tool was tested in practice at a junior high school in Shenzhen, China. Through data analysis and discussion, we conclude that (a) the AR tool has a significant supplemental learning effect as a computer-assisted learning tool; (b) the AR tool is more effective for low-achieving students than high-achieving ones; (c) students generally have positive attitudes toward this software; and (d) students’ learning attitudes are positively correlated with their evaluation of the software.     

Authoring diagram-based CBA with CourseMarker

The CourseMarker system has been used to assess free-response computer based assessment (CBA) exercises since 1998. The aim of the studies reported here was to evaluate the feasibility and usefulness of developing and deploying diagram-based exercises using DATsys, an authoring environment for diagram-based CBA, together with CourseMarker. Postgraduate students constructed diagram-based exercises in four domains. The process of constructing the exercises was captured as an indicator of feasibility. The exercises were then used to assess two cohorts of undergraduate students. Instruments including system submission logs and student questionnaires were used to assess usefulness.Findings indicate that there is considerable potential for the assessment of free-response domains such as diagrams. Such an approach can help students as part of an iterative process of learning by allowing repeated submission of coursework, which may be most appropriate within a formative assessment context. The exercises are popular with students and demonstrate a gradual, though decelerating, increase in marks over subsequent submissions. The techniques are reliable, but further development allowing for alternative model solutions and assessment of the aesthetic appearance of diagrams would increase validity. Our techniques and findings are novel for CBA, and have implications for the increasingly important research area of formative assessment.     

Unifying heterogeneous e-learning modalities in a single platform: CADI, a case study

Current e-learning forms are commonly based on improving the learning process through the enhancement of certain skills in students, such as collaborative, competitive or problem-based learning. However, it seems that there is still no e-learning formula that gathers the implementation of a number of more generic educational principles in a single e-learning system or platform. Therefore, this paper presents a web-based framework for the creation, development and implementation of heterogeneous learning environments called CADI. It offers a graphical user interface for students to interact with only the need for any generic WiFi compliant device. Its innovative design allows the instructor to select and combine the developed resources to create varied activities, providing content adapted to every single device. The usefulness of the system was tested by introducing a case study to a group of senior college students. The experience showed that the use of the system helped students to achieve better results in their evaluation. It also showed that the students had no problems interacting with it from the outset, promoting their active learning. Design, implementation and assessment results of the system are also presented and analyzed.     

The design and the formative evaluation of a web-based course for simulation analysis experiences

Simulation output analysis has received little attention comparing to modeling and programming in real-world simulation applications. This is further evidenced by our observation that students and beginners acquire neither adequate details of knowledge nor relevant experience of simulation output analysis in traditional classroom learning. With the advances in the Internet learning environment, web-based computer-based training (CBT) can efficiently and effectively enhance the traditional classroom learning. This research addresses the issue of designing a web-based CBT course to assist the classroom instruction in helping students learn better simulation problem solving. The goal is to provide an effective way to learn the implicit experience of simulation output analysis via an efficient interactive environment. A prototype asynchronous web CBT system with simulation analysis guidelines was built to verify our design through a three-stage formative usability evaluation. The results showed that the simulation analysis guidelines organized by this research significantly improved the test scores and the interactive CBT system maintained students’ interests longer than the written format.     

A crowdsourcing method to develop virtual human conversational agents

Educators in medicine, psychology, and the military want to provide their students with interpersonal skills practice. Virtual humans offer structured learning of interview skills, can facilitate learning about unusual conditions, and are always available. However, the creation of virtual humans with the ability to understand and respond to natural language requires costly engineering by conversation knowledge engineers (generally computer scientists), and incurs logistical cost for acquiring domain knowledge from domain experts (educators). We address these problems using a novel crowdsourcing method entitled Human-centered Distributed Conversational Modeling. This method facilitates collaborative development of virtual humans by two groups of end-users: domain experts (educators) and domain novices (students). We implemented this method in a web-based authoring tool called Virtual People Factory. Using Virtual People Factory, medical and pharmacy educators are now creating natural language virtual patient interactions on their own. This article presents the theoretical background for Human-centered Distributed Conversational Modeling, the implementation of the Virtual People Factory authoring tool, and five case studies showing that Human-centered Distributed Conversational Modeling has addressed the logistical cost for acquiring knowledge.     

A blended E-learning experience in a course of object oriented programming fundamentals

In this paper, we present a blended e-learning experience consisting of supplying an undergraduate student population (in addition to traditional on-site classes) with a learning tool called OOPS (Object Oriented Programming System) and a testing system called SIETTE. OOPS is a problem-solving environment in which students can resolve Object Oriented Programming exercises. The system applies an assessment for learning strategy where students are formatively assessed, i.e. OOPS diagnoses their knowledge level but also generates feedback and hints to help students to understand and overcome their misconceptions and to reinforce correctly learnt concepts. In conjunction with OOPS, we have used SIETTE, a web-based assessment system in which students can take tests and teachers can construct them Subsequently, we have explored whether or not the use of OOPS contributes to improve the students’ knowledge about Object Oriented Programming.