Implementation of Web-based dynamic assessment in facilitating junior high school students to learn mathematics

This research adopts the Graduated Prompting Assessment Module of the WATA system (GPAM-WATA) and applies it to the remedial teaching of junior high school mathematics. The theoretical basis of the development of GPAM-WATA is the idea of ‘graduated prompt approach’ proposed by Campione and Brown, 1985, Campione and Brown, 1987. In GPAM-WATA, when examinees fail to answer items correctly, they obtain instructional prompts (IPs) in a graduated way. This research developed the contents of IPs based on the mathematical problem-solving theory of Mayer (1992, pp. 458–460). A quasi-experimental design was adopted. Ninety-six junior high school seventh graders from three different classes participated in this research. The three classes were randomly divided into the GPAM-WATA group (n = 31), the N-WBT group (n = 31) and the PPT group (n = 34). All students received traditional mathematics instruction from the same teacher. After traditional mathematics instruction, all students took the pre-test of the summative assessment. The students in the three different groups then respectively received remedial teaching in the form of GPAM-WATA, normal Web-based test (N-WBT), and paper-and-pencil test (PPT). After the remedial teaching, all students took the post-test of the summative assessment. The results indicate that compared with other groups, performing remedial teaching using GPAM-WATA has significantly better effectiveness. Moreover, it is found that the IPs in GPAM-WATA are effective in remedial teaching for not only those students most lacking in different types of mathematical problem-solving knowledge but also all the other students.     

Can online course‐based assessment methods be fair and equitable? Relationships between students' preferences and performance within online and offline assessments

To address concerns raised regarding the use of online course‐based summative assessment methods, a quasi‐experimental design was implemented in which students who completed a summative assessment either online or offline were compared on performance scores when using their self‐reported preferred or non‐preferred modes. Performance scores were found not to differ depending on whether the assessment was completed in the preferred or non‐preferred mode. These findings provide preliminary support for the validity of online assessment methods. Future studies could help determine the extent to which this finding generalizes beyond the assessment procedures and type of sample used here. Suggestions for follow‐up studies are offered, including exploring the validity of more complex computer‐related online assessment tasks and investigating the impact of using preferred and non‐preferred modes upon the quality of the student experience.     

Increasing the effectiveness of associative classification in terms of class imbalance by using a novel pruning algorithm

Having received considerable interest in recent years, associative classification has focused on developing a class classifier, with lesser attention paid to the probability classifier used in direct marketing. While contributing to this integrated framework, this work attempts to increase the prediction accuracy of associative classification on class imbalance by adapting the scoring based on associations (SBA) algorithm. The SBA algorithm is modified by coupling it with the pruning strategy of association rules in the probabilistic classification based on associations (PCBA) algorithm, which is adjusted from the CBA for use in the structure of the probability classifier. PCBA is adjusted from CBA by increasing the confidence through under-sampling, setting different minimum supports (minsups) and minimum confidences (minconfs) for rules of different classes based on each distribution, and removing the pruning rules of the lowest error rate. Experimental results based on benchmark datasets and real-life application datasets indicate that the proposed method performs better than C5.0 and the original SBA do, and the number of rules required for scoring is significantly reduced.     

Measuring Information and Communication Technology Literacy using a performance assessment: Validation of the Student Tool for Technology Literacy (ST2L)

This paper reports the validation scores of the Student Tool for Technology Literacy (ST²L), a performance-based assessment based on the National Educational Technology Standards for Students (NETS*S) used to measure middle grade students Information and Communication Technology (ICT) Literacy. Middle grade students (N = 5884) from school districts across the state of Florida were recruited for this study. This paper first provides an overview of various methods to measure ICT literacy and related constructs, and provides documented evidence of score reliability and validity. Following sound procedures based on prior research, this paper provides validity and reliability evidence for the ST²L scores using both item response theory and testlet response theory. This paper examines both the internal and external validity of the instrument. The ST²L, with minimal revision, was found to be a sound measure of ICT literacy for low-stakes assessment purposes. A discussion of the results is provided with emphasis on the psychometric properties of the tool and some practical insights on with whom the tool should be used in future research and practice.     

Toward an integrated model for designing assessment systems: An analysis of the current status of computer-based assessments in science

Drawing upon an integrated model proposed by Bennett and Bejar (1998), this review study examined how 66 computer-based science assessments (CBSAs) in basic science and medicine took advantage of advanced technologies. The model regarded a CBSA as an integrated system, included several assessment components (e.g., assessment purpose, measured construct, test and task design, examinee interface, and scoring procedure), and emphasized the interplay among these components. Accordingly, this study systematically analyzed the item presentations of interactive multimedia, the constructs measured, the response formats in formative and summative assessments, the scoring procedures, the adaptive test activities administrated based on the algorithms related to the item response theory (IRT) and other rules beyond IRT, and the strategies for the automatic provision of informative hints and feedbacks in the CBSAs. Our analysis revealed that although only 19 out of 66 assessments took advantage of dynamic and interactive media for item presentations, CBSAs with these media could measure integrated understanding of science phenomena and complex problem-solving skills. However, we also found that limitations in automated scoring may lead to infrequent use of the automated provision of hints and feedbacks with open-ended and extended responses. These findings suggest the interrelatedness of the assessment components, and thus we argue that designers should repeatedly consider the relationships between components of CBSAs to ensure the validity of the assessments. Finally, we also indicate the issues for future research in computer-based assessments.     

Developing an assessment-centered e-Learning system for improving student learning effectiveness

This research used Web-based two-tier diagnostic assessment and Web-based dynamic assessment to develop an assessment-centered e-Learning system, named the ‘GPAM-WATA e-Learning system.’ This system consists of two major designs: (1) personalized dynamic assessment, meaning that the system automatically generates dynamic assessment for each learner based on the results of the pre-test of the two-tier diagnostic assessment; (2) personalized e-Learning material adaptive annotation, meaning that the system annotates the e-Learning materials each learner needs to enhance learning based on the results of the pre-test of the two-tier diagnostic assessment and dynamic assessment. This research adopts a quasi-experimental design, applying GPAM-WATA e-Learning system to remedial Mathematics teaching of the ‘Speed’ unit in an elementary school Mathematics course. 107 sixth-graders from four classes in an elementary school participated in this research (55 male and 52 female). With each class as a unit, they were divided into four different e-Learning models: (1) the personalized dynamic assessment and personalized e-Learning material adaptive annotation group (n = 26); (2) the personalized dynamic assessment and non-personalized e-Learning material adaptive annotation group (n = 28); (3) the non-personalized dynamic assessment and personalized e-Learning material adaptive annotation group (n = 26); and (4) the non-personalized dynamic assessment and non-personalized e-Learning material adaptive annotation group (n = 27). Before remedial teaching, all students took the prior knowledge assessment and the pre-test of the summative assessment and two-tier diagnostic assessment. Students then received remedial teaching and completed all teaching activities. After remedial teaching, all students took the post-test of the summative assessment and two-tier diagnostic assessment. It is found that compared to the e-Learning models without personalized dynamic assessment, e-Learning models with personalized dynamic assessment are significantly more effective in facilitating student learning achievement and improvement of misconceptions, especially for students with low-level prior knowledge. This research also finds that personalized e-Learning material adaptive annotation significantly affects the percentage of reading time students spend on the e-Learning materials they need to enhance learning. However, it does not appear to predict student learning achievement and improvement of misconceptions.     

Assessing scorecard performance: A literature review and classification

The assessment of scorecard performance in the field of credit scoring is of major relevance to firms. This study presents the first systematic academic literature review of how empirical benchmark studies assess scorecard performance in the field of credit scoring. By analysing 62 comparative studies, this study provides two main contributions. First, this study provides a systematic overview of the assessment-related decisions of all the reviewed studies based on a classification framework. Second, the assessment criteria of consistency, application fit, and transparency are introduced and used to discuss the observed assessment-related decisions. As the findings show, researchers often pay insufficient attention to ensuring the consistent assessment of scorecard performance. Moreover, the majority of the reviewed studies choose performance indicators that failed to fit the application context and provided non-transparent assessment documentation. In conclusion, these researchers pay a great deal of attention to the development of scorecards, but they often fail to implement a straightforward assessment procedure.     

基于二元模糊匹配的编程题智能评分方法

针对传统编程题自动评分方法不能准确衡量学生程序与参考答案之间的接近程度等问题,提出了一种基于二元模糊匹配的编程题智能评分方法。第一元为结构匹配,检测学生程序在变量声明、数据输入、函数调用、控制结构等方面与参考答案的相似程度,目的是快速判断学生程序中是否存在关键的采分点。第二元为词语匹配,首先进行词频统计,以确定每个词的权重。然后,分别构造学生程序与参考答案的向量空间模型,并计算两者的余弦相似度,来作为评判词语相似的依据。最终的分数由结构相似度和词语相似度的加权分数计算得出。由于该二元匹配方法不关心程序中结构/词语的先后顺序和是否为精确表达,因此被称为是模糊的。仿真实验表明,该方法具有不错的评分准确性,尽管与人工评分相比还存在一定的差距,但当试题规模较大时,可以作为人工评分的替代手段。     

The efficacy of a World-Wide Web mediated formative assessment

Abstract  Two studies evaluated the effectiveness of a WWW-based formative assessment package used in undergraduate psychology courses. Students taking on-line multiple-choice tests received instant feedback on areas of weakness and how to address them. In Study 1, students used the package as an integral part of their course syllabus. Level of use correlated with performance in the end-of-course summative assessment. In Study 2, the package was used as an 'optional extra'. Both studies found that students who used the package performed better than those who did not. Such systems may be useful learning tools which students may use to enhance performance.     

Effects of different physical workload parameters on mental workload and performance

The design and evaluation of an occupational task should include an assessment of mental workload, since excessive levels of mental workload can cause errors or delayed information processing. Physically demanding work that is performed concurrently with a cognitive task may impact mental workload by impairing mental processing or decreasing performance. The primary objective of this study was to determine whether there is a differential effect of various types of physical activity on both mental workload and cognitive performance. Objective and subjective assessment tools (heart rate variability and visual analog scale) were used as indicators of mental workload, while correct responses during an arithmetic task reflected levels of performance. Thirty participants (ages 18-24 years) performed a combination of tasks inducing both physical and mental workload. Type of physical effort, frequency of movement, and force exertion level were manipulated to alter the workload associated with the physical activity. Changes in subjective ratings generally corresponded to changes in both performance on the arithmetic task and objective mental workload assessment. Some discrepancies occurred at the highest physical force exertion level as participants perceived an increase in effort to maintain the same level of performance. Further research is needed to determine the force exertion threshold, beyond which the physical effort required interferes with mental workload and/or cognitive performance.

Relevance to industry

Technological advancements have increased the requirement for many workers to execute cognitive tasks concurrently with physical activity. When designing and evaluating such situations it is important to determine the interactive effects of these activities. A simple, uni-dimensional tool is suggested as a screening tool to identify situations requiring excessive or increased mental workload that many degrade performance or place additional stress on the individual.     

Developing Web-based assessment strategies for facilitating junior high school students to perform self-regulated learning in an e-Learning environment

This research refers to the self-regulated learning strategies proposed by Pintrich (1999) in developing a multiple-choice Web-based assessment system, the Peer-Driven Assessment Module of the Web-based Assessment and Test Analysis system (PDA-WATA). The major purpose of PDA-WATA is to facilitate learner use of self-regulatory learning behaviors to perform self-regulated learning and in turn improve e-Learning effectiveness. PDA-WATA includes five main strategies: ‘Adding Answer Notes,’ ‘Stating Confidence,’ ‘Reading Peer Answer Notes,’ ‘Recommending Peer Answer Notes’ and ‘Querying Peers’ Recommendation on Personal Answer Notes’. Using these strategies, examinees are allowed to add answer notes to explain why they chose a certain option as the correct answer and state their confidence in their own answer and answer notes, for peers’ reference. In addition to reading peer answer notes, examinees can also recommend peer answer notes as valuable references. The recommendation information can also be queried by all examinees. Quasi-experimental design was adopted to understand the effectiveness of PDA-WATA in facilitating learner use of self-regulatory learning behaviors to perform self-regulated learning and in improving learner e-Learning effectiveness. Participants were 123 seventh-grade junior high school students from four classes. These four classes were randomly divided into the PDA-WATA group (n = 63) and the N-WBT group (n = 60). Before e-Learning instruction, all students took the pre-test of the Learning Process Inventory (LPI), used to understand how often learners use self-regulatory learning behaviors in the learning process, and the pre-test of the summative assessment. After a two-week e-Learning instruction, the students all took the post-test of the LPI and the summative assessment. Results indicate that students in the PDA-WATA group appear to be more willing to take the Web-based formative assessment than students in the N-WBT group. In addition, PDA-WATA appears to be significantly more effective than N-WBT in facilitating learner use of self-regulatory learning behaviors to perform self-regulated learning and in improving their e-Learning effectiveness. Moreover, this research also finds that in the PDA-WATA group, there is no significant difference between the learning effectiveness of students with a low level of self-regulated learning and students with a high level of self-regulated learning, but similar result cannot be found in the N-WBT group.     

Interactive Rendering of CSG Models

We describe a CSG rendering algorithm that requires no evaluation of the CSG tree beyond normalization and pruning. It renders directly from the normalized CSG tree and primitives described (to the graphics system) by their facetted boundaries. It behaves correctly in the presence of user defined, “near” and “far” clipping planes. It has been implemented on standard graphics workstations using Iris GL¹ and Open GL² graphics libraries. Modestly sized models can be evaluated and rendered at interactive (less than a second per frame) speeds. We have combined the algorithm with an existing B-rep based modeller to provide interactive rendering of incremental updates to large models.     

Web-based dynamic assessment: Taking assessment as teaching and learning strategy for improving students’ e-Learning effectiveness

This research combines the idea of cake format dynamic assessment defined by Sternberg and Grigorenko (2001) and the ‘graduated prompt approach’ proposed by 9 and 10 to develop a multiple-choice Web-based dynamic assessment system. This research adopts a quasi-experimental design to investigate the effectiveness of this Web-based dynamic assessment system (GPAM-WATA) and normal Web-based test (N-WBT). One hundred and sixteen sixth grade elementary students from four classes participated in this research. These four classes were randomly divided into the GPAM-WATA group and N-WBT group. Before e-Learning instruction, all the students took the prior knowledge assessment and the pre-test of the summative assessment. After 2-week e-Learning instruction, all the students took the post-test of the summative assessment. The research findings show that students in the GPAM-WATA group experience better e-Learning effectiveness than those in the N-WBT group. GPAM-WATA is also found to be effective in improving the e-Learning effectiveness of students with low-level prior knowledge.     

Effects of a collaborative annotation method on students’ learning and learning-related motivation and affect

Two studies tested the effectiveness of a web-based collaborative annotation system (Hy-Lighter) for learning comprehension, and learning-related affect and motivation. In an undergraduate course setting, students (N = 27) in study 1, (1) highlighted and annotated selected articles, and (2) highlighted and annotated selected articles and reviewed peer highlights and annotations. In a graduate course setting, students (N = 40) in study 2, (1) highlighted and annotated selected articles, and (2) highlighted and annotated selected articles and reviewed peer highlights and annotations. Control groups in both studies read a hard copy of the articles -without using HyLighter and engaging in its associated annotation practices. The main dependent variables included: (a) performance on quizzes, and (b) a number of affective and motivational variables related to reading assignments and academic success. Although not statistically significant, summative assessment scores were higher for students using HyLigther relative to the ones exposed to conventional instruction. HyLighter use also seemed to be associated with more positive affect in undergraduate students relative to their graduate counterparts. Somewhat equivocal findings between the two studies were attributed to the differential implementation of the software in and outside of the classroom. Recommendations for optimal use and desired outcomes were advanced.     

A blended learning lecture delivery model for large and diverse undergraduate cohorts

A blended learning model was developed to enhance lecture delivery in a large, diverse introductory psychology unit, introducing the use of an online, personalized learning system for lecture preparation and using lecture time to extend students' understanding. Changes to the assessment included diagnostic, formative and summative online quizzes. Using hierarchical multiple regression to account for the variance associated with prior achievement and background knowledge, the results show that students who completed the online formative assessments had significantly higher scores on summative assessment tasks, and that scores were even higher for students who used these resources repeatedly. Changes to future implementations of the model are suggested to improve student engagement in formative assessment, and to facilitate lecturer's use of reports on student progress to focus and improve the quality of discussion in the face-to-face lectures.     

The partition of N-dimensional space,using shells

In the field of pattern recognition there are problems where a decision can be made upon a set of training patterns. A training pattern represents a particular known case of the problem under investigation. Here it is represented as a point in the n-dimensional space Rⁿ, where each coordinate denotes a particular parameter of an observation or measurement. Each pattern point belongs to a single answer, which is the solution of this case. In this way, solved examples of the problem under investigation are represented by points of training patterns in the n-dimensional space. The question arises how to make the decision in order to find the answer for a given unknown pattern, i.e. a case without an answer, upon a set of training patterns used as basic knowledge.By the proposed method, shells are used to divide the n-dimensional space into regions where points of training patterns are located. Each shell has the shape of an n-dimensional rectangle and covers pattern points of the same answer. The partition of the n-dimensional space is achieved first in the adaptation phase, where a single shell belongs to the same answer as the pattern points it covers. The coverage of the space obtained after the adaptation phase is then improved in the following self-adaptation phase. Here, pairs of shells belonging to the same answer are merged into substitute shells. Thus, the number of shells is reduced without damage to the obtained quality of the coverage of the space Rⁿ. Upon such a coverage, a decision can be made for a given pattern by searching a shell which covers its pattern point. The answer belonging to this shell is also the answer for that pattern.The efficiency of this model has been satisfactorily demonstrated in two medical fields: prognosis of acute pancreatitis and in the diagnosis of disseminated cancer of unknown origin.     

E-portfolios supporting primary students' writing performance and peer feedback

E-portfolios show great promise for supporting students' writing performance and facilitating peer feedback, but empirical research of their use in primary education is limited. To address this gap, a yearlong study was conducted in a 4th grade primary class in Cyprus implementing e-portfolios (n = 20) to help students document their progress and send feedback to peers. A generic, open source weblog tool localized into Greek was used as an e-portfolio tool. Data sources included 176 student-essays, 1306 instances of peer feedback, students pre- and post-tests on writing performance, nine videotaped student interviews and one teacher interview. A paired samples t-test analysis showed that there was a statistically significant difference between students' pre-test (M = 62.6, SD = 9.82, n = 20) and post-test (M = 76.5, SD = 12.6, n = 20) on writing performance (t (19) = −8.03, p < .01). A repeated measures analysis of variance on e-portfolio artifacts showed that there were learning gains with respect to students' writing performance over time [F(4.01, 76.25) = 31.59, p < .01, η² = 0.89]. A qualitative analysis of students' comments showed that students provided more thorough peer feedback over time and became gradually more capable of providing corrective feedback. The qualitative analysis of student interviews provided evidence that students valued peer feedback in their portfolios. The analysis of the teacher interview showed that average and high-ability students benefited the most from peer feedback. This study provided evidence that e-portfolios can support the development of students' writing performance and peer feedback skills in the context of primary school essay writing. Instructional implications for portfolio implementation are offered.     

Student engagement with assessment and feedback: Some lessons from short-answer free-text e-assessment questions

Students were observed directly, in a usability laboratory, and indirectly, by means of an extensive evaluation of responses, as they attempted interactive computer-marked assessment questions that required free-text responses of up to 20 words and as they amended their responses after receiving feedback. This provided more general insight into the way in which students actually engage with assessment and feedback, which is not necessarily the same as their self-reported behaviour. Response length and type varied with whether the question was in summative, purely formative, or diagnostic use, with the question itself, and most significantly with students’ interpretation of what the question author was looking for. Feedback was most effective when it was understood by the student, tailored to the mistakes that they had made and when it prompted students rather than giving the answer. On some occasions, students appeared to respond to the computer as if it was a human marker, supporting the ‘computers as social actors’ hypothesis, whilst on other occasions students seemed very aware that they were being marked by a machine.     

A flexible quantitative method for NC machining verification using a space-division based solid model

A geometric-modeling method called Graftree and a system design of an NC (numerical control) machining verifier based on Graftree are proposed. Graftree is constructed by combining Oct-tree and constructive solid geometry (CSG) so as to simulate machining processes precisely in three-dimensional space. Using Graftree, the Boolean operation produces no risk of yielding topological conflicts, which often cause the simulation to stop abnormally. Based on the properties of Graftree, an NC machining verifier is designed to consist of three individual subsystems: data-input operations, geometric simulation, and interactive verification, i.e., visuluatzation of machining scenes and evaluation of the machined product using models of the measuring instruments. This design lets the users watch selectively only the machining steps they want to check precisely, and verify the machining according to dimensional tolerance.