The design and application of a web-based self- and peer-assessment system

This study describes the web-based self- and peer-assessments system, or the Web-SPA, which has been shown to provide teachers with a flexible interface with which to arrange various self- and peer-assessment procedures. Secondly, this study examines the effects of the application of the progressively focused self- and peer-assessment (PFSPA) procedures that are made available with the Web-SPA. In an evaluative study with 76 third-graders (14 or 15 years old) of junior school, it has been found that the upon completion of the PFSPA activities, the students demonstrate greater objectivity in their self-assessment scores; it has also been shown that significant consistency is found between the students’ self- and peer-assessments results and that the assessment results of teachers. Furthermore, the quality of the students’ works improved after the assessment activities.     

Multiple peer-assessment modes to augment online student question-generation processes

In view of the current theoretical and empirical support for a student-generated questions approach to learning along with the advantageous features of network technology, several online student question-generation learning systems with a peer-assessment component have been developed. Despite this, all existing systems are limited in terms of the types of communication modes permissible for peer-assessment. Online discourse experience and the quantity and quality of interaction may vary as a result of the specific interaction mode students are exposed to. Because of this and the fact that versatile learning spaces are both possible and potentially desirable during the various stages of learning and teaching, multiple peer-assessment modes were created, and the overall attitudes of learners toward peer-assessment as well as their preferences toward respective peer-assessment modes were examined. The collected data confirmed the perceived usefulness of peer-assessment for developing higher-order thinking and cognitive elaboration. Support was demonstrated for student question-generation activities, for which learners typically have limited prior experience. Additionally, in light of the apparent overwhelming preference for and superiority of the more interactive two-way and multi-way modes over the one-way mode and the perceived learning potential of these modes, it is suggested that designers of similar systems should consider their inclusion. Finally, even though multi-way learning was found to be both the preferred and most supportive mode for learning, students also revealed distinct reasons for their preferences for respective interaction modes. In general, these reasons supported the premise that multiple peer-assessment modes are needed in order to accommodate individual preferences and needs. Suggestions and implications for instructional implementation, system development, and future studies are offered.     

A comparative analysis of the consistency and difference among teacher-assessment, student self-assessment and peer-assessment in a Web-based portfolio assessment environment for high school students

This study explored the consistency and difference of teacher-, student self- and peer-assessment in the context of Web-based portfolio assessment. Participants were 72 senior high school students enrolled in a computer application course. Through the assessment system, the students performed portfolio creation, inspection, self- and peer-assessment; three teachers reviewed portfolios and evaluated learning performances. There were significant differences in the results of the three assessment methods, among which teacher-raters adopted the most rigorous scoring standards, while peer-raters tended to use the most lax standards. The results of self- and teacher-assessment were discovered to be consistent; however, consistency was not discovered between self- and peer-assessment as well as peer- and teacher-assessment. In analyzing their consistency with the end-of-course examination, teacher- and self-assessment demonstrated high consistency, whereas peer-assessment showed a low level of consistency.     

Reliability and validity of Web-based portfolio peer assessment: A case study for a senior high school’s students taking computer course

This study examined the reliability and validity of Web-based portfolio peer assessment. Participants were 72 second-grade students from a senior high school taking a computer course. The results indicated that: 1) there was a lack of consistency across various student raters on a portfolio, or inter-rater reliability; 2) two-thirds of the raters demonstrated inconsistency assessing different portfolios, i.e. inner-rater reliability; 3) peer-assessment scores were not consistent with teacher-assessment scores (criterion-related validity); 4) significant differences were found between peer-assessment scores and end-of-course examination scores, implying that Web-based portfolio peer assessment failed to reflect learning achievements (criterion-related validity). In short, Web-based portfolio peer assessment was not a reliable and valid method.     

An interactive peer-assessment criteria development approach to improving students' art design performance using handheld devices

Engaging students in higher order thinking such as evaluation and analysis has been recognized as being an important strategy for helping them develop knowledge and skills, in particular, in creativeness-oriented learning activities such as artwork design. In this paper, an interactive peer-assessment criteria development approach is proposed to help students develop assessment criteria, learn from viewing peers' work, and make reflections in artwork design activities using mobile devices. To evaluate the effectiveness of the proposed approach, an experiment was conducted in an elementary school art course. A total of 103 students participating in the experiment were assigned to an experimental group and a control group. The students in the experimental group learned with the proposed approach, while those in the control group learned with the conventional peer assessment approach. From the experimental results, it was found that the proposed approach significantly improved the students' learning achievement, learning motivation and meta-cognitive awareness, suggesting the effectiveness of engaging students in assessment criteria development in an interactive manner.     

Different identity revelation modes in an online peer-assessment learning environment: Effects on perceptions toward assessors,classroom climate and learning activities

The effects of four different identity revelation modes (three fixed modes: real-name, anonymity, nickname and one dynamic user self-choice mode) on participants’ perceptions toward their assessors, classroom climate, and past experience with the learning activity in which they were engaged were examined. A pretest–posttest quasi-experimental research design was adopted. Eight fifth-grade classes (age 10–11, N = 243) were randomly assigned to four different identity revelation modes in order for them to participate in the study. An online learning system that allows students to contribute to and benefit from the process of question-generation and peer-assessment was adopted. Data analysis confirmed that different identity modes lead participants to view their assessors differently. Specifically, participants assigned to the self-choice and real-name identity revelation modes tended to view their assessors more favorably than those in the anonymity and nickname groups. The empirical significance of the study as well as suggestions for learning system development, instructional implementation and future study are provided.     

Design and evaluation of a microprocessor course combining three cooperative methods: SDLA,PjBL and CnBL

This paper reports on the design, implementation and assessment of a new approach course structure based on the combination of three cooperative methodologies. The main goal is to reduce the percentage of non-passed students focusing the learning process on students by offering different alternatives and motivational activities based on working in small groups on a robotic project and changing the course assessment. The microprocessor course is a subject in the undergraduate program of telecommunication engineering at the University of the Balearic Islands (UIB) in Spain. Its traditional teaching methodology was based on Lecture sessions, Problem sessions and Practical activities in the lab (L/P/P). This methodology was changed to comply with the Bologna requirements, where new skills must be addressed and learned during the educational process. Self-Directed Learning activities (SDLA) for lecture sessions and Project based learning (PjBL) for Laboratory sessions were implemented in combination with a final competition activity between groups (CnBL). This paper describes the design, implementation and results; showing the benefits of these new teaching methods, improving student’s motivation including a robotic buggy project and allowing new learning strategies based on a practical point of view and adaptability to students’ previous skills.     

QuesTInSitu: From tests to routes for assessment in situ activities

Test-based assessment tools are mostly focused on the use of computers. However, advanced Information and Communication Technologies, such as handheld devices, opens up the possibilities of creating new assessment scenarios, increasing the teachers’ choices to design more appropriate tests for their subject areas. In this paper we use the term Computing-Based Testing (CBT) instead of Computer-Based Testing, as it captures better the emerging trends. Within the CBT context, the paper is centered on proposing an approach for “Assessment in situ” activities, where questions have to be answered in front of a real space/location (situ). In particular, we present the QuesTInSitu software implementation that includes both an editor and a player based on the IMS Question and Test Interoperability specification and GoogleMaps. With QuesTInSitu teachers can create geolocated questions and tests (routes), and students can answer the tests using mobile devices with GPS when following a route. Three illustrating scenarios and the results from the implementation of one of them in a real educational situation show that QuesTInSitu enables the creation of innovative, enriched and context-aware assessment activities. The results also indicate that the use of mobile devices and location-based systems in assessment activities facilitates students to put explorative and spatial skills into practice and fosters their motivation, reflection and personal observation.     

Searching the Web for conflicting topics: Page and user factors

Web users tend to search only the pages displayed at the top of the search engine results page (the ‘top link’ heuristic). Although it might be reasonable to use this heuristic to navigate simple and unambiguous facts, it might be risky when searching for conflicting socio-scientific topics, such as potential measures to reduce greenhouse gas emissions. In the present study, we explored the extent to which students consider other Web page characteristics, such as topic relevance and trustworthiness, when searching and bookmarking pages concerning a conflicting topic. We also examined the extent to which prior background knowledge moderates students’ behavior. The results revealed that while the study participants actually used a ‘top link’ heuristic to navigate the results, they engaged in more systematic processes to bookmark pages for further study. Furthermore, the students’ background knowledge was related to the assessment of Web page trustworthiness. We discuss these results from the perspective of a dual-processing model.     

Web-based self- and peer-assessment of teachers’ digital competencies

Although there exist several alternative frameworks and standards for describing the digital competencies expected from teachers, there is a lack of Web-based assessment tools that allow authentic, reliable and valid assessment of these competencies. This paper addresses the design challenges related to a software solution for self- and peer-assessment of teachers’ digital competencies. The empirical part of the paper describes the participatory design process and results from the first user testing of a Web-based self- and peer-assessment tool DigiMina, which supports teachers in building and sharing a personal competency profile. In DigiMina, the competencies are assessed by a teacher herself or by her peers using the performance indicators that are based on the competency model NETS for Teachers created by the International Society of Technology in Education.     

Web-based peer assessment: feedback for students with various thinking-styles

Abstract This study used aptitude treatment interaction design to examine how feedback formats (specific vs. holistic) and executive thinking styles (high vs. low) affect web-based peer assessment. An Internet-based (anonymous) peer-assessment system was developed and used by 58 computer science students who submitted assignments for peer review. The results indicated that while students with high executive thinking styles significantly improved over two rounds of peer assessment, low executive students did not improve through the cycles. In addition, high executive students contributed substantially better feedback than their low executive counterparts. In the second round of peer assessment, thinking style and feedback format interactively affected student learning. Low executive students receiving specific feedback significantly outperformed those receiving holistic feedback. In receiving holistic feedback, high executive thinkers outperformed their low executive counterparts. This study suggests that future web-based peer assessment adopts a specific feedback format for all students.     

A comparative analysis of the consistency and difference among online self-, peer-, external- and instructor-assessments: The competitive effect

In the last few years, self- and peer-assessment have been increasingly employed not only as an evaluation method, but also as a learning procedure. The consistency and difference between self- and peer-assessments as compared to instructor-assessments have been previously studied, and a friendship bias was discovered. In this study, we introduce external-assessment (products are assessed by students from a different university that are enrolled in a similar course), and compare self-, peer-, external- and instructor-assessments. The experience was conducted at two different universities separated by a significant distance, during two consecutive years, including a total of 97 students. At both universities, students developed websites and online tools were employed to organise the different types of assessments. The obtained results indicate that there is a high-level of consistency across the different kinds of assessments. Moreover, a competitive effect was discovered: students tended to award higher grades to students from their same university while they were harsher with the products from a distant university. From the learning perspective, and according to the students' final grade, the assessment experience correlated with learning gains.     

Web-based quiz-game-like formative assessment: Development and evaluation

This research aims to develop a multiple-choice Web-based quiz-game-like formative assessment system, named GAM-WATA. The unique design of ‘Ask-Hint Strategy’ turns the Web-based formative assessment into an online quiz game. ‘Ask-Hint Strategy’ is composed of ‘Prune Strategy’ and ‘Call-in Strategy’. ‘Prune Strategy’ removes one incorrect option and turns the original 4-option item into a 3-option one. ‘Call-in Strategy’ provides the rate at which other test takers choose each option when answering a question. This research also compares the effectiveness of three different types of formative assessment in an e-Learning environment: paper-and-pencil test (PPT), normal Web-based test (N-WBT) and GAM-WATA. In total, 165 fifth grade elementary students (from six classes) in central Taiwan participated in this research. The six classes of students were then divided into three groups and each group was randomly assigned one type of formative assessment. Overall results indicate that different types of formative assessment have significant impacts on e-Learning effectiveness and that the e-Learning effectiveness of the students in the GAM-WATA group appears to be better. Students in the GAM-WATA group more actively participate in Web-based formative assessment to do self-assessment than students in the N-WBT group. The effectiveness of formative assessment will not be significantly improved only by replacing the paper-and-pencil test with Web-based test. The strategies included in GAM-WATA are recommended to be taken into consideration when researchers design Web-based formative assessment systems in the future.     

Implementing computer-based assessment - A web-based mock examination changes attitudes

Interest in the educational use of information and communication technologies (ICT) in higher education is increasing. For successful implementation, it is important to know students’ attitudes and reservations and how they can be positively influenced. The objective of the present study was to examine possible attitudinal changes towards computer-based assessment (CBA) in students, after undergoing one such assessment. A web-based mock examination was provided to all fourth year medical students at Leipzig Medical School in 2008 and 2009. Before and after the web test, students were asked to document their agreement with statements concerning CBA. A large number of students made use of the offered web-based assessment. 383 participants could be analysed for the pre-post comparisons. The majority of the students rated their computer self-efficacy as high. In summary, students’ attitudes towards CBA in higher education tended to be positive. Gender differences seemed to be substantially influenced by differences in computer self-efficacy and were reduced considerably after only one practical experience. The actual experience had a positive influence on attitudes towards CBA. Nevertheless there were strong reservations about technical problems influencing the test performance when used for summative assessment. These concerns should not be ignored when trying to implement CBA. Optional formative CBA, perhaps early in higher education, seems to be a promising possibility of attracting students to computer- or web-based examination and learning methods, and may be a useful component of a successful implementation strategy.     

The Digital Learning Classroom: Improving English Language Learners’ academic success in mathematics and reading using interactive whiteboard technology

This study presents the findings from the first-year evaluation of the Round Rock Independent School District’s (ISD) Digital Learning Classroom project, an initiative focused on the improvement of English Language Learners’ (ELL) learning using interactive whiteboard (IWB) technology. An objective of the evaluation was to determine the extent IWB technology could foster performance parity in academic achievement between ELL and regular students, that is, reduce the student achievement gap between these two student groups in 3rd and 5th grade mathematics and reading. These grade levels and subjects were the primary focus of the project because students in grades 3 and 5 that do not pass the state’s standardized assessments in mathematics and reading cannot be promoted to the next grade level and therefore, these are “high stakes” tests for students. A second evaluation objective was to determine whether and the extent to which the Digital Learning Classroom could increase ELL students’ academic learning relative to that of ELL students in traditional classrooms (i.e., without IWBs). Using a quasi-experimental design, the results strongly indicate that IWBs can foster performance parity thereby closing the achievement gap between ELL and regular students while increasing ELL student achievement. Pedagogical implications for teachers of ELL students within the context of Digital Learning Classroom project implementation are presented, as well as recommendations for future study of the Digital Learning Classroom in ELL classroom settings.     

Can interactive visualization tools engage and support pre-university students in exploring non-trivial mathematical concepts?

Many students find it difficult to engage with mathematical concepts. As a relatively new class of learning tools, visualization tools may be able to promote higher levels of engagement with mathematical concepts. Often, development of new tools may outpace empirical evaluations of the effectiveness of these tools, especially in educational contexts. This seems to be the case with educational visualization tools. Much evidence about the effectiveness of these tools appears to be more suggestive than based on empirical evaluations. In this paper, we attempt to fill this gap and provide empirical evidence for the use of visualization tools in supporting exploratory and other learning-related activities. In particular, we aim to investigate whether visualization tools can be used to engage pre-university students in exploring non-trivial mathematical concepts. We focus particularly on this age group and content domain because of the difficulty these students may encounter when trying to investigate more challenging mathematical concepts. Also, it is during their formative years before university that students’ predisposition and likeness towards mathematical ideas are formed. We report in this paper a study assessing whether a visualization tool, whose design was informed explicitly by research from information visualization and human–computer interaction, could engage pre-university students in their exploration and learning of more advanced mathematical concepts. Students who participated in this study came from multiple grade levels and have diverse cognitive and language skills as well as preferences towards mathematics. The results of this study indicate that visualization tools can effectively engage these students and support their exploration of non-trivial mathematical concepts, only if the tool is designed such that it can cater the diverse needs of these students.     

Who assigns the most ICT activities? Examining the relationship between teacher and student usage

The expansion of information and communication technology (ICT) infrastructure in schools is expected to promote learning. To what extent teachers are utilizing the new ICT tools to engage students in learning activities remains a question. This study reports what kind of activities teachers are likely to assign students, and what type of teachers are more likely to assign such activities. Teacher ICT usage and student ICT assignments are examined using a sample of 3729 elementary and junior high school teachers in Taiwan. The results from correlation and regression analysis indicate that teachers who infrequently use basic ICT tools such as word processing rarely assign student ICT activities. At the other end of spectrum, teachers who create complicated multimedia materials are most likely to assign student multimedia activities. Regression results show that teachers’ frequency of building websites is the best predictor for assigning ICT-based sharing activities to students, though the likelihood is greater for junior high school than elementary school teachers. The results suggest that teachers assign students activities which require a variety of ICT tools, and teachers’ own ICT practices influence the type of ICT activities they assign to students.     

Laptopers in an educational practice: Promoting the personal learning situation

In this article an ethnographical study of laptopers’ activities during lectures in a university milieu is analyzed as different kinds of involvements. Through interviews and observations we have focused on how these involvements influence the laptopers’ alignment towards the educational practice. The analysis shows the importance of separating the educational practice and the personal learning situation. Studying students’ learning intentions, rather than certain laptop related activities, we get a deeper understanding of the role the laptop can take during learning activities. Five general characteristics of laptoping are found. Negotiating the laptops’ different roles in the educational practice opens up for an understanding of the students personal learning situation as being more than just listening to a lecturer. Additionally, competent integration of digital tools into the learning situation extends the dimensions of the lecture beyond the lecturing hours and personal note taking. Such knowledge is vital for creating foundations for digital competency in a digitized society.