A Web-based model for developing assessment literacy of secondary in-service teachers

This research investigates the effect of a web-based model, named ‘Practicing, Reflecting, and Revising with Web-based Assessment and Test Analysis system (P2R-WATA) Assessment Literacy Development Model,’ on enhancing assessment knowledge and perspectives of secondary in-service teachers, and adopts a single group experimental research design. The WATA system provides teachers with personalized learning resources and situated environment to practice assembling, administering tests on-line, and appraising test-related statistical information to reflect and revise test items. The sample consisted of forty-seven secondary in-service mathematics and science teachers in a summer program for 36 h within six weeks. This research collects and analyzes quantitative data, including the pre-test and post-test of teachers’ knowledge and perspectives on assessment. The major results generally confirm the effectiveness of P2R-WATA model. Firstly, the assessment knowledge of the participants has improved after training, especially for teachers with low-level prior knowledge. Secondly, the findings also reveal that there is a significant improvement on teachers’ assessment perspectives.     

Web-based dynamic assessment: Taking assessment as teaching and learning strategy for improving students’ e-Learning effectiveness

This research combines the idea of cake format dynamic assessment defined by Sternberg and Grigorenko (2001) and the ‘graduated prompt approach’ proposed by 9 and 10 to develop a multiple-choice Web-based dynamic assessment system. This research adopts a quasi-experimental design to investigate the effectiveness of this Web-based dynamic assessment system (GPAM-WATA) and normal Web-based test (N-WBT). One hundred and sixteen sixth grade elementary students from four classes participated in this research. These four classes were randomly divided into the GPAM-WATA group and N-WBT group. Before e-Learning instruction, all the students took the prior knowledge assessment and the pre-test of the summative assessment. After 2-week e-Learning instruction, all the students took the post-test of the summative assessment. The research findings show that students in the GPAM-WATA group experience better e-Learning effectiveness than those in the N-WBT group. GPAM-WATA is also found to be effective in improving the e-Learning effectiveness of students with low-level prior knowledge.     

Web-based quiz-game-like formative assessment: Development and evaluation

This research aims to develop a multiple-choice Web-based quiz-game-like formative assessment system, named GAM-WATA. The unique design of ‘Ask-Hint Strategy’ turns the Web-based formative assessment into an online quiz game. ‘Ask-Hint Strategy’ is composed of ‘Prune Strategy’ and ‘Call-in Strategy’. ‘Prune Strategy’ removes one incorrect option and turns the original 4-option item into a 3-option one. ‘Call-in Strategy’ provides the rate at which other test takers choose each option when answering a question. This research also compares the effectiveness of three different types of formative assessment in an e-Learning environment: paper-and-pencil test (PPT), normal Web-based test (N-WBT) and GAM-WATA. In total, 165 fifth grade elementary students (from six classes) in central Taiwan participated in this research. The six classes of students were then divided into three groups and each group was randomly assigned one type of formative assessment. Overall results indicate that different types of formative assessment have significant impacts on e-Learning effectiveness and that the e-Learning effectiveness of the students in the GAM-WATA group appears to be better. Students in the GAM-WATA group more actively participate in Web-based formative assessment to do self-assessment than students in the N-WBT group. The effectiveness of formative assessment will not be significantly improved only by replacing the paper-and-pencil test with Web-based test. The strategies included in GAM-WATA are recommended to be taken into consideration when researchers design Web-based formative assessment systems in the future.     

Developing Web-based assessment strategies for facilitating junior high school students to perform self-regulated learning in an e-Learning environment

This research refers to the self-regulated learning strategies proposed by Pintrich (1999) in developing a multiple-choice Web-based assessment system, the Peer-Driven Assessment Module of the Web-based Assessment and Test Analysis system (PDA-WATA). The major purpose of PDA-WATA is to facilitate learner use of self-regulatory learning behaviors to perform self-regulated learning and in turn improve e-Learning effectiveness. PDA-WATA includes five main strategies: ‘Adding Answer Notes,’ ‘Stating Confidence,’ ‘Reading Peer Answer Notes,’ ‘Recommending Peer Answer Notes’ and ‘Querying Peers’ Recommendation on Personal Answer Notes’. Using these strategies, examinees are allowed to add answer notes to explain why they chose a certain option as the correct answer and state their confidence in their own answer and answer notes, for peers’ reference. In addition to reading peer answer notes, examinees can also recommend peer answer notes as valuable references. The recommendation information can also be queried by all examinees. Quasi-experimental design was adopted to understand the effectiveness of PDA-WATA in facilitating learner use of self-regulatory learning behaviors to perform self-regulated learning and in improving learner e-Learning effectiveness. Participants were 123 seventh-grade junior high school students from four classes. These four classes were randomly divided into the PDA-WATA group (n = 63) and the N-WBT group (n = 60). Before e-Learning instruction, all students took the pre-test of the Learning Process Inventory (LPI), used to understand how often learners use self-regulatory learning behaviors in the learning process, and the pre-test of the summative assessment. After a two-week e-Learning instruction, the students all took the post-test of the LPI and the summative assessment. Results indicate that students in the PDA-WATA group appear to be more willing to take the Web-based formative assessment than students in the N-WBT group. In addition, PDA-WATA appears to be significantly more effective than N-WBT in facilitating learner use of self-regulatory learning behaviors to perform self-regulated learning and in improving their e-Learning effectiveness. Moreover, this research also finds that in the PDA-WATA group, there is no significant difference between the learning effectiveness of students with a low level of self-regulated learning and students with a high level of self-regulated learning, but similar result cannot be found in the N-WBT group.     

What strategies are effective for formative assessment in an e‐learning environment?

Abstract  The web-based formative assessment developed in this research is named Formative Assessment Module of the Web-based Assessment and Test Analysis System (FAM-WATA). FAM-WATA is a multiple-choice web-based formative assessment module containing six effective strategies: 'repeat the test', 'correct answers are not given', 'query scores', 'ask questions', 'monitor answering history', and 'all pass and then reward'. This research explored the effectiveness of FAM-WATA, cognitive styles and e-learning, and student attitudes towards the six strategies of FAM-WATA. A total of 503 seventh-grade students in central Taiwan were valid in this research. Overall results indicated that students displayed a positive attitude towards the six strategies of FAM-WATA. In addition, results also showed that students in an e-learning environment equipped with FAM-WATA achieved better learning effectiveness, and that field independent students appeared to make better use of FAM-WATA strategies than field dependent students. This research concluded that FAM-WATA benefited student learning in an e-learning environment.     

Integrating cognitive apprenticeship methods in a Web-based educational technology course for P-12 teacher education

The purpose of this study is to investigate the integration of a cognitive apprenticeship model in a Web-based course. The subject of this study is an educational technology course for pre-service P-12 teacher education students. Specifically, this study presents student reports of how cognitive apprenticeship methods impacted student learning processes of (a) technology skills and (b) technology integration methods for teaching. The methodological framework for this qualitative investigation is an interpretive case study. Student reflections and teacher observations revealed that students found modeling, coaching, scaffolding, and exploration key to fostering skill knowledge, and they found the use of cognitive apprenticeship methods fostered an understanding of integrating technology for teaching and learning.     

Learning styles and formative assessment strategy: enhancing student achievement in Web-based learning

Abstract The purpose of this research was to investigate the effects of formative assessment and learning style on student achievement in a Web-based learning environment. A quasi-experimental research design was used. Participants were 455 seventh grade students from 12 classes of six junior high schools. A Web-based course, named BioCAL, combining three different formative assessment strategies was developed. The formative assessment strategies included Formative Assessment Module of the Web-Based Assessment and Test Analysis system (FAM-WATA) (with six Web-based formative assessment strategies), Normal Module of Web-Based Assessment and Test Analysis system (N-WATA) (only with partial Web-based formative assessment strategy) and Paper and Pencil Test (PPT) (without Web-based formative assessment strategy). Subjects were tested using Kolb's Learning Style Inventory, and assigned randomly by class into three groups. Each group took Web-based courses using one of the formative assessment strategies. Pre- and post-achievement testing was carried out. A one-way ANCOVA analysis showed that both learning style and formative assessment strategy are significant factors affecting student achievement in a Web-based learning environment. However, there is no interaction between these two factors. A post hoc comparison showed that performances of the FAM-WATA group are higher than the N-WATA and PPT groups. Learners with a 'Diverger' learning style performed best followed by, 'Assimilator', 'Accommodator', and 'Converger', respectively. Finally, FAM-WATA group students are satisfied with six strategies of the FAM-WATA.     

Web-based Assessment and Test Analyses (WATA) system: development and evaluation

This study introduces the development of a Web‐based assessment system, the Web‐based Assessment and Test Analyses (WATA) system, and examines its impacts on teacher education. The WATA system is a follow‐on system, which applies the Triple‐A Model (assembling, administering, and appraising). Its functions include (1) an engine for teachers to administer and manage testing, (2) an engine for students to apply tests, and (3) an engine for generating test results and analyses for teachers. Two studies were undertaken to assess the usefulness and potential benefits of the WATA system for teacher education. In the first study, 47 in‐service teachers were asked to assess the functions of the WATA system. The results indicated that they were satisfied with the Triple‐A Model of the WATA system. In the second study, 30 pre‐service teachers were required to use the WATA system during the teacher‐training program. After 4 months of experience in using the WATA system, the pre‐service teachers' perspectives of assessment have been changed significantly. The findings of these two studies might provide some guidance to help those who are interested in the development of Web‐based assessment and intend to infuse information technology into teacher education.     

Attention paid to feedback provided by a computer-based assessment for learning on information literacy

Three studies are presented on attention paid to feedback provided by a computer-based assessment for learning on information literacy. Results show that the attention paid to feedback varies greatly. In general the attention focuses on feedback of incorrectly answered questions. In each study approximately fifty percent of the respondents paid attention to feedback of incorrect answers only. Approximately another twenty-five percent did not pay attention to feedback at all. Results suggest that differences in attention paid to feedback are influenced by task difficulty and test length. Supervision, however, does not seem to influence the average attention paid to feedback. On the other hand, results show that indirect and direct supervision lead to a greater impact of feedback provided by a computer-based assessment for learning as the number increases of students taking the test and, as a consequence, paying attention to feedback.     

A novel approach for enhancing student reading comprehension and assisting teacher assessment of literacy

For students of English as a Foreign Language (EFL), reading exercises are critical not only for developing strong reading comprehension, but also for developing listening, speaking, and writing skills. Prior research suggests that social, collaborative learning environments are best suited for improving language ability. However, opportunities for English learners to collaboratively practice reading comprehension are minimal, and due to resource constraints and a lack of accurate evaluation methods, English instructors rarely assess student literacy effectively. In response to these problems, we propose a Tag-based Collaborative reading learning System (TACO) that makes use of Web 2.0 Internet social tagging techniques to provide a collaborative environment for reading English. We test our system’s ability to both improve reading comprehension and aid teachers in accurately assessing literacy by conducting a three-month trial with 56 participating Taiwanese high school students from February to May 2009. During this period, post-testing results show a significant improvement in reading scores among participants in our tag-based system, and survey feedback from teachers suggests an improved capacity for literacy assessment.     

The development of a formative scenario-based computer assisted assessment tool in psychology for teachers: The PePCAA project

Formative computer assisted assessment has become increasingly attractive in Higher Education where providing useful feedback to large numbers of students can be difficult. However, the nature of such assessments has often been limited to objective questions such as multiple-choice. This paper reports on the development and initial trialling of a more innovative, formative use of computer assisted assessment in a Higher Education context. The European funded PePCAA (Pedagogical Psychology Computer Assisted Assessment) project developed a series of scenario-based computer-delivered formative assessments of pedagogical psychology for teachers and trainee teachers, using a range of software features, including the addition of confidence measurement. The project had a two-fold aim: to provide a tool to improve understanding of pedagogical psychology and to explore the potential of more innovative techniques of computer assisted assessment to motivate students and to assess deeper learning. The combination of computer-delivered formative assessment with innovative question styles and confidence ratings is believed to be unique for pedagogical psychology. Scenarios were based on realistic classroom situations and focused on problem solving or on utilising best practice. The PePCAA Learning Assessment Circle (PLAC) provided a framework for indexing the kinds of processes required of users. In the UK, small scale trialling involved a total of 23 teacher trainees such that each assessment was attempted by about seven participants. Participants completed evaluation questionnaires after each assessment. Responses from learners indicated that the UK scenarios were generally very well received and had at least partly achieved the aim of stimulating deeper learning. Transfer of assessments between countries proved more difficult than expected. The next stage of development should be to conduct a larger pilot, thus allowing full investigation of the reliability and validity of the assessments. There is also scope for further development of the PePCAA approach and for its application in other subjects.     

Designing of a type-2 fuzzy logic filter for improving edge-preserving restoration of interlaced-to-progressive conversion

This paper focuses on recently advanced fuzzy models and the application of type-2 fuzzy sets in video deinterlacing. The final goal of the proposed deinterlacing algorithm is to exactly determine an unknown pixel value while preserving the edges and details of the image. To begin, we will discuss some artefacts of spatial, temporal, and spatio-temporal domain deinterlacing methods. In order to address the aforementioned issues, we adopted type-2 fuzzy sets concepts to design a weight evaluating approach. In the proposed method, the upper and lower fuzzy membership functions of the type-2 fuzzy logic filters are derived from the type-1 (or primary) fuzzy membership function. The weights from upper and lower membership functions are considered to be multiplied with the candidate deinterlaced pixels. Experimental results proved that the performance of the proposed method was superior, both objectively and subjectively to other different conventional deinterlacing methods. Moreover, the proposed method preserved the smoothness of the original image edges and produced a high-quality progressive image.     

A comparative analysis of the consistency and difference among teacher-assessment, student self-assessment and peer-assessment in a Web-based portfolio assessment environment for high school students

This study explored the consistency and difference of teacher-, student self- and peer-assessment in the context of Web-based portfolio assessment. Participants were 72 senior high school students enrolled in a computer application course. Through the assessment system, the students performed portfolio creation, inspection, self- and peer-assessment; three teachers reviewed portfolios and evaluated learning performances. There were significant differences in the results of the three assessment methods, among which teacher-raters adopted the most rigorous scoring standards, while peer-raters tended to use the most lax standards. The results of self- and teacher-assessment were discovered to be consistent; however, consistency was not discovered between self- and peer-assessment as well as peer- and teacher-assessment. In analyzing their consistency with the end-of-course examination, teacher- and self-assessment demonstrated high consistency, whereas peer-assessment showed a low level of consistency.     

Measuring Information and Communication Technology Literacy using a performance assessment: Validation of the Student Tool for Technology Literacy (ST2L)

This paper reports the validation scores of the Student Tool for Technology Literacy (ST²L), a performance-based assessment based on the National Educational Technology Standards for Students (NETS*S) used to measure middle grade students Information and Communication Technology (ICT) Literacy. Middle grade students (N = 5884) from school districts across the state of Florida were recruited for this study. This paper first provides an overview of various methods to measure ICT literacy and related constructs, and provides documented evidence of score reliability and validity. Following sound procedures based on prior research, this paper provides validity and reliability evidence for the ST²L scores using both item response theory and testlet response theory. This paper examines both the internal and external validity of the instrument. The ST²L, with minimal revision, was found to be a sound measure of ICT literacy for low-stakes assessment purposes. A discussion of the results is provided with emphasis on the psychometric properties of the tool and some practical insights on with whom the tool should be used in future research and practice.     

Tools for the job: a report of two surveys of information and communications technology training and use for literacy in primary schools in the West of England

Substantial investment has been made in improving computer hardware and raising the profile of information and communications technology (ICT) in English primary schools in recent years, in line with the belief that skills in ICT are essential for our future workforce ( DfEE/QCA 2000 ). This study compares practising teachers' responses in 1998 and 2001 to a questionnaire about the aims and uses of ICT in primary schools for literacy activities. It discusses the changes in what teachers want and their understanding of the task the government has given them to do. A commonly cited problem is finding time to absorb the new technology to maximise its pedagogic potential.     

Reliability and validity of Web-based portfolio peer assessment: A case study for a senior high school’s students taking computer course

This study examined the reliability and validity of Web-based portfolio peer assessment. Participants were 72 second-grade students from a senior high school taking a computer course. The results indicated that: 1) there was a lack of consistency across various student raters on a portfolio, or inter-rater reliability; 2) two-thirds of the raters demonstrated inconsistency assessing different portfolios, i.e. inner-rater reliability; 3) peer-assessment scores were not consistent with teacher-assessment scores (criterion-related validity); 4) significant differences were found between peer-assessment scores and end-of-course examination scores, implying that Web-based portfolio peer assessment failed to reflect learning achievements (criterion-related validity). In short, Web-based portfolio peer assessment was not a reliable and valid method.     

PAIDD: a hybrid P2P-based architecture for improving data distribution in social networks

Rapid growth in social networks(SNs)presents a unique scalability challenge for SN operators because of the massive amounts of data distribution among large number of concurrent online users.A request from any user may trigger hundreds of server activities to generate a customized page and which has already become a huge burden.Based on the theoretical model and analytical study considering realistic network scenarios,this article proposes a hybrid P2P-based architecture called PAIDD.PAIDD fulfills effective data distribution primarily through P2P connectivity and social graph among users but with the help of central servers.To increase system efficiency,PAIDD performs optimized content prefetching based on social interactions among users.PAIDD chooses interaction as the criteria because user’s interaction graph is measured to be much smaller than the social graph.Our experiments confirm that PAIDD ensures satisfactory user experience without incurring extensive overhead on clients’network.More importantly,PAIDD can effectively achieve one order of magnitude of load reduction at central servers.     

Structuring the peer assessment process: a multilevel approach for the impact on product improvement and peer feedback quality

In order to optimize students' peer feedback processes, this study investigates how an instructional intervention in the peer assessment process can have a beneficial effect on students' performance in a wiki environment in first‐year higher education. The main aim was to study the effect of integrating a peer feedback template with a varying structuring degree. The present study involved three conditions: a no structure, a basic structure and an elaborate structure condition. Due to a clear hierarchical structure, in which over time (level 1), 168 students (level 2) are nested within 37 groups (level 3), multilevel analysis was performed to examine the effect of time, student and group level influences on students' peer feedback quality and product scores. The results revealed that both peer feedback quality and product scores increase significantly for all conditions over time, after multiple practice occasions. In addition, after several practice occasions, significant differences were found between the conditions in both peer feedback (elaborate higher than no structure) and product scores (elaborate and basic higher than no structure). Building on this, limitations, directions for future research and practical implications are presented.     

Evaluating the characteristics of a non-standardised Model Requirements Analysis (MRA) for the development of policy impact assessment tools

The aim of this paper is to provide a critical analysis of the strengths and weaknesses of a non-standardised Model Requirements Analysis (MRA) used for the purpose of developing the Sustainability Impact Assessment Tool (SIAT). By ‘non-standardised’ we mean not strictly following a published MRA method. The underlying question we are interested in addressing is how non-standardised methods, often employed in research driven projects, compare to defined methods with more standardised structure, with regards their ability to capture model requirements effectively, and with regards their overall usability. Through describing and critically assessing the specific features of the non-standardised MRA employed, the ambition of this paper is to provide insights useful for impact assessment tool (IAT) development. Specifically, the paper will (i) characterise kinds of user requirements relevant to the functionality and design of IATs; (ii) highlight the strengths and weaknesses of non-standardised MRA for user requirements capture, analysis and reflection in the context of IAT; (iii) critically reflect on the process and outcomes of having used a non-standardised MRA in comparison with other more standardised approaches. To accomplish these aims, we first review methods available for IAT development before describing the SIAT development process, including the MRA employed. Major strengths and weaknesses of the MRA method are then discussed in terms of user identification and characterisation, organisational characterisation and embedding, and ability to capture design options for ensuring usability and usefulness. A detailed assessment on the structural differences of MRA with two advanced approaches (Integrated DSS design and goal directed design) and their role in performance of the MRA tool is used to critique the approach employed. The results show that MRA is able to bring thematic integration, establish system performance and technical thresholds as well as detailing quality and transparency guidelines. Nevertheless the discussion points out to a number of deficiencies in application - (i) a need to more effectively characterise potential users, and; (ii) a need to better foster communication among the distinguished roles in the development process. If addressed these deficiencies, SIAT non-standardised MRA could have brought out better outcomes in terms of tool usability and usefulness, and improved embedding of the tool into conditions of targeted end-users.