Measurement and assessment in computer-supported collaborative learning

The overall goal of CSCL research is to design software tools and collaborative environments that facilitate social knowledge construction via a valuable assortment of methodologies, theoretical and operational definitions, and multiple structures [Hadwin, A. F., Gress, C. L. Z., & Page, J. (2006). Toward standards for reporting research: a review of the literature on computer-supported collaborative learning. In Paper presented at the 6th IEEE International Conference on Advanced Learning Technologies, Kerkrade, Netherlands; Lehtinen, E. (2003). Computer-supported collaborative learning: an approach to powerful learning environments. In E. De Corte, L. Verschaffel, N. Entwistle & J. Van Merriëboer (Eds.), Unravelling basic components and dimensions of powerful learning environments (pp. 35–53). Amsterdam, Netherlands: Elsevier]. Various CSCL tools attempt to support constructs associated with effective collaboration, such as awareness tools to support positive social interaction [Carroll, J. M., Neale, D. C., Isenhour, P. L., Rosson, M. B., & McCrickard, D. S. (2003). Notification and awareness: Synchronizing task-oriented collaborative activity. International Journal of Human–Computer Studies 58, 605] and negotiation tools to support group social skills and discussions [Beers, P. J., Boshuizen, H. P. A. E., Kirschner, P. A., & Gijselaers, W. H. (2005). Computer support for knowledge construction in collaborative learning environments. Computers in Human Behavior 21, 623–643], yet few studies developed or used pre-existing measures to evaluate these tools in relation to the above constructs. This paper describes a review of the measures used in CSCL to answer three fundamental questions: (a) What measures are utilized in CSCL research? (b) Do measures examine the effectiveness of attempts to facilitate, support, and sustain CSCL? And (c) When are the measures administered? Our review has six key findings: there is a plethora of self-report yet a paucity of baseline information above collaboration and collaborative activities, findings in the field are dominated by ‘after collaboration’ measurement, there is little replication and an over reliance on text-based measures, and an insufficient collection of tools and measures for examining processes involved in CSCL.     

Visualization of agreement and discussion processes during computer-supported collaborative learning

This study examined the effects of the shared space (SS) on students’ behaviors in a computer-supported collaborative learning (CSCL) environment. The SS visualizes discussion and agreement during online discussions. It was hypothesized the SS would increase the media richness of the CSCL-environment, would stimulate critical and exploratory group-norms, would lead to more positive perceptions of online collaboration, and would have an impact on students’ collaborative activities. In total, 59 students working in 20 groups had access to the SS visualization, while 58 students working in 20 groups did not. The results show that students with access to the SS visualization: (a) perceived higher media richness; (b) had a more exploratory group-norm perception; (b) perceived more positive group behavior; (c) perceived their group’s task strategies to be more effective; (d) engaged in different collaborative activities and (e) performed better on one part of the group task. These results demonstrate the potential benefits of visualizing agreement and discussion during CSCL.     

Scalability of assessments of wiki-based learning experiences in higher education

In recent years, the focus on higher education learning has shifted from knowledge to skills, with interpersonal skills likely being the most difficult to assess and work with. Wikis ease open collaboration among peers. A number of these skills can be objectively assessed by using wikis in an educational environment: collaborative writing, conflict resolution, group management, leadership, etc. However, when the number of students increases, their interactions usually increase at a higher rate. Under these circumstances, traditional assessment procedures suffer from scalability problems: manually evaluating in detail the information stored in a wiki to retrieve objective metrics becomes a complex and time-consuming task. Thus, automated tools are required to support the assessment of such processes. In this paper we compare seven case studies conducted in Computer Science courses of two Spanish universities: Cádiz and Seville. We comment on their different settings: durations, milestones, contribution sizes, weights in the final grade and, most importantly, their assessment methods. We discuss and compare the different methodologies and tools used to assess the desired skills in the context of each case study.     

Exploring regulatory processes during a computer-supported collaborative learning task using process mining

The purpose of this study was to explore sequences of social regulatory processes during a computer-supported collaborative learning task and their relationship to group performance. Analogous to self-regulation during individual learning, we conceptualized social regulation both as individual and as collaborative activities of analyzing, planning, monitoring and evaluating cognitive and motivational aspects during collaborative learning. We analyzed the data of 42 participants working together in dyads. They had 90 min to develop a common handout on a statistical topic while communicating only via chat and common editor. The log files of chat and editor were coded regarding activities of social regulation. Results show that participants in dyads with higher group performance (N = 20) did not differ from participants with lower group performance (N = 22) in the frequencies of regulatory activities. In an exploratory way, we used process mining to identify process patterns for high versus low group performance dyads. The resulting models show clear parallels between high and low achieving dyads in a double loop of working on the task, monitoring, and coordinating. Moreover, there are no major differences in the process of high versus low achieving dyads. Both results are discussed with regard to theoretical and empirical issues. Furthermore, the method of process mining is discussed.     

The effect of functional roles on perceived group efficiency during computer-supported collaborative learning: a matter of triangulation

In this article, the effect of functional roles on group performance and collaboration during computer-supported collaborative learning (CSCL) is investigated. Especially the need for triangulating multiple methods is emphasised: Likert-scale evaluation questions, quantitative content analysis of e-mail communication and qualitative analysis of open-ended questions were used. A comparison of 41 questionnaire observations, distributed over 13 groups in two research conditions – groups with prescribed functional roles (n = 7, N = 18) and nonrole groups (n = 6, N = 23) – revealed no main effect for performance (grade). Principal axis factoring of the Likert-scales revealed a latent variable that was interpreted as perceived group efficiency (PGE). Multilevel modelling (MLM) yielded a positive marginal effect of PGE. Most groups in the role condition report a higher degree of PGE than nonrole groups. Content analysis of e-mail communication of all groups in both conditions (role n = 7, N = 25; nonrole n = 6, N = 26) revealed that students in role groups contribute more ‘coordination’ focussed statements. Finally, results from cross case matrices of student responses to open-ended questions support the observed marginal effect that most role groups report a higher degree of PGE than nonrole groups.     

Contextual perspective in analysing collaborative knowledge construction of two small groups in web-based discussion

This paper presents a methodology designed to explore the role of context in collaborative knowledge construction activity in asynchronous web-based discussion. The discussions of two student groups participating in a web-based teacher education course were compared. The comparison aimed to highlight the differences and similarities between the groups’ knowledge construction activity through studying the thematic structure, communicative functions and contextual resources used in their discussions. The results indicated that the different backgrounds of the two student groups influenced the way context was created and interpreted, and how meanings were negotiated. The differences and similarities between the groups’ activity illuminated the situated and mediated nature of learning. The possibilities of the methodology used in this study for evaluating collaborative knowledge construction in context are also discussed.     

Differences in learning processes between successful and less successful students in computer-supported collaborative learning in the field of human nutrition and health

This study aimed to investigate the differences in learning processes between successful and less successful pairs of students in computer-supported collaborative learning (CSCL) in the field of human nutrition and health. As part of their regular MSc (and optional BSc) course “Exposure assessment in nutrition and health research” at Wageningen University, 44 students were asked (as an individual pretest) to design and analyze a study which evaluates a certain dietary assessment method. Subsequently, they were asked to discuss their evaluation studies in randomized pairs using a CSCL platform. As an individual posttest, students had to re-design and re-analyze the same evaluation study. The quality of students’ knowledge construction in both tests and characteristics of their learning processes in the CSCL environment were assessed through two coding schemes. Based on their learning outcomes (quality of knowledge construction), pairs of students were divided into two subgroups: successful and less successful students. Next, the learning processes of these subgroups were compared. This study revealed that the learning processes of successful and less successful students in the CSCL environment differed in terms of relevance, width and depth of discussion and justification and reasoning. Based on these findings, recommendations for further research and educational practice are formulated.     

Images in computer-supported learning: Increasing their benefits for metacomprehension through judgments of learning

Images are widely used in computer-based learning although they might bias learners’ judgments on how well they have mastered the material, which might reduce the effectiveness of metacognitive learning control. As that bias seems to result from primarily theory-based processing, we used two-step judgments of learning to induce more experience-based processes that we hypothesized to benefit metacomprehension accuracy. In Experiment 1, 381 participants studied a ten-section text on team building that was either accompanied by conceptual images, decorative images, or no images. Half of the participants made simple judgments by rating after each section the likelihood of correctly answering a knowledge question on that section (judgment of learning; JOL) The other half made combined judgments by rating text difficulty before making a JOL. As postulated, combined JOL benefitted accuracy and knowledge test scores; both were highest in the conceptual images group. In Experiment 2, to further increase accuracy, we manipulated judgment scope. Rather than predicting answers correct for an entire chapter, another 310 participants had to predict answers correct for a specific concept from a chapter. Again, accuracy and test scores were highest in the conceptual images group. Contrary to expectations, however, JOL accuracy did not benefit from term-specific judgments. We discuss implications for research into metacomprehension processes in computer-supported learning and for adaptive learner support based on judgment prompts.     

Gender-related differences in computer-mediated communication and computer-supported collaborative learning

Abstract   A question associated with the introduction of computer-supported collaborative learning (CSCL) is whether all participants profit equally from working in CSCL environments. This article reports on a review study into gender-related differences in participation in CSCL. As many of the processes in CSCL are similar to those in computer-mediated communication (CMC), studies into CMC are also included in the review. Male dominance is found to play a role in many CMC settings. A learning culture with an explicit focus on participation by all students seems to be related to a more gender-balanced participation in CMC, however. A tendency for boys to be more active participants than girls is also present in CSCL environments, but it is less pronounced than in CMC. This may be explained by the fact that participation is explicitly promoted in most CSCL environments. Gender differences in the character of students' contributions are found in both CMC and CSCL. It is concluded that in order to avoid gender-stereotyped participation and communication patterns, it is necessary to explicitly address inclusiveness as an aspect of a collaborative classroom culture. A plea is made for further research into differential participation by students in CSCL, and the effects thereof on cognitive and affective learning outcomes. Research should also focus on the question how classroom cultures can be promoted that support active participation of all students aimed at collaborative knowledge construction.     

Advancing educational research on collaboration through the use of gStudy computer-supported collaborative learning (CSCL) tools: Introduction to special issue

This is an introduction to a special issue on computer-supported collaborative learning.     

Visualization of participation: Does it contribute to successful computer-supported collaborative learning?

This study investigated the effects of visualization of participation during computer-supported collaborative learning (CSCL). It is hypothesized that visualization of participation could contribute to successful CSCL. A CSCL-environment was augmented with the Participation Tool (PT). The PT visualizes how much each group member contributes to his or her group’s online communication. Using a posttest-only design with a treatment (N = 52) and a control group (N = 17), it was examined whether students with access to the PT participated more and more equally during collaboration, reported higher awareness of group processes and activities, collaborated differently, and performed better than students without access to the PT. The results show that students used the PT quite intensively. Furthermore, compared to control group students, treatment group students participated more and engaged more in coordination and regulation of social activities during collaboration by sending more statements that addressed the planning of social activities. However, equality of participation, awareness of group processes and quality of the group products was not higher in the treatment condition. Still, the results of this study demonstrate that visualization of participation can contribute to successful CSCL.     

Design and evaluation of a collaborative learning environment

Collaboration becomes an essential competency in the current knowledge society. In this study, a collaborative learning environment was designed to facilitate students in group collaboration. Instructional support strategies of friendship and meaningful learning tasks were applied to promote collaboration. Scaffolding strategies such as writing progress reports and developing product versions were used to coordinate and monitor the collaborative learning environment. In addition, the online file sharing tool DriveHQ was used to facilitate collaboration. Four classes of students used the learning environment to complete their final project in pairs. Results showed friendship and meaningful learning tasks helped to promote individual accountability and positive interdependence, and the use of progress reports and product versions was useful for coordinating and monitoring the learning process. However, additional strategies might be needed for more effective collaboration to take place. Issues involved in this study are discussed.     

Learning technologies: Affective and social issues in computer-supported collaborative learning

This paper is concerned with affective issues in learning technologies in a collaborative context. Traditionally in learning there has been a division between cognition and affect: where cognition is concerned with skills and processes such as thinking and problem-solving and affect with emotional areas such as motivation, attitudes, feelings. Affective issues have been viewed as somewhat problematic in studying learning, so although it is well known that learner attitude, motivation, and emotional state are very important, they have often been excluded from the frame of research, or studied separately from cognitive learning. This position is gradually changing and this paper considers what previous research has been conducted in these areas. It discusses the role of affective factors in three main areas of collaboration: in settings where learners are co-located, in on-line communities and to support and develop socio-emotional skills. It considers relevant developments in these areas, what the outcomes have been and suggests important directions for future research.     

Modeling individual and collaborative problem-solving in medical problem-based learning

Today a great many medical schools have turned to a problem-based learning (PBL) approach to teaching as an alternative to traditional didactic medical education to teach clinical-reasoning skills at the early stages of medical education. While PBL has many strengths, effective PBL tutoring is time-intensive and requires the tutor to provide a high degree of personal attention to the students, which is difficult in the current academic environment of increasing demands on faculty time. This paper describes the student modeling approach used in the COMET intelligent tutoring system for collaborative medical PBL. To generate appropriate tutorial actions, COMET uses a model of each student’s clinical reasoning for the problem domain. In addition, since problem solving in group PBL is a collaborative process, COMET uses a group model that enables it to do things like focus the group discussion, promote collaboration, and suggest peer helpers. Bayesian networks are used to model individual student knowledge and activity, as well as that of the group. The validity of the modeling approach has been tested with student models in the areas of head injury, stroke, and heart attack. Receiver operating characteristic (ROC) curve analysis shows that the models are highly accurate in predicting individual student actions. Comparison with human tutors shows that the focus of group activity determined by the model agrees with that suggested by the majority of the human tutors with a high degree of statistical agreement (McNemar test, p = 0.774, Kappa = 0.823).