Student success prediction in MOOCs

Predictive models of student success in Massive Open Online Courses (MOOCs) are a critical component of effective content personalization and adaptive interventions. In this article we review the state of the art in predictive models of student success in MOOCs and present a categorization of MOOC research according to the predictors (features), prediction (outcomes), and underlying theoretical model. We critically survey work across each category, providing data on the raw data source, feature engineering, statistical model, evaluation method, prediction architecture, and other aspects of these experiments. Such a review is particularly useful given the rapid expansion of predictive modeling research in MOOCs since the emergence of major MOOC platforms in 2012. This survey reveals several key methodological gaps, which include extensive filtering of experimental subpopulations, ineffective student model evaluation, and the use of experimental data which would be unavailable for real-world student success prediction and intervention, which is the ultimate goal of such models. Finally, we highlight opportunities for future research, which include temporal modeling, research bridging predictive and explanatory student models, work which contributes to learning theory, and evaluating long-term learner success in MOOCs.     

A critique to Akdemir and Oguz (2008): Methodological and statistical issues to consider when conducting educational experiments

In the paper “Computer-based testing: An alternative for the assessment of Turkish undergraduate students”, Akdemir and Oguz (2008) discuss an experiment to compare student performance in paper-and-pencil tests with computer-based tests, and conclude that students taking computer-based tests do not underperform compared to students taking pen-and-pencil tests. In this letter, we indicate two severe methodological and statistical flaws in this paper. We show how, in general, such flaws can affect experimental research. Due to these flaws, the conclusions by Akdemir and Oguz are unfounded: one cannot reach these conclusions on basis of this design and analysis. We provide a set of guidelines and advices to avoid methodological problems when setting up an educational experiment.     

Student modeling approaches: A literature review for the last decade

This paper constitutes a literature review on student modeling for the last decade. The review aims at answering three basic questions on student modeling: what to model, how and why. The prevailing student modeling approaches that have been used in the past 10 years are described, the aspects of students’ characteristics that were taken into consideration are presented and how a student model can be used in order to provide adaptivity and personalisation in computer-based educational software is highlighted. This paper aims to provide important information to researchers, educators and software developers of computer-based educational software ranging from e-learning and mobile learning systems to educational games including stand alone educational applications and intelligent tutoring systems. In addition, this paper can be used as a guide for making decisions about the techniques that should be adopted when designing a student model for an adaptive tutoring system. One significant conclusion is that the most preferred technique for representing the student’s mastery of knowledge is the overlay approach. Also, stereotyping seems to be ideal for modeling students’ learning styles and preferences. Furthermore, affective student modeling has had a rapid growth over the past years, while it has been noticed an increase in the adoption of fuzzy techniques and Bayesian networks in order to deal the uncertainty of student modeling.     

Latent Semantic Analysis: five methodological recommendations

The recent influx in generation, storage, and availability of textual data presents researchers with the challenge of developing suitable methods for their analysis. Latent Semantic Analysis (LSA), a member of a family of methodological approaches that offers an opportunity to address this gap by describing the semantic content in textual data as a set of vectors, was pioneered by researchers in psychology, information retrieval, and bibliometrics. LSA involves a matrix operation called singular value decomposition, an extension of principal component analysis. LSA generates latent semantic dimensions that are either interpreted, if the researcher's primary interest lies with the understanding of the thematic structure in the textual data, or used for purposes of clustering, categorization, and predictive modeling, if the interest lies with the conversion of raw text into numerical data, as a precursor to subsequent analysis. This paper reviews five methodological issues that need to be addressed by the researcher who will embark on LSA. We examine the dilemmas, present the choices, and discuss the considerations under which good methodological decisions are made. We illustrate these issues with the help of four small studies, involving the analysis of abstracts for papers published in the European Journal of Information Systems.     

An adaptive Student Centered Curriculum for an intelligent training system

An intelligent tutoring system customizes its presentation of knowledge to the individual needs of each student based on a model of the student. Student models are more complex than other user models because the student is likely to have misconceptions. We have addressed several difficult issues in reasoning about a student's knowledge and skills within a real-time simulation-based training system. Our conceptual framework enables important aspects of the tutor's reasoning to be based upon simple, comprehensible representations that are the basis for a Student Centered Curriculum. We have built a system for teaching cardiac resuscitation techniques in which the decisions abouthow to teach are separated from the decisions aboutwhat to teach. The training context (i.e., choice of topics) is changed based on a tight interaction between student modeling techniques and simulation management. Although complex student models are still required to support detailed reasoning about how to teach, we argue that the decision about what to teach can be adequately supported by qualitatively simpler techniques, such as overlay models. This system was evaluated in formative studies involving medical school faculty and students. Construction of the student model involves monitoring student actions during a simulation and evaluating these actions in comparison with an expert model encoded as a multi-agent plan. The plan recognition techniques used in this system are novel and allow the expert knowledge to be expressed in a form that is natural for domain experts.     

Content-free collaborative learning modeling using data mining

Modeling user behavior (user modeling) via data mining faces a critical unresolved issue: how to build a collaboration model based on frequent analysis of students in order to ascertain whether collaboration has taken place. Numerous human-based and knowledge-based solutions to this problem have been proposed, but they are time-consuming or domain-dependent. The diversity of these solutions and their lack of common characteristics are an indication of how unresolved this issue remains. Bearing this in mind, our research has made progress on several fronts. First, we have found supportive evidence, based on a collaborative learning experience with hundreds of students over three consecutive years, that an approach using domain independent learning that is transferable to current e-learning platforms helps both students and teachers to manage student collaboration better. Second, the approach draws on a domain-independent modeling method of collaborative learning based on data mining that helps clarify which user-modeling issues are to be considered. We propose two data mining methods that were found to be useful for evaluating student collaboration, and discuss their respective advantages and disadvantages. Three data sources to generate and evaluate the collaboration model were identified. Third, the features being modeled were made accessible to students in several meta-cognitive tools. Their usage of these tools showed that the best approach to encourage student collaboration is to show only the most relevant inferred information, simply displayed. Moreover, these tools also provide teachers with valuable modeling information to improve their management of the collaboration. Fourth, an ontology, domain independent features and a process that can be applied to current e-learning platforms make the approach transferable and reusable. Fifth, several open research issues of particular interest were identified. We intend to address these open issues through research in the near future.     

The historical and situated nature design experiments – Implications for data analysis

This article is a methodological contribution to the use of design experiments in educational research. We will discuss the implications of a historical and situated interpretation to design experiments, the consequences this has for the analysis of the collected data and empirically based suggestions to improve the designs of the computer-based learning resources. This interpretation differs from that of the majority of other researchers who consider design experiments as fixed interventions. Our interpretation allows for an understanding of students' learning trajectories as part of the school's overall activities, which in turn has implications for suggestions regarding the improvement of computer-based learning resources. We develop argument in three different ways. First, we discuss our interpretation of design experiments and compare it with the dominant debate about design experiments as a methodological approach. Second, we discuss the implications this position has for interaction analysis methods. Finally, we empirically demonstrate this methodological implication by illustrating how vital it might be to consider data collected as part of design experiments as an element of a larger, established institutional setting characterized by inherent socio-cultural features. A conclusion that may be drawn from our situated interpretation of this design experiment is that, to improve students' knowledge constructions, it is not enough – nor is it in principle possible – to perfect the design of the technology. We also need to improve institutional aspects on how schools support students' knowledge constructions when using these technological tools, where the teacher's role in this work is invaluable. A historical and situated interpretation of design experiments provides insight into how these improvements may be made.     

Performance modeling of distributed and replicated databases

The paper surveys performance models for distributed and replicated database systems. Over the last 20 years (1980-2000), a variety of such performance models have been developed and they differ in: (1) which aspects of a real system are or are not captured in the model (e.g., replication, communication, nonuniform data access, etc.); and (2) how these aspects are modeled. We classify the different alternatives and modeling assumptions and discuss their interdependencies and expressiveness for the representation of distributed databases. This leads to a set of building blocks for analytical performance models. To illustrate the work that is surveyed, we select a combination of these proven modeling concepts and give an example of how to compose a balanced analytical model of a replicated database. We use this example to show how to derive meaningful performance values and to discuss the applicability and expressiveness of performance models for distributed and replicated databases. Finally, we compare the analytical results to measurements in a distributed database system     

Diagnostic,predictive and compositional modeling with data mining in integrated learning environments

Models represent a set of generic patterns to test hypotheses. This paper presents the CogMoLab student model in the context of an integrated learning environment. Three aspects are discussed: diagnostic and predictive modeling with respect to the issues of credit assignment and scalability and compositional modeling of the student profile in the context of an intelligent tutoring system/adaptive hypermedia learning system architectural pattern. The SOM–PCA, a collaborative-based data mining approach, is shown to be reusable for all three purposes above, enabling fast, objective implementations without requiring much intensive data collection.     

Evaluating the sustainability of electronic media: Strategies for life cycle inventory data collection and their implications for LCA results

This paper compares two Life Cycle Assessment (LCA) studies independently carried out to assess the environmental impacts of electronic versus print media. Although the two studies lead to the same overall conclusion for the case of a news magazine – namely that the tablet version of the magazine has environmental advantages over the print version – there are significant differences in the details of the LCA results. We show how these differences can be explained by differences in the methodological approaches used for life cycle inventory (LCI) modelling, in particular the use of rough average data versus the attempt to use the most specific and detailed data as possible. We conclude that there are several issues in LCA practice (at least when applied in the domain of media) that can significantly influence the results already at the LCI level: The data collection strategy used (e.g. relying on desk-based research or dismantling a given device) and the decisions made at inventory level with regard to parameters with significant geographic variability, such as the electricity mix or recycling quotas.     

A framework for understanding learning from management simulations

Article proposes a framework for understanding certain aspects of learning from computer‐based management simulations. The framework is developed as a result of small‐scale, exploratory research amongst postgraduate students at three European Business Schools. An overview from the general literature on management learning is provided and claims made in the specific literature regarding student learning from management simulations are considered. There is reference to the methodological problems associated with establishing learning outcomes. The need to conduct research into the learning experience from the student perspective is argued. A framework for mapping student perceptions of management simulation learning is developed using the established notions of learning style and depth of learning. Data from the exploratory research are used as illustration for the framework.     

Model-driven development of interactive groupware systems: Integration into the software development process

Modeling interactive groupware systems is a complex and multi-disciplinary task. It is necessary to provide designers and engineers with a set of methods, notations and tools to specify the different aspects to consider when designing this type of systems. In this work we present a methodological framework based on the integration of several notations and processes for modeling some of these aspects, in particular: interaction, collaboration and functionality. The objective of this work is to provide a more complete support to the design of groupware systems, considering different viewpoints and modeling perspectives of the several stakeholders involved in the development of such applications.     

Toward a dialectic relation between the results in CSCL: Three critical methodological aspects of content analysis schemes

The research field of Computer-Supported Collaborative Learning (CSCL) includes a large variety of approaches which present significant theoretical and methodological differences. This diversity complicates the articulation of the knowledge that is produced within this investigative framework. The paper addresses this problem from a dialectic view. We propose that the main reason for this problem is not the theoretical and methodological diversity itself, but rather the difficulty of situating one specific result within this diversity in a way that makes dialectic relations between results visible and mutual transformation of the approaches possible. In the present paper, we propose a set of indicators, applicable to content analysis approaches, aimed to facilitate this reciprocal positioning of the results in the field. These indicators come from what we term “critical methodological aspects”: those aspects of the methodological infrastructure that are directly related to theoretical positions. We consider three critical methodological aspects in content analysis schemes: the units of analysis, the relations to be established, and the dimensions of analysis. Indicators regarding these aspects are proposed and defined, and their use for facilitating dialectical relations between results is exemplified by means of the examination of five specific approaches.     

A paradigm for active decision support

Active decision support is concerned with developing advanced forms of decision support where the support tools are capable of actively participating in the decision making process, and decisions are made by fruitful callaboration between the human and the machine. It is currently an active and leading area of research within the field of decision support systems. The objective of this paper is to share the details of our research in this area. We present our overall research strategy for exploring advanced forms of decision support and discuss in detail our research prototype called JANUS that implements our ideas. We establish the contributions of our work and discuss our experiences and plans for future.     

Predictive Statistical Models for User Modeling

The limitations of traditional knowledge representation methods for modeling complex human behaviour led to the investigation of statistical models. Predictive statistical models enable the anticipation of certain aspects of human behaviour, such as goals, actions and preferences. In this paper, we motivate the development of these models in the context of the user modeling enterprise. We then review the two main approaches to predictive statistical modeling, content-based and collaborative, and discuss the main techniques used to develop predictive statistical models. We also consider the evaluation requirements of these models in the user modeling context, and propose topics for future research.     

Assessing a knowledge-based approach to commercial loan underwriting

We discuss the challenges in developing decision support tools for commercial underwriting and discuss how several different approaches to the underwriting problem have been addressed. We then describe an expert system-based approach to credit underwriting that has been in commercial usage for over ten years in a variety of financial institutions. The expert system approach addresses many features of the underwriting process that alternative approaches do not. The system is characterized by a functional representation of knowledge and a graph-based inference mechanism. The inference mechanism is unique in its pragmatic approach to the implementation of probability theory. This approach offers flexibility for modeling various aspects of real world credit decisions not always treated by traditional approaches. We give examples of how this approach can be and is currently being applied to facilitating underwriting decisions in commercial lending contexts.     

Corpus-based vocabulary lists for language learners for nine languages

We present the KELLY project and its work on developing monolingual and bilingual word lists for language learning, using corpus methods, for nine languages and thirty-six language pairs. We describe the method and discuss the many challenges encountered. We have loaded the data into an online database to make it accessible for anyone to explore and we present our own first explorations of it. The focus of the paper is thus twofold, covering pedagogical and methodological aspects of the lists’ construction, and linguistic aspects of the by-product of the project, the KELLY database.     

What middleware for network centric operations?

The main intent of this paper is to address the issue of middleware in network centric operations. To this end, we characterize a set of Information Technology capabilities that such a middleware should implement. Afterwards, we will discuss the design and architectural aspects of these capabilities. This will lead us to an efficient and practical decision support system that we call a digital cockpit. The latter is essentially a multi-tier IT platform that provides a plethora of services such as: data and service integration, monitoring, analysis, and process optimization. Moreover, the platform uses advanced display mechanisms to render the information in a structured and navigational representation that offers the possibility to drill down into the details. A significant subgoal of the paper is to discuss the quality attributes of such an NCO middleware. Finally, we present the results of an implementation of the aforesaid platform architecture.     

Modeling software bidding risks

We discuss a method of developing a software bidding model that allows users to visualize the uncertainty involved in pricing decisions and make appropriate bid/no bid decisions. We present a generic bidding model developed using the modeling method. The model elements were identified after a review of bidding research in software and other industries. We describe the method we developed to validate our model and report the main results of our model validation, including the results of applying the model to four bidding scenarios.     

A Critical Guide to Empirical Validation of Agent-Based Models in Economics: Methodologies,Procedures, and Open Problems

This paper addresses the methodological problems of empirical validation in agent-based (AB) models in economics and how these are currently being tackled. We first identify a set of issues that are common to all modelers engaged in empirical validation. We then propose a novel taxonomy, which captures the relevant dimensions along which AB economics models differ. We argue that these dimensions affect the way in which empirical validation is carried out by AB modelers and we critically discuss the main alternative approaches to empirical validation being developed in AB economics. We conclude by focusing on a set of (as yet) unresolved issues for empirical validation that require future research.