From signals to knowledge: A conceptual model for multimodal learning analytics

Multimodality in learning analytics and learning science is under the spotlight. The landscape of sensors and wearable trackers that can be used for learning support is evolving rapidly, as well as data collection and analysis methods. Multimodal data can now be collected and processed in real time at an unprecedented scale. With sensors, it is possible to capture observable events of the learning process such as learner's behaviour and the learning context. The learning process, however, consists also of latent attributes, such as the learner's cognitions or emotions. These attributes are unobservable to sensors and need to be elicited by human‐driven interpretations. We conducted a literature survey of experiments using multimodal data to frame the young research field of multimodal learning analytics. The survey explored the multimodal data used in related studies (the input space) and the learning theories selected (the hypothesis space). The survey led to the formulation of the Multimodal Learning Analytics Model whose main objectives are of (O1) mapping the use of multimodal data to enhance the feedback in a learning context; (O2) showing how to combine machine learning with multimodal data; and (O3) aligning the terminology used in the field of machine learning and learning science.     

Label distribution for multimodal machine learning

Multimodal machine learning(MML)aims to understand the world from multiple related modalities.It has attracted much attention as multimodal data has become increasingly available in real-world application.It is shown that MML can perform better than single-modal machine learning,since multi-modalities containing more information which could complement each other.However,it is a key challenge to fuse the multi-modalities in MML.Different from previous work,we further consider the side-information,which reflects the situation and influences the fusion of multi-modalities.We recover multimodal label distribution(MLD)by leveraging the side-information,representing the degree to which each modality contributes to describing the instance.Accordingly,a novel framework named multimodal label distribution learning(MLDL)is proposed to recover the MLD,and fuse the multimodalities with its guidance to learn an in-depth understanding of the jointly feature representation.Moreover,two versions of MLDL are proposed to deal with the sequential data.Experiments on multimodal sentiment analysis and disease prediction show that the proposed approaches perform favorably against state-of-the-art methods.     

The effect of a graph‐oriented computer‐assisted project‐based learning environment on argumentation skills

The purpose of this quasi‐experimental study was to explore how seventh graders in a suburban school in the United States developed argumentation skills and science knowledge in a project‐based learning environment that incorporated a graph‐oriented, computer‐assisted application. A total of 54 students (three classes) comprised this treatment condition and were engaged in a project‐based learning environment that incorporated a graph‐oriented, computer‐assisted application, whereas a total of 57 students (three classes) comprised the control condition and were engaged in a project‐based learning environment without this graph‐oriented, computer‐assisted application. Verbal collaborative argumentation was recorded and the students' post essays were collected. A random effects analysis of variance (ANOVA) was conducted and a significant difference in science knowledge about alternative energies between conditions was observed. A multivariate analysis of variance (MANOVA) was conducted and there was a significant difference in counterargument and rebuttal skills between conditions. A qualitative analysis was conducted to examine how the graph‐oriented, computer‐assisted application supported students' development of argumentation skills and affected the quality of collaborative argumentation. The difference in argumentation structure and quality of argumentation between conditions might explain a difference in science knowledge as well counterargument and rebuttal skills (argumentation) between both conditions. This study concluded that a project‐based learning environment incorporating a graph‐oriented, computer‐assisted application was effective in improving students' science knowledge and developing their scientific argumentation skills.     

Social skills as predictors of communication,performance and quality of collaboration in project‐based learning

The assumption that social skills are necessary ingredients of collaborative learning is well established but rarely empirically tested. In addition, most theories on collaborative learning focus on social skills only at the personal level, while the social skill configurations within a learning group might be of equal importance. Using the integrative framework, this study investigates which social skills at the personal level and at the group level are predictive of task‐related e‐mail communication, satisfaction with performance and perceived quality of collaboration. Data collection took place in a technology‐enhanced long‐term project‐based learning setting for pre‐service teachers. For data collection, two questionnaires were used, one at the beginning and one at the end of the learning cycle which lasted 3 months. During the project phase, the e‐mail communication between group members was captured as well. The investigation of 60 project groups (N = 155 for the questionnaires; group size: two or three students) and 33 groups for the e‐mail communication (N = 83) revealed that personal social skills played only a minor role compared to group level configurations of social skills in predicting satisfaction with performance, perceived quality of collaboration and communication behaviour. Members from groups that showed a high and/or homogeneous configuration of specific social skills (e.g., cooperation/compromising, leadership) usually were more satisfied and saw their group as more efficient than members from groups with a low and/or heterogeneous configuration of skills.     

A cross‐cultural study of the effect of a graph‐oriented computer‐assisted project‐based learning environment on middle school students' science knowledge and argumentation skills

The purpose of this mixed‐methods study was to explore how seventh graders in a suburban school in the United States and sixth graders in an urban school in Taiwan developed argumentation skills and science knowledge in a project‐based learning environment that incorporated a graph‐oriented, computer‐assisted application (GOCAA). A total of 42 students comprised the treatment condition and were engaged in a project‐based learning environment that incorporated a GOCAA. Of these 42 students, 21 were located in the United States and 21 were located in Taiwan. A total of 26 students comprised the control condition and were engaged in a project‐based learning environment without the GOCAA. Of these 26 students, 15 were in the United States and 11 were in Taiwan. In each country, verbal collaborative argumentation was recorded and the students' post‐essays were collected. A one‐way analysis of variance (ANOVA) was conducted for each measure of science knowledge about alternative energies. The results showed a significant treatment effect for the outcome of scientific explanation among U.S. students, while among Taiwanese students, a significant treatment effect on scientific facts was observed. A one‐way ANOVA was additionally conducted for each measure of argumentation skills and a significant treatment effect on counterarguments and rebuttals was observed among the U.S. students, while in Taiwan, a significant treatment effect on reasoning and rebuttals was observed. A qualitative analysis was conducted to examine how the GOCAA supported students' development of argumentation skills in different countries. This study found distinct argumentation patterns between the U.S. and Taiwanese intervention teams. Additionally, a distinct gender difference in the use of evidence and division of labour was noted when the Taiwanese teams were compared with the U.S. teams, which may be explained by cultural differences. This study concluded that, in both the United States and Taiwan, a project‐based learning environment incorporating a GOCAA was effective in improving students' science knowledge and developing their scientific argumentation skills.     

Multimodal teaching analytics: Automated extraction of orchestration graphs from wearable sensor data

The pedagogical modelling of everyday classroom practice is an interesting kind of evidence, both for educational research and teachers' own professional development. This paper explores the usage of wearable sensors and machine learning techniques to automatically extract orchestration graphs (teaching activities and their social plane over time) on a dataset of 12 classroom sessions enacted by two different teachers in different classroom settings. The dataset included mobile eye‐tracking as well as audiovisual and accelerometry data from sensors worn by the teacher. We evaluated both time‐independent and time‐aware models, achieving median F1 scores of about 0.7–0.8 on leave‐one‐session‐out k‐fold cross‐validation. Although these results show the feasibility of this approach, they also highlight the need for larger datasets, recorded in a wider variety of classroom settings, to provide automated tagging of classroom practice that can be used in everyday practice across multiple teachers.     

Mapping the study of learning analytics in higher education

ABSTRACT

In recent years, the application of technological innovation in higher education has become more and more widely spread, and technological innovation has been improving the level of education. In the research of higher education with innovation technology, one of the main focuses is on the dynamic data which can lay a foundation for the analysis of educational activities by learning analytics. The dynamic data created by technological innovation will become the key basis for analytical research and development in higher education. The methods and analysis results of learning analytics will directly affect decision-making and strategy about higher education. In this paper, we use bibliometric and visualisation methods to review the literature, in order to highlight the development of learning analytics in higher education. Using bibliometric analysis, our study depicts the development process of the main methods used in learning analytics, and summarises the current situation in this field, which increases the level of understanding provided by those studies. Finally, we summarise the research hotspots and study trends, which will be useful for future study in this field.     

Meta‐heuristic multi‐ and many‐objective optimization techniques for solution of machine learning problems

Recently, multi‐ and many‐objective meta‐heuristic algorithms have received considerable attention due to their capability to solve optimization problems that require more than one fitness function. This paper presents a comprehensive study of these techniques applied in the context of machine learning problems. Three different topics are reviewed in this work: (a) feature extraction and selection, (b) hyper‐parameter optimization and model selection in the context of supervised learning, and (c) clustering or unsupervised learning. The survey also highlights future research towards related areas.     

Challenges and opportunities of multimodal data in human learning: The computer science students' perspective

Multimodal data have the potential to explore emerging learning practices that extend human cognitive capacities. A critical issue stretching in many multimodal learning analytics (MLA) systems and studies is the current focus aimed at supporting researchers to model learner behaviours, rather than directly supporting learners. Moreover, many MLA systems are designed and deployed without learners' involvement. We argue that in order to create MLA interfaces that directly support learning, we need to gain an expanded understanding of how multimodal data can support learners' authentic needs. We present a qualitative study in which 40 computer science students were tracked in an authentic learning activity using wearable and static sensors. Our findings outline learners' curated representations about multimodal data and the non-technical challenges in using these data in their learning practice. The paper discusses 10 dimensions that can serve as guidelines for researchers and designers to create effective and ethically aware student-facing MLA innovations.     

Advanced machine learning approaches to personalise learning: learning analytics and decision making

The aim of the paper is to present methodology to personalise learning using learning analytics and to make further decisions on suitability, acceptance and use of personalised learning units. In the paper, first of all, related research review is presented. Further, an original methodology to personalise learning applying learning analytics in virtual learning environments and empirical research results are presented. Using this learning personalisation methodology, decision-making model and method are proposed to evaluate suitability, acceptance and use of personalised learning units. Personalised learning units evaluation methodology presented in the paper is based on (1) well-known principles of Multiple Criteria Decision Analysis for identifying evaluation criteria; (2) Educational Technology Acceptance & Satisfaction Model (ETAS-M) based on well-known Unified Theory on Acceptance and Use of Technology (UTAUT) model, and (3) probabilistic suitability indexes to identify learning components’ suitability to particular students’ needs according to their learning styles. In the paper, there are also examples of implementing the methodology using different weights of evaluation criteria. This methodology is applicable in real life situations where teachers have to help students to create and apply learning units that are most suitable for their needs and thus to improve education quality and efficiency.     

The impact of team‐based learning on students with different self‐regulated learning abilities

Team‐based learning (TBL) stresses applying knowledge rather than absorbing knowledge in class; studies have investigated the use of TBL and its merits in different teaching courses (e.g., medical science and business). TBL is most effective when students learn autonomously before class. However, the ability of autonomous learning is highly associated with the ability of self‐regulated learning (SRL); most importantly, not every student possesses good (or high) SRL ability. Nevertheless, few studies have compared the effectiveness of TBL in students with different SRL abilities. To address this issue, this study analyzed approximately 90 students, whose course teaching involves office application software (Microsoft Excel). This study also developed an online TBL system (called Online TBL) to facilitate performing TBL and to collect the learning behaviours of students with different (high or low) SRL abilities on each TBL stage. The analytical results show that compared with the low‐SRL students, the high‐SRL students were more prepared for class because they spent more reviewing material and had better scores on personal uploaded Excel and Individual Readiness Assurance Test. From the feedback of the peer evaluation, the results also show that the high‐SRL students received more credits than the low‐SRL students did. The questionnaire survey revealed that both low‐SRL and high‐SRL students had a favourable impression of TBL. Further discussion is given to explain the above results.     

Applying learning analytics dashboards based on process-oriented feedback to improve students' learning effectiveness

With the development of a technology-supported environment, it is plausible to provide rich process-oriented feedback in a timely manner. In this paper, we developed a learning analytics dashboard (LAD) based on process-oriented feedback in iTutor to offer learners their final scores, sub-scale reports, and corresponding suggestions on further learning content. We adopted a quasi-experimental design to investigate the effectiveness of the report on students' learning. Ninety-four freshman from two classes participated in this research. The two classes were divided into the LAD group and the original analytics report (OAR) based on a product-oriented feedback group. Before the experiment, all the students took the prior knowledge assessment. After a semester's instruction, all the students took the post-test of the summative assessment. Results indicated that students in the LAD group experienced better learning effectiveness than students in the OAR group. LAD based on process-oriented feedback was also effective in improving the skill learning effectiveness of the students with low-level prior knowledge.     

A static region‐based compiler for the Dalvik virtual machine

The Dalvik virtual machine (VM) is an integral component used to execute applications in Android, which is one of the leading operating systems for mobile devices. The Dalvik VM is an interpreter and is equipped with a trace‐based just‐in‐time compiler for enhancing the execution performance of frequently executed paths, or traces. However, traces generated by the Dalvik VM can be stopped in a conditional branch or a method call/return, which means that these traces usually have a short lifetime, decreasing the effectiveness of the compiler optimizations applied to them. Furthermore, the just‐in‐time compiler applies only a few simple optimizations because of performance considerations. In this article we present a traces‐to‐region (T2R) framework that extends traces to regions and statically compiles these regions into native binaries so as to improve the execution of Android applications. The T2R framework involves three main stages: (i) the profiling stage, in which the run‐time trace information of an application is extracted; (ii) the compilation stage, in which regions are constructed from the extracted traces and are statically compiled into a native binary; and (iii) the execution stage, in which the compiled binary is loaded into the code cache when the application starts to execute. Experiments performed on an Android tablet demonstrated that the T2R framework was effective in improving the execution performance of applications by 10.5–16.2% and decreasing the size of the code cache by 4.6–28.5%. Copyright © 2015 John Wiley & Sons, Ltd.     

基于多任务学习的时序多模态情感分析模型

针对时序多模态情感分析中存在的单模态特征表示和跨模态特征融合问题,结合多头注意力机制,提出一种基于多任务学习的情感分析模型.首先,使用卷积神经网络(CNN)、双向门控循环神经网络(BiGRU)和多头自注意力(MHSA)实现了对时序单模态的特征表示;然后,利用多头注意力实现跨模态的双向信息融合;最后,基于多任务学习思想,...     

Creating user‐generated content for location‐based learning: an authoring framework

Two recent emerging trends are that of Web 2.0, where users actively create content and publish it on the Web, and also location awareness, where a digital device utilizes a person's physical location as the context to provide specific services and/or information. This paper examines how these two phenomena can be brought together so that user‐generated content on mobile devices is used to provide informal learning opportunities relevant to a person's location. However, the generative process of such media does not always have much guidance on how or what to create, so the quality of such information can be highly variable. To overcome this, a framework has been designed to guide the authoring of user‐generated content so that it can be used for informal learning about one's immediate surroundings (particularly in an outdoor setting), combining pedagogical aspects with those from human–computer interaction and environmental aesthetics. The framework consists of six dimensions that include aspects such as curriculum area (e.g. science, art), type of communication, use of language/media related to the landscape, knowledge level of content, contextual aspects, and types of interaction. In order to test the framework before it could be used to scaffold new content, it was first used to analyse and evaluate over 200 items of existing user‐generated content, to investigate the appropriateness of the proposed dimensions and the items contained therein or if any were missing. This paper presents the findings of this initial testing phase, together with a discussion of how the framework can be improved, in order to help scaffold the creation of new user‐generated content in the future.     

Applying social learning analytics to message boards in online distance learning: A case study

Social learning analytics introduces tools and methods that help improving the learning process by providing useful information about the actors and their activity in the learning system. This study examines the relation between SNA parameters and student outcomes, between network parameters and global course performance, and it shows how visualizations of social learning analytics can help observing the visible and invisible interactions occurring in online distance education.The findings from our empirical study show that future research should further investigate whether there are conditions under which social network parameters are reliable predictors of academic performance, but also advises against relying exclusively in social network parameters for predictive purposes. The findings also show that data visualization is a useful tool for social learning analytics, and how it may provide additional information about actors and their behaviors for decision making in online distance learning.     

Scaffolding wiki‐supported collaborative learning for small‐group projects and whole‐class collaborative knowledge building

While educators value wikis' potential, wikis may fail to support collaborative constructive learning without careful scaffolding. This article proposes literature‐based instructional methods, revised based on two expert instructors' input, presents the collected empirical evidence on the effects of these methods and proposes directions for future refinements. The instructional methods were implemented by an expert instructor teaching a 12‐week 68‐student undergraduate design class in Canada. Data were collected from observations, interviews and content analysis of wikis. The findings revealed that in small‐group project (SGP), the wiki instructional methods enhanced collaborative learning with most instructional methods derived from cooperative learning, but in whole‐class collaborative knowledge building (CKB), the wiki instructional mehtods failed to turn the class into a self‐sustained learning community after the scaffolding faded. We conclude that the genre of wikis should be different for SGP and CKB. While the students easily adopted the ‘reproduced’ genre of wikis for SGP with familiar tasks, they felt overwhelmed or resistant to the unfamiliar ‘emergent’ genre of wikis for CKB in massive collaborative constructive learning. Therefore, we propose that future refinements for wiki‐supported CKB should focus on providing students scaffolding for intersubjectivity (understanding collaborative constructive learning) and transfer of responsibility (developing autonomy).     

基于马氏椭球学习机的监督野点探测

针对有少量野点出现的情况,提出一种基于马氏椭球学习机的监督野点探测（supervised outlier detection based on Mahalanobis ellipsoidal learning machine,SODMELM）方法。这种方法通过一个超椭球对正常类进行较好的描述的同时,将野点排除在该椭球外面,继承了马氏椭球学习机（Mahalanobis Ellipsoidal Learning Machine,MELM）将样本点的协方差矩阵即样本点的分布信息考虑进去的优点。真实数据上的实验表明了所提的方法在一般意义上能提高野点探测的效率。