Supervised machine learning in multimodal learning analytics for estimating success in project‐based learning

Multimodal learning analytics provides researchers new tools and techniques to capture different types of data from complex learning activities in dynamic learning environments. This paper investigates the use of diverse sensors, including computer vision, user‐generated content, and data from the learning objects (physical computing components), to record high‐fidelity synchronised multimodal recordings of small groups of learners interacting. We processed and extracted different aspects of the students' interactions to answer the following question: Which features of student group work are good predictors of team success in open‐ended tasks with physical computing? To answer this question, we have explored different supervised machine learning approaches (traditional and deep learning techniques) to analyse the data coming from multiple sources. The results illustrate that state‐of‐the‐art computational techniques can be used to generate insights into the "black box" of learning in students' project‐based activities. The features identified from the analysis show that distance between learners' hands and faces is a strong predictor of students' artefact quality, which can indicate the value of student collaboration. Our research shows that new and promising approaches such as neural networks, and more traditional regression approaches can both be used to classify multimodal learning analytics data, and both have advantages and disadvantages depending on the research questions and contexts being investigated. The work presented here is a significant contribution towards developing techniques to automatically identify the key aspects of students success in project‐based learning environments, and to ultimately help teachers provide appropriate and timely support to students in these fundamental aspects.     

Multimodal teaching analytics: Automated extraction of orchestration graphs from wearable sensor data

The pedagogical modelling of everyday classroom practice is an interesting kind of evidence, both for educational research and teachers' own professional development. This paper explores the usage of wearable sensors and machine learning techniques to automatically extract orchestration graphs (teaching activities and their social plane over time) on a dataset of 12 classroom sessions enacted by two different teachers in different classroom settings. The dataset included mobile eye‐tracking as well as audiovisual and accelerometry data from sensors worn by the teacher. We evaluated both time‐independent and time‐aware models, achieving median F1 scores of about 0.7–0.8 on leave‐one‐session‐out k‐fold cross‐validation. Although these results show the feasibility of this approach, they also highlight the need for larger datasets, recorded in a wider variety of classroom settings, to provide automated tagging of classroom practice that can be used in everyday practice across multiple teachers.     

Label distribution for multimodal machine learning

Multimodal machine learning(MML)aims to understand the world from multiple related modalities.It has attracted much attention as multimodal data has become increasingly available in real-world application.It is shown that MML can perform better than single-modal machine learning,since multi-modalities containing more information which could complement each other.However,it is a key challenge to fuse the multi-modalities in MML.Different from previous work,we further consider the side-information,which reflects the situation and influences the fusion of multi-modalities.We recover multimodal label distribution(MLD)by leveraging the side-information,representing the degree to which each modality contributes to describing the instance.Accordingly,a novel framework named multimodal label distribution learning(MLDL)is proposed to recover the MLD,and fuse the multimodalities with its guidance to learn an in-depth understanding of the jointly feature representation.Moreover,two versions of MLDL are proposed to deal with the sequential data.Experiments on multimodal sentiment analysis and disease prediction show that the proposed approaches perform favorably against state-of-the-art methods.     

Applying social learning analytics to message boards in online distance learning: A case study

Social learning analytics introduces tools and methods that help improving the learning process by providing useful information about the actors and their activity in the learning system. This study examines the relation between SNA parameters and student outcomes, between network parameters and global course performance, and it shows how visualizations of social learning analytics can help observing the visible and invisible interactions occurring in online distance education.The findings from our empirical study show that future research should further investigate whether there are conditions under which social network parameters are reliable predictors of academic performance, but also advises against relying exclusively in social network parameters for predictive purposes. The findings also show that data visualization is a useful tool for social learning analytics, and how it may provide additional information about actors and their behaviors for decision making in online distance learning.     

Predicting students' knowledge after playing a serious game based on learning analytics data: A case study

Serious games have proven to be a powerful tool in education to engage, motivate, and help students learn. However, the change in student knowledge after playing games is usually measured with traditional (paper) prequestionnaires–postquestionnaires. We propose a combination of game learning analytics and data mining techniques to predict knowledge change based on in-game student interactions. We have tested this approach in a case study for which we have conducted preexperiments–postexperiments with 227 students playing a previously validated serious game on first aid techniques. We collected student interaction data while students played, using a game learning analytics infrastructure and the standard data format Experience API for Serious Games. After data collection, we developed and tested prediction models to determine whether knowledge, given as posttest results, can be accurately predicted. Additionally, we compared models both with and without pretest information to determine the importance of previous knowledge when predicting postgame knowledge. The high accuracy of the obtained prediction models suggests that serious games can be used not only to teach but also to measure knowledge acquisition after playing. This will simplify serious games application for educational settings and especially in the classroom easing teachers' evaluation tasks.     

Challenges and opportunities of multimodal data in human learning: The computer science students' perspective

Multimodal data have the potential to explore emerging learning practices that extend human cognitive capacities. A critical issue stretching in many multimodal learning analytics (MLA) systems and studies is the current focus aimed at supporting researchers to model learner behaviours, rather than directly supporting learners. Moreover, many MLA systems are designed and deployed without learners' involvement. We argue that in order to create MLA interfaces that directly support learning, we need to gain an expanded understanding of how multimodal data can support learners' authentic needs. We present a qualitative study in which 40 computer science students were tracked in an authentic learning activity using wearable and static sensors. Our findings outline learners' curated representations about multimodal data and the non-technical challenges in using these data in their learning practice. The paper discusses 10 dimensions that can serve as guidelines for researchers and designers to create effective and ethically aware student-facing MLA innovations.     

视觉语言多模态预训练综述

在多模态机器学习领域,为特定任务而制作的人工标注数据昂贵,且不同任务难以进行迁移,从而需要大量重新训练,导致训练多个任务时效率低下、资源浪费。预训练模型通过以自监督为代表的方式进行大规模数据训练,对数据集中不同模态的信息进行提取和融合,以学习其中蕴涵的通用知识表征,从而服务于广泛的相关下游视觉语言多模态任务,这一方法逐渐成为人工智能各领域的主流方法。依靠互联网所获取的大规模图文对与视频数据,以及以自监督学习为代表的预训练方法的进步,视觉语言多模态预训练模型在很大程度上打破了不同视觉语言任务之间的壁垒,提升了多个任务训练的效率并促进了具体任务的性能表现。本文总结视觉语言多模态预训练领域的进展,首先对常见的预训练数据集和预训练方法进行汇总,然后对目前最新方法以及经典方法进行系统概述,按输入来源分为图像—文本预训练模型和视频—文本多模态模型两大类,阐述了各方法之间的共性和差异,并将各模型在具体下游任务上的实验情况进行汇总。最后,总结了视觉语言预训练面临的挑战和未来发展趋势。     

基于多任务学习的时序多模态情感分析模型

针对时序多模态情感分析中存在的单模态特征表示和跨模态特征融合问题,结合多头注意力机制,提出一种基于多任务学习的情感分析模型.首先,使用卷积神经网络(CNN)、双向门控循环神经网络(BiGRU)和多头自注意力(MHSA)实现了对时序单模态的特征表示;然后,利用多头注意力实现跨模态的双向信息融合;最后,基于多任务学习思想,...     

Assessing the validity of a learning analytics expectation instrument: A multinational study

To assist higher education institutions in meeting the challenge of limited student engagement in the implementation of Learning Analytics services, the Questionnaire for Student Expectations of Learning Analytics (SELAQ) was developed. This instrument contains 12 items, which are explained by a purported two-factor structure of “Ethical and Privacy Expectations” and “Service Feature Expectations.” As it stands, however, the SELAQ has only been validated with students from UK university, which is problematic on account of the interest in Learning Analytics extending beyond this context. Thus, the aim of the current work was to assess whether the translated SELAQ can be validated in three contexts (an Estonian, a Spanish, and a Dutch University). The findings show that the model provided acceptable fits in both the Spanish and Dutch samples, but was not supported in the Estonian student sample. In addition, an assessment of local fit is undertaken for each sample, which provides important points that need to be considered in future work. Finally, a general comparison of expectations across contexts is undertaken, which are discussed in relation to the General Data Protection Regulation (2018).     

Applying learning analytics dashboards based on process-oriented feedback to improve students' learning effectiveness

With the development of a technology-supported environment, it is plausible to provide rich process-oriented feedback in a timely manner. In this paper, we developed a learning analytics dashboard (LAD) based on process-oriented feedback in iTutor to offer learners their final scores, sub-scale reports, and corresponding suggestions on further learning content. We adopted a quasi-experimental design to investigate the effectiveness of the report on students' learning. Ninety-four freshman from two classes participated in this research. The two classes were divided into the LAD group and the original analytics report (OAR) based on a product-oriented feedback group. Before the experiment, all the students took the prior knowledge assessment. After a semester's instruction, all the students took the post-test of the summative assessment. Results indicated that students in the LAD group experienced better learning effectiveness than students in the OAR group. LAD based on process-oriented feedback was also effective in improving the skill learning effectiveness of the students with low-level prior knowledge.     

Advanced machine learning approaches to personalise learning: learning analytics and decision making

The aim of the paper is to present methodology to personalise learning using learning analytics and to make further decisions on suitability, acceptance and use of personalised learning units. In the paper, first of all, related research review is presented. Further, an original methodology to personalise learning applying learning analytics in virtual learning environments and empirical research results are presented. Using this learning personalisation methodology, decision-making model and method are proposed to evaluate suitability, acceptance and use of personalised learning units. Personalised learning units evaluation methodology presented in the paper is based on (1) well-known principles of Multiple Criteria Decision Analysis for identifying evaluation criteria; (2) Educational Technology Acceptance & Satisfaction Model (ETAS-M) based on well-known Unified Theory on Acceptance and Use of Technology (UTAUT) model, and (3) probabilistic suitability indexes to identify learning components’ suitability to particular students’ needs according to their learning styles. In the paper, there are also examples of implementing the methodology using different weights of evaluation criteria. This methodology is applicable in real life situations where teachers have to help students to create and apply learning units that are most suitable for their needs and thus to improve education quality and efficiency.     

A conceptual framework for trajectory-based medical analytics with IoT contexts

With the increasing availability of medical IoT devices, it becomes feasible to collect health contexts from personal IoT devices and perform analytics to assess the health conditions and identify potential diseases. However, there exist technical challenges in developing such personal healthcare applications; transforming disease diagnosis methods of physicians in machine-readable forms and software implementation of the medical analytic schemes. In this paper, we present a methodology foundation for analyzing medical contexts. That is, we formalize five representative medical diagnosis schemes for personal healthcare application.     

Construction equipment activity recognition for simulation input modeling using mobile sensors and machine learning classifiers

Although activity recognition is an emerging general area of research in computer science, its potential in construction engineering and management (CEM) domain has not yet been fully investigated. Due to the complex and dynamic nature of many construction and infrastructure projects, the ability to detect and classify key activities performed in the field by various equipment and human crew can improve the quality and reliability of project decision-making and control. In particular to simulation modeling, process-level knowledge obtained as a result of activity recognition can help verify and update the input parameters of simulation models. Such input parameters include but are not limited to activity durations and precedence, resource flows, and site layout. The goal of this research is to investigate the prospect of using built-in smartphone sensors as ubiquitous multi-modal data collection and transmission nodes in order to detect detailed construction equipment activities which can ultimately contribute to the process of simulation input modeling. A case study of front-end loader activity recognition is presented to describe the methodology for action recognition and evaluate the performance of the developed system. In the designed methodology, certain key features are extracted from the collected data using accelerometer and gyroscope sensors, and a subset of the extracted features is used to train supervised machine learning classifiers. In doing so, several important technical details such as selection of discriminating features to extract, sensitivity analysis of data segmentation window size, and choice of the classifier to be trained are investigated. It is shown that the choice of the level of detail (LoD) in describing equipment actions (classes) is an important factor with major impact on the classification performance. Results also indicate that although decreasing the number of classes generally improves the classification output, considering other factors such as actions to be combined as a single activity, methodologies to extract knowledge from classified activities, computational efficiency, and end use of the classification process may as well influence one’s decision in selecting an optimal LoD in describing equipment activities (classes).     

ProcessAtlas: A scalable and extensible platform for business process analytics

In today's knowledge‐, service‐, and cloud‐based economy, an overwhelming amount of business‐related data are being generated at a fast rate daily from a wide range of sources. These data increasingly show all the typical properties of big data: wide physical distribution, diversity of formats, nonstandard data models, and independently managed and heterogeneous semantics. In this context, there is a need for new scalable and process‐aware services for querying, exploration, and analysis of process data in the enterprise because (1) process data analysis services should be capable of processing and querying large amount of data effectively and efficiently and, therefore, have to be able to scale well with the infrastructure's scale and (2) the querying services need to enable users to express their data analysis and querying needs using process‐aware abstractions rather than other lower‐level abstractions. In this paper, we introduce ProcessAtlas, ie, an extensible large‐scale process data querying and analysis platform for analyzing process data in the enterprise. The ProcessAtlas platform offers an extensible architecture by adopting a service‐based model so that new analytical services can be plugged into the platform. In ProcessAtlas, we present a domain‐specific model for representing process knowledge, ie, process‐level entities, abstractions, and the relationships among them modeled as graphs. We provide services for discovering, extracting, and analyzing process data. We provide efficient mapping and execution of process‐level queries into graph‐level queries by using scalable process query services to deal with the process data size growth and with the infrastructure's scale. We have implemented ProcessAtlas as a MapReduce‐based prototype and report on experiments performed on both synthetic and real‐world datasets.     

LMR-CBT: learning modality-fused representations with CB-Transformer for multimodal emotion recognition from unaligned multimodal sequences

Learning modality-fused representations and processing unaligned multimodal sequences are meaningful and challenging in multimodal emotion recognition. Existing approaches use directional pairwise attention or a message hub to fuse language, visual, and audio modalities. However, these fusion methods are often quadratic in complexity with respect to the modal sequence length, bring redundant information and are not efficient. In this paper, we propose an efficient neural network to learn modality-fused representations with CB-Transformer (LMR-CBT) for multimodal emotion recognition from unaligned multi-modal sequences. Specifically, we first perform feature extraction for the three modalities respectively to obtain the local structure of the sequences. Then, we design an innovative asymmetric transformer with cross-modal blocks (CB-Transformer) that enables complementary learning of different modalities, mainly divided into local temporal learning, cross-modal feature fusion and global self-attention representations. In addition, we splice the fused features with the original features to classify the emotions of the sequences. Finally, we conduct word-aligned and unaligned experiments on three challenging datasets, IEMOCAP, CMU-MOSI, and CMU-MOSEI. The experimental results show the superiority and efficiency of our proposed method in both settings. Compared with the mainstream methods, our approach reaches the state-of-the-art with a minimum number of parameters.     

Text analytics in industry: Challenges,desiderata and trends

The recent decades have witnessed an unprecedented expansion in the volume of unstructured data in digital textual formats. Companies are now starting to recognize the potential economic value lying untapped in their text data repositories and sources, including external ones, such as social media platforms, and internal ones, such as safety reports and other company-specific document collections. Information extracted from these textual data sources is valuable for a range of enterprise application and for informed decision making. In this article we provide a systematic review of the current state of the art in the application of text analytics in industry. Our review is structured along three dimensions: the application context, the methods and techniques utilized, and the evaluation procedure. Based on the review, we identify the different challenges and constraints that an real-world, industrial environment imposes on text analytics techniques, as opposed to their deployment in more controlled, research environments. In addition, we formulate a set of desiderata that text analytics techniques should satisfy in order to alleviate these challenges and to ensure their successful deployment in industry. Furthermore, we discuss future trends in text analytics and their potential application in industry.