Performance-enhanced real-time lifestyle tracking model based on human activity recognition (PERT-HAR) model through smartphones

The identification of human actions and their representation and categorization in an automated system through training and learning is considered the human activity recognition (HAR) process. Tracking systems capture and read human actions ranging from the identification of plain movements to the comprehension of habits and practices. As such, HARs have found their use in areas such as health care, with a special focus on elderly patients’ care, safety arrangements and supervision areas and in applications designed for smart homes. Sensor and visual devices enable HAR, and there is a multitude of sensor classifications, such as sensors that can be worn, sensors tagged to a target and sensors tagged to the background. The automated learning methodologies in HAR are either handcrafted or deep learning or a combination of both. Handcrafted models can be regional or wholesome recognition models such as RGB, 3D mapping and skeleton data models, and deep learning models are categorized into generative models such as LSTM (long short-term memory), discriminative models such as convolutional neural networks (CNNs) or a synthesis of such models. Several datasets are available for undertaking HAR analysis and representation. The hierarchy of processes in HAR is classified into gathering information, preliminary processing, property derivation and guiding based on framed models. The proposed study considers the role of smartphones in HARs with a particular interest in keeping a tab on the lifestyle of subjects. Smartphones act as HAR devices with inbuilt sensors with custom-made applications, and the merits of both handcrafted and deep learning models are considered in framing a model that can enable lifestyle tracking in real time. This performance-enhanced real-time tracking human activity recognition (PERT-HAR) model is economical and effective in accurate identification and representation of actions of the subjects and thereby provides more accurate data for real-time investigation and remedial measures. This model achieves an accuracy of 97–99% in a properly controlled environment.

     

Unsupervised learning for human activity recognition using smartphone sensors

To provide more sophisticated healthcare services, it is necessary to collect the precise information on a patient. One impressive area of study to obtain meaningful information is human activity recognition, which has proceeded through the use of supervised learning techniques in recent decades. Previous studies, however, have suffered from generating a training dataset and extending the number of activities to be recognized. In this paper, to find out a new approach that avoids these problems, we propose unsupervised learning methods for human activity recognition, with sensor data collected from smartphone sensors even when the number of activities is unknown. Experiment results show that the mixture of Gaussian exactly distinguishes those activities when the number of activities k is known, while hierarchical clustering or DBSCAN achieve above 90% accuracy by obtaining k based on Caliński–Harabasz index, or by choosing appropriate values for ɛ and MinPts when k is unknown. We believe that the results of our approach provide a way of automatically selecting an appropriate value of k at which the accuracy is maximized for activity recognition, without the generation of training datasets by hand.     

Probabilistic knowledge infusion through symbolic features for context-aware activity recognition

In the general machine learning domain, solutions based on the integration of deep learning models with knowledge-based approaches are emerging. Indeed, such hybrid systems have the advantage of improving the recognition rate and the model’s interpretability. At the same time, they require a significantly reduced amount of labeled data to reliably train the model. However, these techniques have been poorly explored in the sensor-based Human Activity Recognition (HAR) domain. The common-sense knowledge about activity execution can potentially improve purely data-driven approaches. While a few knowledge infusion approaches have been proposed for HAR, they rely on rigid logic formalisms that do not take into account uncertainty. In this paper, we propose P-NIMBUS, a novel knowledge infusion approach for sensor-based HAR that relies on probabilistic reasoning. A probabilistic ontology is in charge of computing symbolic features that are combined with the features automatically extracted by a CNN model from raw sensor data and high-level context data. In particular, the symbolic features encode probabilistic common-sense knowledge about the activities consistent with the user’s surrounding context. These features are infused within the model before the classification layer. We experimentally evaluated P-NIMBUS on a HAR dataset of mobile devices sensor data that includes 14 different activities performed by 25 users. Our results show that P-NIMBUS outperforms state-of-the-art neuro-symbolic approaches, with the advantage of requiring a limited amount of training data to reach satisfying recognition rates (i.e., more than 80% of F1-score with only 20% of labeled data).     

A survey on the use of machine learning methods in context-aware middlewares for human activity recognition

Human activity recognition (HAR) essentially uses (past) sensor data or complex context information for inferring the activities a user performs in his daily tasks. HAR has been extensively studied using different paradigms, such as different reasoning mechanisms, including probabilistic, rule-based, statistical, logical reasoning, or the machine learning (ML) paradigm, to construct inference models to recognize or predict user activities. ML for HAR allows that activities can be recognized and even anticipated through the analysis of collected data from different sensors, with greater accuracy than the other paradigms. On the other hand, context-aware middlewares (CAMs) can efficiently integrate a large number of different devices and sensors. Moreover, they provide a programmable and auto-configurable infrastructure for streamline the design and construction of software solutions in scenarios where lots of sensors and data are their bases, such as ambient intelligence, smart cities, and e-health domains. In this way, the full integration of ML capabilities as services in CAMs can advance the development of software solutions in these domains when ML is necessary, specially for HAR, which is the basis for many scenarios in these domains. In this work, we present a survey for identifying the state-of-the-art in using ML for HAR in CAMs through a systematic literature review (SLR). In our SLR, we worked to answer four research questions: (i) what are the different types of context reasoners available in CAMs; (ii) what are the ML algorithms and methods used for generating models for context reasoning; (iii) which CAMs support data processing in real time; and (iv) what are the HAR scenarios usually tackled by the research works. In our analysis, we observed that, although ML offers viable approaches to construct inference models for HAR using different ML approaches, including batch learning, adaptive learning and data stream learning, there are yet some gaps and research challenges to be tackled, specially on the use of data stream learning considering concept drift on data, mechanisms for adapting the inference models, and further considering all of this as services in CAMs, specially for HAR.

     

Intelligent Deep Learning Enabled Human Activity Recognition for Improved Medical Services

Human Activity Recognition (HAR) has been made simple in recent years, thanks to recent advancements made in Artificial Intelligence (AI) techniques. These techniques are applied in several areas like security, surveillance, healthcare, human-robot interaction, and entertainment. Since wearable sensor-based HAR system includes in-built sensors, human activities can be categorized based on sensor values. Further, it can also be employed in other applications such as gait diagnosis, observation of children/adult’s cognitive nature, stroke-patient hospital direction, Epilepsy and Parkinson’s disease examination, etc. Recently-developed Artificial Intelligence (AI) techniques, especially Deep Learning (DL) models can be deployed to accomplish effective outcomes on HAR process. With this motivation, the current research paper focuses on designing Intelligent Hyperparameter Tuned Deep Learning-based HAR (IHPTDL-HAR) technique in healthcare environment. The proposed IHPTDL-HAR technique aims at recognizing the human actions in healthcare environment and helps the patients in managing their healthcare service. In addition, the presented model makes use of Hierarchical Clustering (HC)-based outlier detection technique to remove the outliers. IHPTDL-HAR technique incorporates DL-based Deep Belief Network (DBN) model to recognize the activities of users. Moreover, Harris Hawks Optimization (HHO) algorithm is used for hyperparameter tuning of DBN model. Finally, a comprehensive experimental analysis was conducted upon benchmark dataset and the results were examined under different aspects. The experimental results demonstrate that the proposed IHPTDL-HAR technique is a superior performer compared to other recent techniques under different measures.     

基于可穿戴传感器的人体活动识别研究综述

人体活动识别（HAR）在医疗、安全、娱乐等方面有着广泛的应用。随着传感器器件的发展,各类能准确采集人体行为活动数据的传感器在手环、手表、手机等可穿戴设备上得到了广泛使用,相比基于视频图像的行为识别方法,基于传感器的行为识别具有成本低、灵活、可移植性好的特点,因此,基于可穿戴传感器的人体活动识别研究成为行为识别中的研究热点。介绍了人体活动识别研究中原始数据采集、特征提取、特征选择以及分类方法,对识别流程中每一部分常用的技术以及研究现状进行了综述总结,最后分析人体活动识别研究当前存在的主要问题并展望了今后可能的研究方向。     

STAR-Lite: A light-weight scalable self-taught learning framework for older adults’ activity recognition

Recognizing activities for older adults is challenging as we observe a variety of activity patterns caused due to aging (e.g., limited dexterity, limb control, slower response time) or/and underlying health conditions (e.g., dementia). However, existing literature with deep learning methods has successfully recognized activities when the dataset contains high-quality annotations and is captured in a controlled environment. On the contrary, data captured in a real-world environment, especially with older adults exhibiting memory-related symptoms, varying psychological and mental health status, reliance on caregivers to perform daily activities, and unavailability of domain-specific annotators, makes obtaining quality data with annotations challenging; leaving us with limited labeled data and abundant unlabeled data. In this paper, we hypothesize that projecting the labeled data representations comprising a specific set of activities onto a new representation space characterized by the unlabeled data comprising activities beyond the limited activities in the labeled dataset would help us rely less on the annotated data to improve activity detection performance. Motivated by this, we propose STAR-Lite, a self-taught learning framework that involves a pre-training framework to prepare the new representation space considering activities beyond the initial labels in the labeled dataset. STAR-Lite projects the labeled data representations on the new representation space characterized by unlabeled data labels and learns higher-level representations of the labeled dataset while optimizing inter- and intra- class distances without explicitly using a computation hungry similarity-based approach. We demonstrate that our proposed approach, STAR-Lite (a) improves activity recognition performance in a supervised setting and (b) is feasible for real-world deployment. To enhance the feasibility of deploying STAR-Lite on devices with limited memory resources, we explore model compression techniques such as pruning and quantization and propose a novel layer-wise pruning-rate optimization technique that effectively compresses the network while preserving the model performance. The evaluation was performed using the Alzheimer’s Activity Recognition dataset (AAR) captured from 25 individuals living in a retirement community center with IRB approval (#Y18NR12035) using an in-house SenseBox infrastructure while concurrently assessing the clinical evaluation of the participants for dementia, and independent living. Our extensive evaluation reveals that STAR-Lite can detect activities with an F1-score of 85.12% despite 62% reduction in model size and 5% improvement of execution time on a resource constrained device.     

带有协方差矩阵的卷积神经网络在人体运动识别中的应用

目前,深度学习已经在各种人体运动识别（HAR）任务中发挥了重要作用。但是,由于运动数据具有时间序列和包含肢体动作的特殊性,现有神经网络在进行卷积操作时会导致数据高度相关,并且随着网络影响到下一层,这限制了模型的识别效果。为此,提出了一种带有协方差矩阵的改进卷积神经网络用于HAR场景,通过矩阵变换搭建一种去相关的网络结构来消除相关性问题,可以在网络表现不佳时替代现有的批量归一化（BN）层用于归一化数据。在4个HAR公共数据集上进行实验,并与传统CNN和带有BN层的模型进行比较。实验结果表明,对比此前的深度学习网络,改进的神经网络有1%~2%的性能提升,验证了该方法的有效性,并将程序移植到了移动端进行实时运动识别。     

Deep continual transfer learning with dynamic weight aggregation for fault diagnosis of industrial streaming data under varying working conditions

Catastrophic forgetting of learned knowledges and distribution discrepancy of different data are two key problems within fault diagnosis fields of rotating machinery. However, existing intelligent fault diagnosis methods generally tackle either the catastrophic forgetting problem or the domain adaptation problem. In complex industrial environments, both the catastrophic forgetting problem and the domain adaptation problem will occur simultaneously, which is termed as continual transfer problem. Therefore, it is necessary to investigate a more practical and challenging task where the number of fault categories are constantly increasing with industrial streaming data under varying operation conditions. To address the continual transfer problem, a novel framework named deep continual transfer learning network with dynamic weight aggregation (DCTLN-DWA) is proposed in this study. The DWA module is used to retain the diagnostic knowledge learned from previous phases and learn new knowledge from the new samples. The adversarial training strategy is applied to eliminate the data distribution discrepancy between source and target domains. The effectiveness of the proposed framework is investigated on an automobile transmission dataset. The experimental results demonstrate that the proposed framework can effectively handle the industrial streaming data under different working conditions and can be utilized as a promising tool for solving actual industrial problem.     

Human Activity Recognition Based on Parallel Approximation Kernel K-Means Algorithm

Recently, owing to the capability of mobile and wearable devices to sense daily human activity, human activity recognition (HAR) datasets have become a large-scale data resource. Due to the heterogeneity and nonlinearly separable nature of the data recorded by these sensors, the datasets generated require special techniques to accurately predict human activity and mitigate the considerable heterogeneity. Consequently, classic clustering algorithms do not work well with these data. Hence, kernelization, which converts the data into a new feature vector representation, is performed on nonlinearly separable data. This study aims to present a robust method to perform HAR data clustering to mitigate heterogeneity in data with minimal resource consumption. Therefore, we propose a parallel approximated clustering approach to handle the computational cost of big data by addressing noise, heterogeneity, and nonlinearity in data using data reduction, filtering, and approximated clustering methods on parallel computing environments that have not been previously addressed. Our key contribution is to treat HAR as big data implemented by approximation kernel K-means approaches and fill the gap between the HAR clustering cost and parallel computing fields. We implemented our approach on Google cloud on a parallel spark cluster, which helped us to process large-scale HAR data across multiple machines of clusters. The normalized mutual information is used as validation metric to assess the quality of the clustering algorithm. Additionally, the precision, recall, f-score metrics values are obtained somehow to compare the results with a classification technique. The experimental results of our clustering approach prove its effectiveness compared with a classification technique and can efficiently detect physical activity and mitigate the heterogeneity of the datasets.     

Concept drift detection and adaptation for federated and continual learning

Smart devices, such as smartphones, wearables, robots, and others, can collect vast amounts of data from their environment. This data is suitable for training machine learning models, which can significantly improve their behavior, and therefore, the user experience. Federated learning is a young and popular framework that allows multiple distributed devices to train deep learning models collaboratively while preserving data privacy. Nevertheless, this approach may not be optimal for scenarios where data distribution is non-identical among the participants or changes over time, causing what is known as concept drift. Little research has yet been done in this field, but this kind of situation is quite frequent in real life and poses new challenges to both continual and federated learning. Therefore, in this work, we present a new method, called Concept-Drift-Aware Federated Averaging (CDA-FedAvg). Our proposal is an extension of the most popular federated algorithm, Federated Averaging (FedAvg), enhancing it for continual adaptation under concept drift. We empirically demonstrate the weaknesses of regular FedAvg and prove that CDA-FedAvg outperforms it in this type of scenario.

     

Hierarchical querying scheme of human motions for smart home environment

With the recent development of ubiquitous technologies, many new applications have been emerging for smart home implementation. Usually, such applications are based on diverse sensors. One fundamental operation in the applications is to find out semantically meaningful events or activities from huge sensor data stream. Usually, such event or activity is represented by a salient sequence pattern. Among the diverse research issues, detecting salient sequence patterns of human motions from image sensor data stream has received much attention for security and surveillance purposes. In the case of detecting human motions from image sensor data, finding and matching their salient sequence patterns could become more complicated since semantically same motions could show diverse variations such as different motion time. Based on this observation, in this paper, we propose a new querying and answering scheme for continuous sensor data stream to detect abnormal human motions. More specifically, we first present a new hierarchical querying scheme to consider variable length of semantically same human motions. Secondly, we present an indexing scheme to efficiently find semantically meaningful motion sequences in the sensor data stream. Thirdly, we present Dynamic Group Warping algorithm to effectively filter out unnecessary human motions. Through extensive experiments, we show that our proposed method achieves outstanding performance.     

Collaborative concept mapping activities in a classroom scenario

The aim of this study is to test collaborative concept mapping activities using computers in a classroom scenario and to evaluate the possibilities that Elkar-CM offers for collaboratively learning non-technical topics. Elkar-CM is a multi-lingual and multi-media software program designed for drawing concept maps (CMs) collaboratively. Concept mapping is a widely accepted technique that promotes meaningful learning. Graphically representing concepts of the learning domain and relationships between them helps students integrate new knowledge into their current cognitive structure. This study was carried out with Social Education degree students at the University of the Basque Country (UPV/EHU). The experiment included two learning activities. First, all students collaboratively constructed in the classroom a CM on the subject of Moral Development. Second, students were organised into groups to complete the CM generated in the first part.     

Using meta-learning to predict performance metrics in machine learning problems

Machine learning has been facing significant challenges over the last years, much of which stem from the new characteristics of machine learning problems, such as learning from streaming data or incorporating human feedback into existing datasets and models. In these dynamic scenarios, data change over time and models must adapt. However, new data do not necessarily mean new patterns. The main goal of this paper is to devise a method to predict a model's performance metrics before it is trained, in order to decide whether it is worth it to train it or not. That is, will the model hold significantly better results than the current one? To address this issue, we propose the use of meta-learning. Specifically, we evaluate two different meta-models, one built for a specific machine learning problem, and another built based on many different problems, meant to be a generic meta-model, applicable to virtually any problem. In this paper, we focus only on the prediction of the root mean square error (RMSE). Results show that it is possible to accurately predict the RMSE of future models, event in streaming scenarios. Moreover, results also show that it is possible to reduce the need for re-training models between 60% and 98%, depending on the problem and on the threshold used.     

A Novel Framework for Human Activity Recognition with Time Labelled Real Time Sensor Data

Human activity recognition is an effective approach for identifying the characteristics of historical data. In the past decades, different shallow classifiers and handcrafted features were used to identify the activities from the sensor data. These approaches are configured for offline processing and are not suitable for sequential data. This article proposes an adaptive framework for human activity recognition using a deep learning mechanism. This deep learning approach forms the deep belief network (DBN), which contains a visible layer and hidden layers. The processing of raw sensor data is performed by these layers and the activity is identified at the top most layers. The DBN is tested using the real time environment with the help of mobile devices that contain an accelerometer, a magnetometer, and a gyroscope. The results are analyzed with the metrics of precision, recall, and the F1-score. The results proved that the proposed method has a higher F1_score when compared to the existing approach.     

Human activity recognition with smartphone sensors using deep learning neural networks

Human activities are inherently translation invariant and hierarchical. Human activity recognition (HAR), a field that has garnered a lot of attention in recent years due to its high demand in various application domains, makes use of time-series sensor data to infer activities. In this paper, a deep convolutional neural network (convnet) is proposed to perform efficient and effective HAR using smartphone sensors by exploiting the inherent characteristics of activities and 1D time-series signals, at the same time providing a way to automatically and data-adaptively extract robust features from raw data. Experiments show that convnets indeed derive relevant and more complex features with every additional layer, although difference of feature complexity level decreases with every additional layer. A wider time span of temporal local correlation can be exploited (1 × 9–1 × 14) and a low pooling size (1 × 2–1 × 3) is shown to be beneficial. Convnets also achieved an almost perfect classification on moving activities, especially very similar ones which were previously perceived to be very difficult to classify. Lastly, convnets outperform other state-of-the-art data mining techniques in HAR for the benchmark dataset collected from 30 volunteer subjects, achieving an overall performance of 94.79% on the test set with raw sensor data, and 95.75% with additional information of temporal fast Fourier transform of the HAR data set.     

基于无线体域网的在线人体活动识别

基于智能手机传感器的人体活动识别是普适计算领域的研究热点.为扩展可识别的活动种类,并提高准确率和实时性,提出了由智能手环和智能手机组建无线体域网通过深度神经网络在线识别人体活动的方法.首先,设计由智能手环和智能手机组成的无线体域网的总体框架;然后,对预处理后的传感信号,构造带有Inception结构的卷积神经网络和长短时记忆递归神经网络来分别提取时空域特征,并结合两类网络结构来融合多模态传感数据,离线进行神经网络模型训练;最后,对训练好的神经网络模型进行优化,并部署到智能手机上,在线实时识别人体活动.实验结果表明,本文方法无需手工设计特征,可自动融合各类异构传感数据,更加准确、高效地识别了更多种类的活动.     

一种改进的GRU-InFCN人体行为识别模型

为了克服传统机器学习方法在采用传感器数据进行人体行为识别领域上识别效果对人工特征选取依赖严重、识别准确率不高等问题,提出一种改进的全卷积神经网络和多层循环神经网络并联的深度学习模型(GRU-InFCN),并对传感器数据特征进行自动提取,实现人体动作的识别。该模型通过多尺度卷积神经网络和双层GRU网络(Gated Recurrent Unit,GRU)分别对传感器数据进行特征提取,将特征矩阵在矩阵维度上进行特征拼接再通过Softmax完成特征分类。实验结果表明,在开源人体行为识别(HAR)数据集上采用该方法进行人体行为识别,准确率达到了97.76%。该模型在取得高准确率的同时,避免了复杂的信号预处理和特征工程。     

A knowledge engineering approach to developing mindtools for context-aware ubiquitous learning

Recent developments in computing and mobile technologies have enabled the mobile and ubiquitous learning approach, which situates students in an environment that combines real-world and digital-world learning resources. Although such an approach seems to be innovative and interesting, several problems have been revealed when applying it to practical learning activities. One major problem is owing to the lack of proper learning strategies or tools that can guide or assist the students to learn in such a complex learning scenario. Students might feel excited or interested when using the mobile devices to learn in the real world; however, their learning achievements could be disappointing. To cope with this problem, in this study, a knowledge engineering approach is proposed to develop Mindtools for such innovative learning scenarios. Experimental results from a natural science course of an elementary school show that this innovative approach not only enhances learning motivation, but also improves the learning achievements of the students.     

Transfer learning for activity recognition: a survey

Many intelligent systems that focus on the needs of a human require information about the activities being performed by the human. At the core of this capability is activity recognition, which is a challenging and well-researched problem. Activity recognition algorithms require substantial amounts of labeled training data yet need to perform well under very diverse circumstances. As a result, researchers have been designing methods to identify and utilize subtle connections between activity recognition datasets, or to perform transfer-based activity recognition. In this paper, we survey the literature to highlight recent advances in transfer learning for activity recognition. We characterize existing approaches to transfer-based activity recognition by sensor modality, by differences between source and target environments, by data availability, and by type of information that is transferred. Finally, we present some grand challenges for the community to consider as this field is further developed.