Multi-sensor information fusion based on machine learning for real applications in human activity recognition: State-of-the-art and research challenges

This paper firstly introduces common wearable sensors, smart wearable devices and the key application areas. Since multi-sensor is defined by the presence of more than one model or channel, e.g. visual, audio, environmental and physiological signals. Hence, the fusion methods of multi-modality and multi-location sensors are proposed. Despite it has been contributed several works reviewing the stateoftheart on information fusion or deep learning, all of them only tackled one aspect of the sensor fusion applications, which leads to a lack of comprehensive understanding about it. Therefore, we propose using a more holistic approach in order to provide a more suitable starting point from which to develop a full understanding of the fusion methods of wearable sensors. Specifically, this review attempts to provide a more comprehensive survey of the most important aspects of multi-sensor applications for human activity recognition, including those recently added to the field for unsupervised learning and transfer learning. Finally, the open research issues that need further research and improvement are identified and discussed.     

Unsupervised learning for human activity recognition using smartphone sensors

To provide more sophisticated healthcare services, it is necessary to collect the precise information on a patient. One impressive area of study to obtain meaningful information is human activity recognition, which has proceeded through the use of supervised learning techniques in recent decades. Previous studies, however, have suffered from generating a training dataset and extending the number of activities to be recognized. In this paper, to find out a new approach that avoids these problems, we propose unsupervised learning methods for human activity recognition, with sensor data collected from smartphone sensors even when the number of activities is unknown. Experiment results show that the mixture of Gaussian exactly distinguishes those activities when the number of activities k is known, while hierarchical clustering or DBSCAN achieve above 90% accuracy by obtaining k based on Caliński–Harabasz index, or by choosing appropriate values for ɛ and MinPts when k is unknown. We believe that the results of our approach provide a way of automatically selecting an appropriate value of k at which the accuracy is maximized for activity recognition, without the generation of training datasets by hand.     

Construction equipment activity recognition for simulation input modeling using mobile sensors and machine learning classifiers

Although activity recognition is an emerging general area of research in computer science, its potential in construction engineering and management (CEM) domain has not yet been fully investigated. Due to the complex and dynamic nature of many construction and infrastructure projects, the ability to detect and classify key activities performed in the field by various equipment and human crew can improve the quality and reliability of project decision-making and control. In particular to simulation modeling, process-level knowledge obtained as a result of activity recognition can help verify and update the input parameters of simulation models. Such input parameters include but are not limited to activity durations and precedence, resource flows, and site layout. The goal of this research is to investigate the prospect of using built-in smartphone sensors as ubiquitous multi-modal data collection and transmission nodes in order to detect detailed construction equipment activities which can ultimately contribute to the process of simulation input modeling. A case study of front-end loader activity recognition is presented to describe the methodology for action recognition and evaluate the performance of the developed system. In the designed methodology, certain key features are extracted from the collected data using accelerometer and gyroscope sensors, and a subset of the extracted features is used to train supervised machine learning classifiers. In doing so, several important technical details such as selection of discriminating features to extract, sensitivity analysis of data segmentation window size, and choice of the classifier to be trained are investigated. It is shown that the choice of the level of detail (LoD) in describing equipment actions (classes) is an important factor with major impact on the classification performance. Results also indicate that although decreasing the number of classes generally improves the classification output, considering other factors such as actions to be combined as a single activity, methodologies to extract knowledge from classified activities, computational efficiency, and end use of the classification process may as well influence one’s decision in selecting an optimal LoD in describing equipment activities (classes).     

Object recognition using laser range finder and machine learning techniques

In recent years, computer vision has been widely used on industrial environments, allowing robots to perform important tasks like quality control, inspection and recognition. Vision systems are typically used to determine the position and orientation of objects in the workstation, enabling them to be transported and assembled by a robotic cell (e.g. industrial manipulator). These systems commonly resort to CCD (Charge-Coupled Device) Cameras fixed and located in a particular work area or attached directly to the robotic arm (eye-in-hand vision system). Although it is a valid approach, the performance of these vision systems is directly influenced by the industrial environment lighting. Taking all these into consideration, a new approach is proposed for eye-on-hand systems, where the use of cameras will be replaced by the 2D Laser Range Finder (LRF). The LRF will be attached to a robotic manipulator, which executes a pre-defined path to produce grayscale images of the workstation. With this technique the environment lighting interference is minimized resulting in a more reliable and robust computer vision system. After the grayscale image is created, this work focuses on the recognition and classification of different objects using inherent features (based on the invariant moments of Hu) with the most well-known machine learning models: k-Nearest Neighbor (kNN), Neural Networks (NNs) and Support Vector Machines (SVMs). In order to achieve a good performance for each classification model, a wrapper method is used to select one good subset of features, as well as an assessment model technique called K-fold cross-validation to adjust the parameters of the classifiers. The performance of the models is also compared, achieving performances of 83.5% for kNN, 95.5% for the NN and 98.9% for the SVM (generalized accuracy). These high performances are related with the feature selection algorithm based on the simulated annealing heuristic, and the model assessment (k-fold cross-validation). It makes possible to identify the most important features in the recognition process, as well as the adjustment of the best parameters for the machine learning models, increasing the classification ratio of the work objects present in the robot's environment.     

Predictive performance modeling for distributed batch processing using black box monitoring and machine learning

In many domains, the previous decade was characterized by increasing data volumes and growing complexity of data analyses, creating new demands for batch processing on distributed systems. Effective operation of these systems is challenging when facing uncertainties about the performance of jobs and tasks under varying resource configurations, e. g., for scheduling and resource allocation. We survey predictive performance modeling (PPM) approaches to estimate performance metrics such as execution duration, required memory or wait times of future jobs and tasks based on past performance observations. We focus on non-intrusive methods, i. e., methods that can be applied to any workload without modification, since the workload is usually a black box from the perspective of the systems managing the computational infrastructure. We classify and compare sources of performance variation, predicted performance metrics, limitations and challenges, required training data, use cases, and the underlying prediction techniques. We conclude by identifying several open problems and pressing research needs in the field.     

Detecting and locating patterns in time series using machine learning

A method is presented to detect and locate user-defined patterns in time series data. The method is based on decomposing time series into a sequence of fixed-length snapshots on which a classifier is applied. Snapshot classification results determine the exact position of the pattern. One advantage of this approach is that it can be applied to any process-specific pattern, e.g., spiking patterns, under- or overshoots, or (time-lagged) correlations.We demonstrate the efficacy of the approach by means of an example from steel production, namely a cold-rolling mill process. We detect two patterns: underswings and time-lagged spike repetition in multivariate series.     

Consumer credit risk: Individual probability estimates using machine learning

Consumer credit scoring is often considered a classification task where clients receive either a good or a bad credit status. Default probabilities provide more detailed information about the creditworthiness of consumers, and they are usually estimated by logistic regression. Here, we present a general framework for estimating individual consumer credit risks by use of machine learning methods. Since a probability is an expected value, all nonparametric regression approaches which are consistent for the mean are consistent for the probability estimation problem. Among others, random forests (RF), k-nearest neighbors (kNN), and bagged k-nearest neighbors (bNN) belong to this class of consistent nonparametric regression approaches. We apply the machine learning methods and an optimized logistic regression to a large dataset of complete payment histories of short-termed installment credits. We demonstrate probability estimation in Random Jungle, an RF package written in C++ with a generalized framework for fast tree growing, probability estimation, and classification. We also describe an algorithm for tuning the terminal node size for probability estimation. We demonstrate that regression RF outperforms the optimized logistic regression model, kNN, and bNN on the test data of the short-term installment credits.     

Feed-axis gearbox condition monitoring using built-in position sensors and EEMD method

This paper describes a kind of feed-axis gearbox condition monitoring system using the built-in position sensors such as motor encoders and linear scales with high resolution and high precision, which is more directly related to machine dynamics and is sensitive to the early and weak fault. To obtain the position information, several data acquisition approaches for open numerical control systems (NCs) and commercial NCs are suggested. Then, the mathematical models between the faults and the position signals are thoroughly investigated. Finally, the ensemble empirical mode decomposition (EEMD) method is introduced to localize the fault of the feed-axis gearbox. The experimental results show that it is effective to use built-in position sensors and EEMD method for on-line monitoring the gearbox state and fault diagnosis.     

Position-aware activity recognition with wearable devices

Reliable human activity recognition with wearable devices enables the development of human-centric pervasive applications. We aim to develop a robust wearable-based activity recognition system for real life situations where the device position is up to the user or where a user is unable to collect initial training data. Consequently, in this work we focus on the problem of recognizing the on-body position of the wearable device ensued by comprehensive experiments concerning subject-specific and cross-subjects activity recognition approaches that rely on acceleration data. We introduce a device localization method that predicts the on-body position with an F-measure of 89% and a cross-subjects activity recognition approach that considers common physical characteristics. In this context, we present a real world data set that has been collected from 15 participants for 8 common activities where they carried 7 wearable devices in different on-body positions. Our results show that the detection of the device position consistently improves the result of activity recognition for common activities. Regarding cross-subjects models, we identified the waist as the most suitable device location at which the acceleration patterns for the same activity across several people are most similar. In this context, our results provide evidence for the reliability of physical characteristics based cross-subjects models.     

A synergistic approach to manufacturing systems control using machine learning and simulation

This paper describes a synergistic approach that is applicable to a wide variety of system control problems. The approach utilizes a machine learning technique, goal-directed conceptual aggregation (GDCA), to facilitate dynamic decision-making. The application domain employed is Flexible Manufacturing System (FMS) scheduling and control. Simulation is used for the dual purpose of providing a realistic depiction of FMSs, and serves as an engine for demonstrating the viability of a synergistic system involving incremental learning. The paper briefly describes prior approaches to FMS scheduling and control, and machine learning. It outlines the GDCA approach, provides a generalized architecture for dynamic control problems, and describes the implementation of the system as applied to FMS scheduling and control. The paper concludes with a discussion of the general applicability of this approach.     

Practical real-time intrusion detection using machine learning approaches

The growing prevalence of network attacks is a well-known problem which can impact the availability, confidentiality, and integrity of critical information for both individuals and enterprises. In this paper, we propose a real-time intrusion detection approach using a supervised machine learning technique. Our approach is simple and efficient, and can be used with many machine learning techniques. We applied different well-known machine learning techniques to evaluate the performance of our IDS approach. Our experimental results show that the Decision Tree technique can outperform the other techniques. Therefore, we further developed a real-time intrusion detection system (RT-IDS) using the Decision Tree technique to classify on-line network data as normal or attack data. We also identified 12 essential features of network data which are relevant to detecting network attacks using the information gain as our feature selection criterions. Our RT-IDS can distinguish normal network activities from main attack types (Probe and Denial of Service (DoS)) with a detection rate higher than 98% within 2 s. We also developed a new post-processing procedure to reduce the false-alarm rate as well as increase the reliability and detection accuracy of the intrusion detection system.     

Context-based ensemble method for human energy expenditure estimation

Monitoring human energy expenditure (EE) is important in many health and sports applications, since the energy expenditure directly reflects the intensity of physical activity. The actual energy expenditure is unpractical to measure; therefore, it is often estimated from the physical activity measured with accelerometers and other sensors. Previous studies have demonstrated that using a person's activity as the context in which the EE is estimated, and using multiple sensors, improves the estimation. In this study, we go a step further by proposing a context-based reasoning method that uses multiple contexts provided by multiple sensors. The proposed Multiple Contexts Ensemble (MCE) approach first extracts multiple features from the sensor data. Each feature is used as a context for which multiple regression models are built using the remaining features as training data: for each value of the context feature, a regression model is trained on a subset of the dataset with that value. When evaluating a data sample, the models corresponding to the context (feature) values in the evaluated sample are assembled into an ensemble of regression models that estimates the EE of the user. Experiments showed that the MCE method outperforms (in terms of lower root means squared error and lower mean absolute error): (i) five single-regression approaches (linear and non-linear); (ii) two ensemble approaches: Bagging and Random subspace; (iii) an approach that uses artificial neural networks trained on accelerometer-data only; and (iv) BodyMedia (a state-of-the-art commercial EE-estimation device).     

Comprehensive preference learning and feature validity for designing energy-efficient residential buildings using machine learning paradigms

Having a reliable approximation of heating load (HL) and cooling load (CL) is a substantial task for evaluating the energy performance of buildings (EPB). Also, the appearance of soft computing techniques has made many traditional methods antiquated. Thus, the main effort of this study was to evaluate the capability of several learning methods for appraising the HL and CL of a residential building. To this end, a proper dataset consisting of eight influential factors was provided. To simplify the problem, we executed feature validity by using a correlation-based feature subset selection (CfsSubsetEval) technique. The results of this process showed that wall area, overall height, orientation and glazing area have the most significant impact on the HL and CL simulation. After preparing the suitable dataset, sixteen learning methods namely, elastic net (EN), Gaussian process regression (GPR), least median of squares regression (LMSR), multiple linear regression (MLR), multi-layer perceptron regression (MPR), multi-layer perceptron (MLP), radial basis function regression (RBFR), sequential minimal optimization regression (SMOR), functions XNV, lazy K-star, lazy LWL, rules decision table (RDT), M5Rules, alternating model tree (AMT), directional path consistency (DPC), and Random Forest (RF) were developed in Weka environment to forecast the HL and CL variables. Referring to the results, it was concluded that RF, lazy K-star, RDT and AMT outperform other predictive models. Also, comparing the results with the results of the previous studies showed that the applied feature reduction not only did not disturb the learning process but also has enhanced the performance of models. Also, due to the excellent accuracy of the MLP, a formula was derived from the optimized structure of it to predict the HL and CL variables.     

An empirical study of machine learning techniques for affect recognition in human–robot interaction

Given the importance of implicit communication in human interactions, it would be valuable to have this capability in robotic systems wherein a robot can detect the motivations and emotions of the person it is working with. Recognizing affective states from physiological cues is an effective way of implementing implicit human–robot interaction. Several machine learning techniques have been successfully employed in affect-recognition to predict the affective state of an individual given a set of physiological features. However, a systematic comparison of the strengths and weaknesses of these methods has not yet been done. In this paper, we present a comparative study of four machine learning methods—K-Nearest Neighbor, Regression Tree (RT), Bayesian Network and Support Vector Machine (SVM) as applied to the domain of affect recognition using physiological signals. The results showed that SVM gave the best classification accuracy even though all the methods performed competitively. RT gave the next best classification accuracy and was the most space and time efficient.     

Prediction of TBM operation parameters using machine learning models based on BPSO

Rational parameters of TBM (Tunnel Boring Machine) are the key to ensuring efficient and safe tunnel construction. Machine learning (ML) has become the main method for predicting operating parameters. Grid Search and optimization algorithms, such as Particle Swarm Optimization (PSO), are often used to find the hyper parameters of ML models but suffer from excessive time and low accuracy. In order to efficiently construct ML models and enhance the accuracy of predicting models, a BPSO (Beetle antennae search Particle Swarm Optimization) algorithm is proposed. Based on the PSO algorithm, the concept of BAS (Beetle Antennae Search) is integrated into the updating process of an individual particle, which improves the random search capability. The convergence of the BPSO algorithm is discussed in terms of inhomogeneous recursive equations and characteristic roots. Then, based on the proposed BPSO prototype, a hybrid ML model BPSO-XGBoost (eXtreme Gradient Boosting) is proposed. We applied the model to the Hangzhou Central Park tunnel project for the prediction of screw conveyer rotational speed. Finally, our model is compared with existing methods. The experimental results show that the BPSO-based model outperforms other traditional ML methods. The BPSO-XGBoost is more accurate than PSO-XGBoost and BPSO-RandomForest for predicting the speed. Also, it is verified that the hyper parameters optimized by the BPSO are better than those optimized by the original PSO. The comprehensive prediction performance ranking of models is as follows: BPSO-XGBoost > PSO-XGBoost > BPSO-RF > PSO-RF. Our models have preferable engineering application value.     

Automated system for construction specification review using natural language processing

Existing attempts to automate construction document analysis are limited in understanding the varied semantic properties of different documents. Due to the semantic conflicts, the construction specification review process is still conducted manually in practice despite the promising performance of the existing approaches. This research aimed to develop an automated system for reviewing construction specifications by analyzing the different semantic properties using natural language processing techniques. The proposed method analyzed varied semantic properties of 56 different specifications from five different countries in terms of vocabulary, sentence structure, and the organizing styles of provisions. First, the authors developed a semantic thesaurus for construction terms including 208 word-replacement rules based on Word2Vec embedding to understand the different vocabularies. Second, the authors developed a named entity recognition model based on bi-directional long short-term memory with a conditional random field layer, which identified the required keywords from given provisions with an averaged F1 score of 0.928. Third, the authors developed a provision-pairing model based on Doc2Vec embedding, which identified the most relevant provisions with an average accuracy of 84.4%. The web-based prototype demonstrated that the proposed system can facilitate the construction specification review process by reducing the time spent, supplementing the reviewer’s experience, enhancing accuracy, and achieving consistency. The results contribute to risk management in the construction industry, with practitioners being able to review construction specifications thoroughly in spite of tight schedules and few available experts.     

Intelligent compilation of patent summaries using machine learning and natural language processing techniques

Patents are a type of intellectual property with ownership and monopolistic rights that are publicly accessible published documents, often with illustrations, registered by governments and international organizations. The registration allows people familiar with the domain to understand how to re-create the new and useful invention but restricts the manufacturing unless the owner licenses or enters into a legal agreement to sell ownership of the patent. Patents reward the costly research and development efforts of inventors while spreading new knowledge and accelerating innovation. This research uses artificial intelligence natural language processing, deep learning techniques and machine learning algorithms to extract the essential knowledge of patent documents within a given domain as a means to evaluate their worth and technical advantage. Manual patent abstraction is a time consuming, labor intensive, and subjective process which becomes cost and outcome ineffective as the size of the patent knowledge domain increases. This research develops an intelligent patent summarization methodology using artificial intelligence machine learning approaches to allow patent domains of extremely large sizes to be effectively and objectively summarized, especially for cases where the cost and time requirements of manual summarization is infeasible. The system learns to automatically summarize patent documents with natural language texts for any given technical domain. The machine learning solution identifies technical key terminologies (words, phrases, and sentences) in the context of the semantic relationships among training patents and corresponding summaries as the core of the summarization system. To ensure the high performance of the proposed methodology, ROUGE metrics are used to evaluate precision, recall, accuracy, and consistency of knowledge generated by the summarization system. The Smart machinery technologies domain, under the sub-domains of control intelligence, sensor intelligence and intelligent decision-making provide the case studies for the patent summarization system training. The cases use 1708 training pairs of patents and summaries while testing uses 30 randomly selected patents. The case implementation and verification have shown the summary reports achieve 90% and 84% average precision and recall ratios respectively.     

Comparative experiments using supervised learning and machine translation for multilingual sentiment analysis

Sentiment analysis is the natural language processing task dealing with sentiment detection and classification from texts. In recent years, due to the growth in the quantity and fast spreading of user-generated contents online and the impact such information has on events, people and companies worldwide, this task has been approached in an important body of research in the field. Despite different methods having been proposed for distinct types of text, the research community has concentrated less on developing methods for languages other than English. In the above-mentioned context, the present work studies the possibility to employ machine translation systems and supervised methods to build models able to detect and classify sentiment in languages for which less/no resources are available for this task when compared to English, stressing upon the impact of translation quality on the sentiment classification performance. Our extensive evaluation scenarios show that machine translation systems are approaching a good level of maturity and that they can, in combination to appropriate machine learning algorithms and carefully chosen features, be used to build sentiment analysis systems that can obtain comparable performances to the one obtained for English.     

Human activity recognition using multi-features and multiple kernel learning

This paper presents two sets of features, shape representation and kinematic structure, for human activity recognition using a sequence of RGB-D images. The shape features are extracted using the depth information in the frequency domain via spherical harmonics representation. The other features include the motion of the 3D joint positions (i.e. the end points of the distal limb segments) in the human body. Both sets of features are fused using the Multiple Kernel Learning (MKL) technique at the kernel level for human activity recognition. Our experiments on three publicly available datasets demonstrate that the proposed features are robust for human activity recognition and particularly when there are similarities among the actions.     

Modeling dynamic construction work template from existing scheduling records via sequential machine learning

Today, construction planning and scheduling is almost always performed manually, by experienced practitioners. The knowledge of those individuals is materialized, maintained, and propagated through master schedules and look-ahead plans. While historical project schedules are available, manually mining their embedded knowledge to create generic work templates for future projects or revising look-ahead schedules is very difficult, time-consuming and error-prone. The rigid work templates from prior research are also not scalable to cover the inter and intra-class variability in historical schedule activities. This paper aims at fulfilling these needs via a new method to automatically learn construction knowledge from historical project planning and scheduling records and digitize such knowledge in a flexible and generalizable data schema. Specifically, we present Dynamic Process Templates (DPTs) based on a novel vector representation for construction activities where the sequencing knowledge is modeled with generative Long Short-Term Memory Recurrent Neural Networks (LSTM-RNNs). Our machine learning models are exhaustively tested and validated on a diverse dataset of 32 schedules obtained from real-world projects. The experimental results show our method is capable of learning planning and sequencing knowledge at high accuracy across different projects. The benefits for automated project planning and scheduling, schedule quality control, and automated generation of project look-aheads are discussed in detail.