Data-driven soft sensor development based on deep learning technique

In industrial process control, some product qualities and key variables are always difficult to measure online due to technical or economic limitations. As an effective solution, data-driven soft sensors provide stable and reliable online estimation of these variables based on historical measurements of easy-to-measure process variables. Deep learning, as a novel training strategy for deep neural networks, has recently become a popular data-driven approach in the area of machine learning. In the present study, the deep learning technique is employed to build soft sensors and applied to an industrial case to estimate the heavy diesel 95% cut point of a crude distillation unit (CDU). The comparison of modeling results demonstrates that the deep learning technique is especially suitable for soft sensor modeling because of the following advantages over traditional methods. First, with a complex multi-layer structure, the deep neural network is able to contain richer information and yield improved representation ability compared with traditional data-driven models. Second, deep neural networks are established as latent variable models that help to describe highly correlated process variables. Third, the deep learning is semi-supervised so that all available process data can be utilized. Fourth, the deep learning technique is particularly efficient dealing with massive data in practice.     

A novel deep learning prediction model for concrete dam displacements using interpretable mixed attention mechanism

Dam displacements can effectively reflect its operational status, and thus establishing a reliable displacement prediction model is important for dam health monitoring. The majority of the existing data-driven models, however, focus on static regression relationships, which cannot capture the long-term temporal dependencies and adaptively select the most relevant influencing factors to perform predictions. Moreover, the emerging modeling tools such as machine learning (ML) and deep learning (DL) are mostly black-box models, which makes their physical interpretation challenging and greatly limits their practical engineering applications. To address these issues, this paper proposes an interpretable mixed attention mechanism long short-term memory (MAM-LSTM) model based on an encoder-decoder architecture, which is formulated in two stages. In the encoder stage, a factor attention mechanism is developed to adaptively select the highly influential factors at each time step by referring to the previous hidden state. In the decoder stage, a temporal attention mechanism is introduced to properly extract the key time segments by identifying the relevant hidden states across all the time steps. For interpretation purpose, our emphasis is placed on the quantification and visualization of factor and temporal attention weights. Finally, the effectiveness of the proposed model is verified using monitoring data collected from a real-world dam, where its accuracy is compared to a classical statistical model, conventional ML models, and homogeneous DL models. The comparison demonstrates that the MAM-LSTM model outperforms the other models in most cases. Furthermore, the interpretation of global attention weights confirms the physical rationality of our attention-based model. This work addresses the research gap in interpretable artificial intelligence for dam displacement prediction and delivers a model with both high-accuracy and interpretability.     

Ensemble deep learning based semi-supervised soft sensor modeling method and its application on quality prediction for coal preparation process

Coal preparation is the most effective and economical technique to reduce impurities and improve the product quality for run-of-mine coal. The timely and accurate prediction for key quality characteristics of separated coal plays a significant role in condition monitoring and production control. However, these quality characteristics are usually difficult to directly measure online in industrial practices Although some computation intelligence based soft sensor modeling methods have been developed and reported in existing research for these quality variables estimation, some problems still exist, i.e., manual feature extraction, considerable unlabeled data, temporal dynamic behavior in data, which will influence the accuracy and efficiency for established soft sensor model. To address above-mentioned problem and develop an more excellent quality prediction model for coal preparation process, a novel deep learning based semi-supervised soft sensor modeling approach is proposed which combining the advantage of unsupervised deep learning technique (i.e., Stacked Auto-Encoder (SAE)) with the advantage of supervised deep bidirectional recurrent learner (i.e., Bidirectional Long Short-Term Memory (BLSTM)). More specifically, the unsupervised SAE networks are implemented to learn the representative features hidden in all available input data (labeled and unlabeled samples) and store them as context vector. Then, partial context vector with corresponding labels and the quality variable measure value at previous time are concatenated to form a new merged input feature vector. After that, the temporal and dynamic features are further extracted from the new merged input feature vector via BLSTM networks. Subsequently, the fully connected layers (FCs) are exploited to learn the higher-level features from the last hidden layer of the BLSTM. Lastly, the learned output features by FCs are fed into a supervised liner regression layer to predict the coal quality metrics. Meanwhile, to avoid over-fitting, some regularization techniques are utilized and discussed in proposed network. The application in ash content estimation for a real dense medium coal preparation process and some comparison experiment result demonstrate that the effectiveness and priority of proposed soft sensor modeling approach.     

基于深度学习的无线传感器网络数据融合

在无线传感器网络数据融合算法中,BP神经网络被广泛用于节点数据的特征提取和分类。为了解决BP神经网络收敛慢,易陷入局部最优值且泛化能力差从而影响数据融合效果的问题,提出一种将深度学习技术和分簇协议相结合的数据融合算法SAESMDA。SAESMDA用基于层叠自动编码器(SAE)的深度学习模型SAESM取代BP神经网络,算法首先在汇聚节点训练SAESM并对网络分簇,接着各簇节点通过SAESM对采集数据进行特征提取,之后由簇首将分类融合后的特征发送至汇聚节点。仿真实验表明,和采用BP神经网络的BPNDA算法相比,SAESMDA在网络能耗大致相同的情况下具有更高的特征提取分类正确率。     

一种用于软测量建模的增量学习集成算法

针对软测量模型在实际应用中遇到的问题, 结合AdaBoost 集成学习思想, 提出适用于软测量回归的集成学习算法, 以提高传统软测量模型的精度. 为了克服模型更新技术对软测量实际应用的制约, 将增量学习机制加入软测量集成建模中, 使软测量模型具有在线实时更新的增量学习能力. 对浆纱过程使用新方法建立上浆率软测量模型, 并使用实际生产数据对模型进行检验, 检验结果表明, 该模型具有很好的预测精度, 并能够较好地实现在线更新.

     

Soft sensor based on stacked auto-encoder deep neural network for air preheater rotor deformation prediction

Soft sensors have been widely used in industrial processes over the past two decades because they use easy-to-measure process variables to predict difficult-to-measure ones. Some success has been achieved by the dominant traditional methods of modeling soft sensors based on statistics, such as principal components analysis (PCA) and partial least square (PLS), but such sensors usually become inaccurate and inefficient when processing strong nonlinear data. In this paper, a new soft sensor modeling approach is proposed based on a deep learning network. First, stacked auto-encoders (SAEs) are employed to extract high-level feature representations of the input data. In the process of training each layer of a SAE, the Limited-memory Broyden-Fletcher-Goldfarb-Shanno algorithm (L-BFGS) is adopted to optimize the weights parameters. Then, a support vector regression (SVR) is added to predict the target value on the basis of the features obtained from the SAE. To improve the model performance, Genetic Algorithm (GA) is used to obtain the optimal parameters of the SVR. To evaluate the proposed method, a soft sensor model for estimating the rotor deformation of air preheaters in a thermal power plant boiler is studied. The experimental results demonstrate that the soft sensor based on the SAE-SVR algorithm is more effective than the existing methods are.     

Temporal convolutional network with soft thresholding and attention mechanism for machinery prognostics

Remaining useful life (RUL) prediction is a challenging task for prognostics and health management (PHM). Due to the complexity physics involved for precisely modeling the machine degradation process, learning-based data-driven methods, which learn the degradation pattern solely from the historical data without referring to physical models, have become promising alternatives to model-based prognostic methods. In this paper, a new temporal convolutional neural network (TCN) with soft threshold and attention mechanism is proposed for machinery prognostics. Multi-channel sensor data are directly used as inputs to the prognostic network without feature extraction as a pre-processing step. A soft thresholding mechanism is embedded in the network, serving as a flexible activation function for certain layers to preserve useful features. The threshold value is adaptively learned by a subnetwork trained with the attention mechanism instead of assigning a deterministic value to the threshold. As a result, each feature map is assigned a customized threshold value such that the network training process can focus on features that are more critical to RUL prediction. To verify the generalization ability of the proposed method, three benchmark datasets related to rolling bearings and cutting tools are tested, and the performance of the developed method is compared with several state-of-the-art prognostic approaches. The results show that for all the three case studies, the developed method has produced accurate RUL prediction with good robustness and generalization ability.     

Active learning strategy for smart soft sensor development under a small number of labeled data samples

This contribution proposes a new active learning strategy for smart soft sensor development. The main objective of the smart soft sensor is to opportunely collect labeled data samples in such a way as to minimize the error of the regression process while minimizing the number of labeled samples used, and thus to reduce the costs related to labeling training samples. Instead of randomly labeling data samples, the smart soft sensor only labels those data samples which can provide the most significant information for construction of the soft sensor. In this paper, without loss of generality, the smart soft sensor is built based on the widely used principal component regression model. For performance evaluation, an industrial case study is provided. Compared to the random sample labeling strategy, both accuracy and stability have been improved by the active learning strategy based smart soft sensor.     

面向工业过程软测量建模的概念漂移检测综述

基于数据驱动的软测量模型广泛用于工业过程中产品质量与环保指标等难测参数的在线测量,该过程中存在的概念漂移问题易导致模型精度下降.如何有效识别过程概念变化并精准检测漂移样本是提高模型测量性能的关键.本文总结并分析目前漂移检测的研究思路与进展,为面向工业过程软测量的漂移检测算法提供设计指导.首先,介绍了概念漂移的通常定义与...     

Development and application of soft sensor model for heterogeneous information of aluminum reduction cells

The soft sensor model for heterogeneous information is presented because of the difficulty of online acquiring heterogeneous information of aluminum reduction cells. Firstly many redundancy samples are optimized by Fuzzy C-Means in order to acquire classified samples. Then dynamic process of heterogeneous information of aluminum reduction cells is modeled by multiple neural networks. Finally soft sensor model for heterogeneous information of aluminum reduction cells is developed. The model was used in 320 KA prebaked aluminum reduction cells in Guangxi Branch of China Aluminum Corporation. The results indicate that there are three types of instabilities for aluminum reduction cells: single anode irrationality, parameters irrationality of heat balance and outside operations. Corresponding measures to eliminate the three types of instabilities for aluminum reduction cells are the following: raising the anode, adjusting the parameters of heat balance and improving the operation of changing anode and taping metal.     

Comparison of variable selection methods for PLS-based soft sensor modeling

Data-driven soft sensors have been widely used in both academic research and industrial applications for predicting hard-to-measure variables or replacing physical sensors to reduce cost. It has been shown that the performance of these data-driven soft sensors could be greatly improved by selecting only the vital variables that strongly affect the primary variables, rather than using all the available process variables. In this work, a comprehensive evaluation of different variable selection methods for PLS-based soft sensor development is presented, and a new metric is proposed to assess the performance of different variable selection methods. The following seven variable selection methods are compared: stepwise regression (SR), partial least squares with regression coefficients (PLS-BETA), PLS with variable importance in projection (PLS-VIP), uninformative variable elimination with PLS (UVE-PLS), genetic algorithm with PLS (GA-PLS), least absolute shrinkage and selection operator (Lasso), and competitive adaptive reweighted sampling with PLS (CARS-PLS). Their strengths and limitations for soft sensor development are demonstrated by a simulated case study and an industrial case study.     

Dynamic ensemble wind speed prediction model based on hybrid deep reinforcement learning

Prediction of wind speed can provide a reference for the reliable utilization of wind energy. This study focuses on 1-hour, 1-step ahead deterministic wind speed prediction with only wind speed as input. To consider the time-varying characteristics of wind speed series, a dynamic ensemble wind speed prediction model based on deep reinforcement learning is proposed. It includes ensemble learning, multi-objective optimization, and deep reinforcement learning to ensure effectiveness. In part A, deep echo state network enhanced by real-time wavelet packet decomposition is used to construct base models with different vanishing moments. The variety of vanishing moments naturally guarantees the diversity of base models. In part B, multi-objective optimization is adopted to determine the combination weights of base models. The bias and variance of ensemble model are synchronously minimized to improve generalization ability. In part C, the non-dominated solutions of combination weights are embedded into a deep reinforcement learning environment to achieve dynamic selection. By reasonably designing the reinforcement learning environment, it can dynamically select non-dominated solution in each prediction according to the time-varying characteristics of wind speed. Four actual wind speed series are used to validate the proposed dynamic ensemble model. The results show that: (a) The proposed dynamic ensemble model is competitive for wind speed prediction. It significantly outperforms five classic intelligent prediction models and six ensemble methods; (b) Every part of the proposed model is indispensable to improve the prediction accuracy.     

Uncertainty-aware soft sensor using Bayesian recurrent neural networks

Data-driven soft sensors have been widely used to measure key variables for industrial processes. Soft sensors using deep learning models have attracted considerable attention and shown superior predictive performance. However, if a soft sensor encounters an unexpected situation in inferring data or if noisy input data is used, the estimated value derived by a standard soft sensor using deep learning may at best be untrustworthy. This problem can be mitigated by expressing a degree of uncertainty about the trustworthiness of the estimated value produced by the soft sensor. To address this issue of uncertainty, we propose using an uncertainty-aware soft sensor that uses Bayesian recurrent neural networks (RNNs). The proposed soft sensor uses a RNN model as a backbone and is then trained using Bayesian techniques. The experimental results demonstrated that such an uncertainty-aware soft sensor increases the reliability of predictive uncertainty. In comparisons with a standard soft sensor, it shows a capability to use uncertainties for interval prediction without compromising predictive performance.     

一种基于注意力机制的小目标检测深度学习模型

小目标检测用来识别图像中小像素尺寸目标.传统目标识别算法泛化性差,而通用的深度卷积神经网络算法容易丢失小目标的特征,对小目标识别的效果不甚理想.针对以上问题,提出了一种基于注意力机制的小目标检测深度学习模型AM-R-CNN,该模型在ResNet101主干网络和候选区域生成网络中使用了通道域注意力和空间域注意力,通道域注...     

A deep feature enhanced reinforcement learning method for rolling bearing fault diagnosis

Fault diagnosis of rolling bearing is crucial for safety of large rotating machinery. However, in practical engineering, the fault modes of rolling bearings are usually compound faults and contain a large amount of noise, which increases the difficulty of fault diagnosis. Therefore, a deep feature enhanced reinforcement learning method is proposed for the fault diagnosis of rolling bearing. Firstly, to improve robustness, the neural network is modified by the Elu activation function. Secondly, attention model is used to improve the feature enhanced ability and acquire essential global information. Finally, deep Q network is established to accurately diagnosis the fault modes. Sufficient experiments are conducted on the rolling bearing dataset. Test result shows that the proposed method is superior to other intelligent diagnosis methods.     

深度强化学习中状态注意力机制的研究

虽然在深度学习与强化学习结合后,人工智能在棋类游戏和视频游戏等领域取得了超越人类水平的重大成就,但是实时策略性游戏星际争霸由于其巨大的状态空间和动作空间,对于人工智能研究者来说是一个巨大的挑战平台,针对Deepmind在星际争霸II迷你游戏中利用经典的深度强化学习算法A3C训练出来的基线智能体的水平和普通业余玩家的水平相比还存在较大的差距的问题。通过采用更简化的网络结构以及把注意力机制与强化学习中的奖励结合起来的方法,提出基于状态注意力的A3C算法,所训练出来的智能体在个别星际迷你游戏中利用更少的特征图层取得的成绩最高,高于Deepmind的基线智能体71分。     

基于注意力机制的深度学习推荐研究进展

近年来,注意力机制AM被广泛应用到基于深度学习的自然语言处理任务中,基于注意力机制的深度学习推荐也成为推荐系统研究的一个新方向.探讨了注意力机制的结构和分类标准,从基于注意力机制的DNN推荐、CNN推荐、RNN推荐、GNN推荐4个方面分析了现有融合注意力机制的深度学习推荐研究的主要进展和不足,阐明了其中的主要难点,最后指出了多特征交互的注意力机制推荐、多模态注意力机制深度学习推荐、融入注意力机制的多种深度神经网络混合推荐和注意力机制的群组推荐等基于注意力机制的深度学习推荐未来的主要研究方向.     

Multi-label fault recognition framework using deep reinforcement learning and curriculum learning mechanism

In the actual working site, the equipment often works in different working conditions while the manufacturing system is rather complicated. However, traditional multi-label learning methods need to use the pre-defined label sequence or synchronously predict all labels of the input sample in the fault diagnosis domain. Deep reinforcement learning (DRL) combines the perception ability of deep learning and the decision-making ability of reinforcement learning. Moreover, the curriculum learning mechanism follows the learning approach of humans from easy to complex. Consequently, an improved proximal policy optimization (PPO) method, which is a typical algorithm in DRL, is proposed as a novel method on multi-label classification in this paper. The improved PPO method could build a relationship between several predicted labels of input sample because of designing an action history vector, which encodes all history actions selected by the agent at current time step. In two rolling bearing experiments, the diagnostic results demonstrate that the proposed method provides a higher accuracy than traditional multi-label methods on fault recognition under complicated working conditions. Besides, the proposed method could distinguish the multiple labels of input samples following the curriculum mechanism from easy to complex, compared with the same network using the pre-defined label sequence.