基于时空相似LSTM的空气质量预测模型

由传统机器学习方法组成的空气质量预测模型得到了普遍应用,但是此类模型对于数据有效性,特别是时空相关数据的选取仍旧存在不足.针对深度学习输入数据有效性问题进行研究,提出了一种基于时空相似LSTM的预测模型(spatial-temporal similarity LSTM model,STS-LSTM),以便在时间和空间层面选取更加有效的数据.STS-LSTM分为前序、中序和后序三个模块,前序模块为时空相似选择输入模块,提出了格兰杰因果权重动态时间折叠(Granger causal index weighted dynamic time warping,GCWDTW)算法,用于选取具有更高时空相似性的数据;中序模块使用LSTM作为深度学习网络进行训练;后序模块根据目标站点特征选择不同的输出组合进行集成.STS-LSTM整体模型在空气质量预测误差上较现有算法提升了8％左右,经过有效性选取的数据对于模型精度达到了最高21％的提升.实验结果表明,对于有效数据的选取该算法取得了显著效果,将数据输入输出方法作为应用型深度学习网络的一部分,可以有效提升深度学习网络的最终效果.     

基于LSTM-CNN-CBAM模型的股票预测研究

为了更好地对股票价格进行预测,进而为股民提供合理化的建议,提出了一种在结合长短期记忆网络（LSTM）和卷积神经网络（CNN）的基础上引入注意力机制的股票预测混合模型（LSTM-CNN-CBAM）,该模型采用的是端到端的网络结构,使用LSTM来提取数据中的时序特征,利用CNN挖掘数据中的深层特征,通过在网络结构中加入注意力机制--Convolutional Attention Block Module（CBAM）卷积模块,可以有效地提升网络的特征提取能力。基于上证指数进行对比实验,通过对比实验预测结果和评价指标,验证了在LSTM与CNN结合的网络模型中加入CBAM模块的预测有效性和可行性。     

单时序特征图卷积网络融合预测方法

近年来,图神经网络逐渐成为深度学习领域广泛讨论的话题和研究的重点,但大多数研究都是基于图节点,在存在多维属性的前提下进行分类和回归预测,对单时序特征的图节点预测并不能产生理想的效果。本文提出一种时序图卷积网络算法,可以在复杂图网络中,只根据节点单一特征的时序序列,实现对该特征的预测。算法通过在传统图卷积网络中对邻接矩阵参数化,解决单一特征条件下的参数退化问题,并结合长短时记忆网络的序列学习方法,将时序信息融入到训练过程中,提高训练精度。在交通流量数据集PeMS和Los上的实验表明,其预测精度要优于GCN、T-GCN、GRU、LSTM等主流算法。     

基于改进的深度残差网络的表情识别研究

提出了一种基于改进的深度残差网络（residual network,ResNet）的表情识别算法。采用小卷积核和深网络结构,利用残差模块学习残差映射解决了随着网络深度的增加网络精度下降问题,通过迁移学习方法克服了因数据量不足导致训练不充分的缺点;网络架构使用了线性支持向量机（SVM）进行分类。实验中首先利用ImageNet数据库进行网络参数预训练,使网络具有良好的提取特征能力,根据迁移学习方法,利用FER-2013数据库以及扩充后的CK+数据库进行参数微调和训练。该算法克服了浅层网络需要依靠手工特征,深层网络难以训练等问题,在CK+数据库以及GENKI-4K数据库上分别取得了91.333%和95.775%识别率。SVM在CK+数据库的分类效果较softmax提高了1%左右。     

Operational time-series data modeling via LSTM network integrating principal component analysis based on human experience

Today’s information technologies involve increasingly intelligent systems, which come at the cost of increasingly complex equipment. Modern monitoring systems collect multi-measuring-point and long-term data which make equipment health prediction a “big data” problem. It is difficult to extract information from such condition monitoring data to accurately estimate or predict health statuses. Deep learning is a powerful tool for big data processing that is widely utilized in image and speech recognition applications, and can also provide effective predictions in industrial processes. This paper proposes the Long Short-term Memory Integrating Principal Component Analysis based on Human Experience (HEPCA-LSTM), which uses operational time-series data for equipment health prognostics. Principal component analysis based on human experience is first conducted to extract condition parameters from the condition monitoring system. The long short-term memory (LSTM) framework is then constructed to predict the target status. Finally, a dynamic update of the prediction model with incoming data is performed at a certain interval to prevent any model misalignment caused by the drifting of relevant variables. The proposed model is validated on a practical case and found to outperform other prediction methods. It utilizes a powerful deep learning analysis method, the LSTM, to fully process big condition monitoring series data; it effectively extracts the features involved with human experience and takes dynamic updates into consideration.     

短文本分类的ResLCNN模型

短文本分类是互联网文本数据处理中的关键任务之一.长短时记忆网络LSTM（long short-term memory）和卷积神经网络CNN（convolutional neural network）是广泛应用于短文本分类任务的两种深度学习模型.在计算机视觉和语音识别领域的深度学习研究表明,深层次的神经网络模型具有较好的表达数据特征的能力.受此启发,面向文本深度学习分类问题,提出基于3层LSTM和CNN网络结构的ResLCNN（residual-LSTM-CNN）深度学习模型.该模型有效结合LSTM获取文本序列数据的长距离依赖特征和CNN通过卷积操作获取句子局部特征的优势,同时借鉴残差模型理论,在第1层LSTM层与CNN层之间加入恒等映射,构建残差层,缓解深层模型梯度消失问题.为了探究深层短文本分类中ResLCNN模型的文本分类能力,在多种数据集上将其与LSTM、CNN及其组合模型进行对比实验.结果表明,相比于单层LSTM与CNN组合模型,ResLCNN深层模型在MR、SST-2和SST-5数据集上分别提高了1.0%、0.5%、0.47%的准确率,取得了更好的分类效果.     

基于条件生成对抗网络的深度点过程二次预测

结合深度神经网络和时序点过程的深度点过程模型在进行时间预测时,会因模型本身系统误差和数值计算精度不足而导致预测值序列中存在较大偏差。为提高预测精度并有效避免模型调优同时降低数值误差,建立一种基于条件生成对抗网络（CGAN）的深度点过程二次预测模型,在深度点过程初次预测值序列的基础上进行二次预测。假设初次预测偏差来自时序点过程分布上的差异,利用CGAN对分布的变换能力来修正初次预测值序列分布为原始时序点过程序列分布,从而降低预测值序列误差。在流程上,将初次预测值序列输入生成器生成伪值序列,将伪值序列与对应的真实值序列输入判别器中判别真假,经过对抗训练得到对初次预测值序列具备修正能力的生成器。同时,为增强CGAN对时序点过程数据的匹配度,在其结构上采用CGAN+LSTM的形式,同时改进损失函数为时序点过程Wasserstein距离的对偶形式及其1-Lipschitz约束。实验结果表明,该模型具有较高的时间预测准确度,二次预测值序列的均方误差相较初次预测值序列平均降低77%以上。     

FSO-LSTM IDS: hybrid optimized and ensembled deep-learning network-based intrusion detection system for smart networks

The Internet of Things (IoT) has achieved exponential growth worldwide. Although the IoT is used by millions of users, these networks are handicapped by attacks such as denial of service, man in the middle, and spoofing. These attacks threaten the entire IoT ecosystem and affect the integrity and security of the user. Hence, the prediction and identification of novel network attacks in an IoT network remains a challenge for researchers. Recently, machine learning and deep learning have played a pivotal role in predicting and classifying different attacks in an IoT network. However, these algorithms suffer from computational complexity as the number of attacks increases. Hence, a novel hybrid optimized long short-term memory (LSTM) approach is proposed. Whereas a convolutional neural network is used to extract the temporal and spatial correlated features of the IoT network, the optimized LSTM is used to predict the different attacks in the network. Furthermore, firefly swarm optimization is integrated with LSTM to reduce the computational overhead, which in turn increases the prediction accuracy. Nearly 19,00,503 real-time normal and attack data were collected from the experimental simulation setup based on the OMNET++–Python–IoT framework. Extensive experimentation was carried out to evaluate the proposed algorithm, and various metrics, such as accuracy, sensitivity, specificity, and F1-score, were calculated and compared with state-of-the-art learning-based network intrusion detection systems. Furthermore, other benchmarks, such as the CIDCC-15, UNSW-NB15, and NSL-KDD datasets, were used to evaluate the performance of the different deep-learning-based intrusion detection systems. The results demonstrate that the proposed deep-learning method outperforms other classical learning models with low complexity and high prediction performance.

     

面向预测的长短时神经网络记忆增强机制

基于时序数据建模的长短时神经网络（LSTM）可用于预测类问题。现实场景中,LSTM预测精度往往与输入序列长度相关,有效的历史信息会被新输入的数据淹没。针对此问题,提出在LSTM节点中构建强化门实现对遗忘信息的提取,并与记忆信息按比例选取、融合、输入记忆单元,增加学习过程中的梯度传导能力,使网络对相对较远的信息保持敏感以提升记忆能力。实验采用工业故障数据,当序列长度超过100时,具有强化门机制的改进模型预测误差低于其他LSTM模型。预测精度的差距随序列增加而增大,当序列长度增至200时,改进模型的预测误差（RMSE/MAE）较原模型分别降低了26.98%与35.85%。     

Stock-Price Forecasting Based on XGBoost and LSTM

Using time-series data analysis for stock-price forecasting (SPF) is complex and challenging because many factors can influence stock prices (e.g., inflation, seasonality, economic policy, societal behaviors). Such factors can be analyzed over time for SPF. Machine learning and deep learning have been shown to obtain better forecasts of stock prices than traditional approaches. This study, therefore, proposed a method to enhance the performance of an SPF system based on advanced machine learning and deep learning approaches. First, we applied extreme gradient boosting as a feature-selection technique to extract important features from high-dimensional time-series data and remove redundant features. Then, we fed selected features into a deep long short-term memory (LSTM) network to forecast stock prices. The deep LSTM network was used to reflect the temporal nature of the input time series and fully exploit future contextual information. The complex structure enables this network to capture more stochasticity within the stock price. The method does not change when applied to stock data or Forex data. Experimental results based on a Forex dataset covering 2008–2018 showed that our approach outperformed the baseline autoregressive integrated moving average approach with regard to mean absolute error, mean squared error, and root-mean-square error.     

基于最小类内方差的伪三维残差网络

作为一种提取视频时空特征的深度学习方法,伪三维残差网络(pseudo-3D residual net,P3D ResNet)利用SVM目标函数来驱动深度网络学习,这样该方法继承了SVM的不足——仅考虑了不同类别间的间隔,忽略了同类样本数据的分布信息.针对该问题,提出了基于最小类内方差的伪三维残差网络方法,不仅体现了大间隔原理,同时又利用了样本数据的分布信息.该方法首先使用P3D ResNet提取的特征向量计算类内散度矩阵;然后利用该矩阵构建了新的目标函数;最后通过新构建的目标函数来驱动P3D ResNet的学习.将该方法应用到行为识别领域,多个数据集上的实验结果表明,相比于传统的P3D ResNet,所提出的方法获得了更高的识别准确率,体现出了更好的泛化性能.     

Data stream forecasting for system fault prediction

Competition among today’s industrial companies is very high. Therefore, system availability plays an important role and is a critical point for most companies. Detecting failures at an early stage or foreseeing them before they occur is crucial for machinery availability. Data analysis is the most common method for machine health condition monitoring. In this paper we propose a fault-detection system based on data stream prediction, data stream mining, and data stream management system (DSMS). Companies that are able to predict and avoid the occurrence of failures have an advantage over their competitors. The literature has shown that data prediction can also reduce the consumption of communication resources in distributed data stream processing.     

Long-time gap crowd prediction using time series deep learning models with two-dimensional single attribute inputs

Crowd prediction is a crucial aspect of modern life with innumerable applications. By predicting future human occupancy in advance, crowd prediction can support the decision-making processes of facility stakeholders, e.g., the campus operator can schedule facility maintenance during the period of lowest pedestrian flow to eliminate any disturbance. Conventional crowd prediction utilizes statistical models and rule-based data mining techniques, which are tedious in data processing and error-prone. Hence, this study formulates crowd prediction into a time-series analysis based on deep learning. Despite its wide adaptability in various research fields, deep learning-based time series analysis is seldom adopted in crowd prediction. There are two major limitations in previous studies: firstly, the prediction accuracy notably degrades with increased prediction length, and secondly only the temporal pattern along a single time dimension is exploited, i.e., the consecutive time steps in the most recent input data. Therefore, a Long-Time Gap Two-Dimensional method, entitled LT2D-method, is proposed to increase the crowd prediction length of with high accuracy. The LT2D-method is composed of two parts, (1) long-time gap prediction, which extends the prediction length to 240 time steps (1 day) with high accuracy, and (2) 2D inputs method, which exploits the prior knowledge from different time dimensions to further improve the prediction accuracy of long-time gap prediction. The proposed LT2D-method can be generally adapted to deep learning models, such as LSTM, BiLSTM, and GRU, to improve the prediction accuracy. By incorporating the proposed LT2D-method into different baseline models, the accuracy is generally improved by around 22%, demonstrating the robustness and generalizability of our method.     

基于改进的深度Q网络结构的商品推荐模型

传统推荐方法存在数据稀疏和特征识别差等问题,为了解决这些问题,根据隐式反馈构建具有时序性的正负反馈数据集。由于正负反馈数据集和商品购买具有强时序性特征,引入长短期记忆（LSTM）网络作为模型构件。考虑用户自身特征和用户动作选择回报由不同的输入数据决定,对竞争架构的深度Q网络进行改进,融合用户正负反馈和商品购买时序性,设计了基于改进的深度Q网络结构的商品推荐模型。模型对正负反馈数据进行区分性训练,对商品购买的时序性特征进行提取。在Retailrocket数据集上,与因子分解机（FM）模型、W&D模型和协同过滤（CF）模型中表现最好的相比,所提模型的准确率、召回率、平均准确率（MAP）和归一化折损累计增益（NDCG）分别提高了158.42%、89.81%、95.00%和67.57%。同时,使用DBGD作为探索方法,改善了推荐商品多样性低的缺陷。     

基于改进的深度Q网络结构的商品推荐模型

传统推荐方法存在数据稀疏和特征识别差等问题，为了解决这些问题，根据隐式反馈构建具有时序性的正负反馈数据集。由于正负反馈数据集和商品购买具有强时序性特征，引入长短期记忆（LSTM）网络作为模型构件。考虑用户自身特征和用户动作选择回报由不同的输入数据决定，对竞争架构的深度Q网络进行改进，融合用户正负反馈和商品购买时序性，设计了基于改进的深度Q网络结构的商品推荐模型。模型对正负反馈数据进行区分性训练，对商品购买的时序性特征进行提取。在Retailrocket数据集上，与因子分解机（FM）模型、W&D模型和协同过滤（CF）模型中表现最好的相比，所提模型的准确率、召回率、平均准确率（MAP）和归一化折损累计增益（NDCG）分别提高了158.42%、89.81%、95.00%和67.57%。同时，使用DBGD作为探索方法，改善了推荐商品多样性低的缺陷。     

基于双向特征金字塔和残差网络的危化品运输车辆检测

危化品运输车辆的主要特征是车顶的危险标志和车牌下的危险品标志,这对于大多数目标检测算法来说检测起来比较困难.为了在提高检测精度的同时加快检测速度,本文提出了一种融合残差网络和双向特征金字塔网络的危化品车辆检测算法.首先通过对高速公路监控视频进行截取,制作危化品车辆数据集,然后通过残差网络进行特征提取,在本文中,使用循环...     

考虑换道意图的LSTM-AdaBoost车辆轨迹预测模型

不合理的车辆的换道行为是导致交通事故发生的主要原因之一,提前预知换道车辆的轨迹并及时做出相应调整有助于减少事故的发生。针对换道车辆轨迹预测问题,采用将深度学习和集成学习相结合的轨迹预测方法,并考虑了换道意图的影响。建立连续隐马尔可夫模型对车辆进行换道意图检测,提前判别车辆的换道状态,并输入至相应的轨迹预测模型中;将LSTM（long short term memory）作为AdaBoost算法（adaptive boosting）的基预测器,建立LSTM-AdaBoost模型,在多个基预测器同时进行轨迹预测的基础上,通过训练调整各个基预测器的权重并将结果加权集成,提升预测模型的精度和稳定性;通过NSGIM（next generation simulation）数据集对模型进行训练和测试,结果显示意图预测模型在变道前一秒的准确率在90%以上,LSTM-AdaBoost集成轨迹预测模型与单一的LSTM模型相比精度和稳定性显著提升,且预测结果中异常数据更少,具有较好的稳定性;同时预测对比结果也表明增加意图预测模块有助于提升换道轨迹预测的精度。     

A semi-supervised learning detection method for vision-based monitoring of construction sites by integrating teacher-student networks and data augmentation

Recently, deep-learning detection methods have achieved huge success in the vision-based monitoring of construction sites in terms of safety control and productivity analysis. However, deep-learning detection methods require large-scale datasets for training purposes, and such datasets are difficult to develop due to the limited accessibility of construction images and the need for labor-intensive annotations. To address this problem, this research proposes a semi-supervised learning detection method for construction site monitoring based on teacher–student networks and data augmentation. The proposed method requires a limited number of labeled data to achieve high detection performance in construction scenarios. Initially, the proposed method trains the teacher object detector with labeled data following weak data augmentation. Next, the trained teacher object detector generates pseudo-detection results from unlabeled images that have been weakly augmented. Finally, the student object detector is trained with the pseudo-detection results and unlabeled images that have been both weakly and strongly augmented. In our experiments, 10,000 annotated construction images from the Alberta Construction Image Dataset (ACID) have been divided into a training set (70%) and a validation set (30%). The proposed method achieved a 91% mean average precision (mAP) on the validation set while only requiring 30% of the training set. In comparison, the existing supervised learning method ResNet50 Faster R-CNN achieved a mAP of 90.8% when training on the full training set. These experimental results show the potential of the proposed method in terms of reducing the time, effort, and costs spent on developing construction datasets. As such, this research has explored the potential of semi-supervised learning methods and increased the practicality of vision-based monitoring systems in the construction industry.     

医学影像疾病诊断的残差神经网络优化算法研究进展

残差神经网络（residual neural network,ResNet）及其优化是深度学习研究的热点之一,在医学图像领域应用广泛,在肿瘤、心脑血管和神经系统疾病等重大疾病的临床诊断、分期、转移、治疗决策和靶区勾画方面取得良好效果。本文对残差神经网络的学习优化进行了总结：阐述了残差神经网络学习算法优化,从激活函数、损失函数、参数优化算法、学习衰减率、归一化和正则化技术等6方面进行总结,其中激活函数的改进方法主要有Sigmoid、tanh、ReLU、PReLU（parameteric ReLU）、随机化ReLU（randomized leaky ReLU,RReLU）、ELU（exponential linear units）、Softplus函数、NoisySoftplus函数以及Maxout共9种;损失函数主要有交叉熵损失、均方损失、欧氏距离损失、对比损失、合页损失、Softmax-Loss、L-Softmax Loss、A-Softmax Loss、L2 Softmax Loss、Cosine Loss、Center Loss和焦点损失共12种;学习率衰减总结了8种,即分段常数衰减、多项式衰减、指数衰减、反时限衰减、自然指数衰减、余弦衰减、线性余弦衰减和噪声线性余弦衰减;归一化算法有批量归一化和提出批量重归一化算法;正则化方法主要有增加输入数据、数据增强、早停法、L1正则化、L2正则化、Dropout和Dropout Connect共7种。综述了残差网络模型在医学图像疾病诊断中的应用研究,梳理了残差神经网络在肺部肿瘤、皮肤疾病、乳腺癌、大脑疾病、糖尿病和血液病等6种疾病诊断中的应用研究;对深度学习在医学图像未来发展进行了总结和展望。     

ResNet结合BiGRU的关系抽取混合模型

关系抽取主要目的是将非结构化或半结构化描述的自然语言文本转化成结构化数据,其主要负责从文本中识别出实体,抽取实体间的语义关系。就关系抽取任务而言,当前流行的网络结构是仅使用CNN作为编码器,经过多层卷积操作后,对池化的结果进行softmax分类。还有部分工作则使用RNN并结合Attention机制对最后的结果做分类。这些网络结构在远程监督带噪声的关系抽取任务中表现并不理想。该文主要根据ResNet残差块的特性,提出了一种混合模型,它有效融合,ResNet和BiGRU,将带有残差特性的CNN和双向RNN结合起来,最后融入注意力机制来完成基于远程监督的关系抽取任务。实验验证了该混合模型在远程监督的噪声过滤方面的有效性。在NYT-Freebase 数据集上,P@N值相比使用单一ResNet提高了2.9%。另外,该文所建混合模型可以很轻易地移植应用到其他NLP任务中。