非饱和区扩展的RNN算法优化

针对长短时记忆网络(LSTM)型循环神经网络(RNN)收敛速度慢,提出了扩展激活函数非饱和区的RNN算法优化.针对LSTM型RNN的训练过程收敛速度慢的原因以及激活函数的性质,提出了加快RNN训练过程收敛的解决方法.通过字符级语言模型对优化方法进行了验证,结果表明:非饱和区扩展的RNN算法优化有效地加快了RNN训练过程的收敛.     

An ANN based approach to recognize initial phonemes of spoken words of Assamese language

Initial phoneme is used in spoken word recognition models. These are used to activate words starting with that phoneme in spoken word recognition models. Such investigations are critical for classification of initial phoneme into a phonetic group. A work is described in this paper using an artificial neural network (ANN) based approach to recognize initial consonant phonemes of Assamese words. A self organizing map (SOM) based algorithm is developed to segment the initial phonemes from its word counterpart. Using a combination of three types of ANN structures, namely recurrent neural network (RNN), SOM and probabilistic neural network (PNN), the proposed algorithm proves its superiority over the conventional discrete wavelet transform (DWT) based phoneme segmentation. The algorithm is exclusively designed on the basis of Assamese phonemical structure which consists of certain unique features and are grouped into six distinct phoneme families. Before applying the segmentation approach using SOM, an RNN is used to take some localized decision to classify the words into six phoneme families. Next the SOM segmented phonemes are classified into individual phonemes. A two-class PNN classification is performed with clean Assamese phonemes, to recognize the segmented phonemes. The validation of recognized phonemes is checked by matching the first formant frequency of the phoneme. Formant frequency of Assamese phonemes, estimated using the pole or formant location determination from the linear prediction model of vocal tract, is used effectively as a priori knowledge in the proposed algorithm.     

非线性时滞系统的稳定自适应控制

针对模型不确定性的连续时间时滞系统，提出了一种新的神经网络自适应控制。系统的辨识模型是由神经网络和系统的已知信息组合构成，在此基础上，建立时滞系统的预测模型。基于神经网络预测模型的自适应控制器能够实现期望轨线的跟踪，理论上证明了闭环系统的稳定性。连续搅拌釜式反应器仿真结果表明了该控制方案的有效性。     

Automated time domain modeling of linear and nonlinear microwave circuits using recurrent neural networks

In this article, a recurrent neural network (RNN) method is employed for dynamic time‐domain modeling of both linear and nonlinear microwave circuits. An automated RNN modeling technique is proposed to efficiently determine the training waveform distribution and internal RNN structure during the offline training process. This technique extends a recent automatic model generation (AMG) algorithm from frequency‐domain model generation to dynamic time‐domain model generation. Two types of applications of the algorithm are presented, transient electromagnetic (EM) behavior modeling of microwave structures, and time‐domain envelope modeling of power amplifiers (PA). For transient EM modeling, we consider EM structures with varying material and geometrical parameters. AMG automatically varies the EM structural parameters during training and drives time‐domain EM simulators to generate necessary amount of data for RNN to learn. AMG aims to model the transient behavior with minimum RNN order while satisfying accuracy requirements. In modeling PA behavior, an envelope formulation is used to specifically learn the AM/AM and AM/PM distortions due to third‐generation (3G) digital modulation input. The RNN PA model is able to model these time domain distortions after training and can accurately model the amplifier behavior in both time (AM/AM, AM/PM) and frequency (spectral re‐growth). © 2008 Wiley Periodicals, Inc. Int J RF and Microwave CAE, 2008.     

自注意力时序点过程生成模型的Wasserstein学习方法

目前学界普遍通过循环神经网络(RNN)建模强度函数来刻画时序点过程,然而此类模型不能捕捉到事件序列之间的长程依赖关系,并且强度函数具体的参数形式会限制模型的泛化能力。针对上述问题,提出一种无强度函数的注意力机制的时序点过程生成模型。该模型使用Wasserstein距离构建损失函数,便于衡量模型分布与真实分布之间的偏差,利用自注意力机制描述历史事件对当前事件的影响程度,使得模型具有可解释性且泛化能力更强。对比实验表明,在缺失强度函数先验信息的情况下,该方法比RNN类的生成模型和极大似然模型在QQ图斜率的偏差和经验强度偏差这两个指标总体上分别减少35.125%和24.200%,证实了所提模型的有效性。     

进化回归神经网络对时序数据和关联数据的建模能力研究

重点研究进化回归神经网络对时序数据和关联数据的建模能力。针对两个标准问题，采用不同形式的建模数据，比较了前向网络和回归神经网络的建模及预测效果，进一步将进化算法用于不同结构回归神经网络的训练并比较了它们的建模能力。仿真结果表明回归神经网络对时序关联数据有很好的建模和预测能力，相比于前向网络，无需过程时序特点的先验知识，可以采用最简单的建模数据形式。而进化算法相比于常规的梯度下降算法，用于训练不同的回归网络结构通用性好，且训练过程不受局部极小问题的困扰，适当规模的训练过程可以获得性能良好的神经网络模型。     

基于单向Transformer和孪生网络的多轮任务型对话技术

循环神经网络和Transformer在多轮对话系统的建模上依赖大量的样本数据且回复准确率过低。为此,提出一种针对任务型对话系统的建模方法。引入预训练模型对句子语意和对话过程进行深度编码,对Transformer模型进行精简,仅保留编码器部分的单向Transformer,将应答部分抽象成不同的指令,采用孪生网络对指令进行相似度排序,选择相似度最高的指令生成应答。在MultiWOZ数据集上的实验结果表明,与LSTM和基于Transformer模型相比,该方法预测速度更快,在小数据集上具有更好的性能,在大数据集上也能取得与当前先进模型相当的效果。     

Physics guided neural network for machining tool wear prediction

Tool wear prediction is of significance to improve the safety and reliability of machining tools, given their widespread applications in nearly every branch of manufacturing. Mathematical modelling, including data driven modelling and physics-based modelling, is an important tool to predict the degree of tool wear. Howerver, the performance of conventional data driven models is restricted by the absent representation of physical inconsistency. The physics-based models usually fail to consider the complex tool cutting conditions and dynamic changes of physical parameters in practice. To address these issues, a novel physics guided neural network model is presented for tool wear prediction. Firstly, a cross physics-data fusion (CPDF) scheme is proposed as the modelling strategy to fuse the hidden information explored by a physics-based model and a data driven model. Secondly, the information hidden in the unlabelled sample is explored by the physics-based model of tool cutting, inspired by semi-supervised learning. Thirdly, a novel loss function which takes the physical discipline into account is proposed to evaluate the physical inconsistency quantitatively. The advantage of the developed method is that it explores sufficient information from both physics and data domains to eliminate the physical inconsistency existing in conventional data driven models.     

Fast character modeling with sketch-based PDE surfaces

Virtual characters are 3D geometric models of characters. They have a lot of applications in multimedia. In this paper, we propose a new physics-based deformation method and efficient character modelling framework for creation of detailed 3D virtual character models. Our proposed physics-based deformation method uses PDE surfaces. Here PDE is the abbreviation of Partial Differential Equation, and PDE surfaces are defined as sculpting force-driven shape representations of interpolation surfaces. Interpolation surfaces are obtained by interpolating key cross-section profile curves and the sculpting force-driven shape representation uses an analytical solution to a vector-valued partial differential equation involving sculpting forces to quickly obtain deformed shapes. Our proposed character modelling framework consists of global modeling and local modeling. The global modeling is also called model building, which is a process of creating a whole character model quickly with sketch-guided and template-based modeling techniques. The local modeling produces local details efficiently to improve the realism of the created character model with four shape manipulation techniques. The sketch-guided global modeling generates a character model from three different levels of sketched profile curves called primary, secondary and key cross-section curves in three orthographic views. The template-based global modeling obtains a new character model by deforming a template model to match the three different levels of profile curves. Four shape manipulation techniques for local modeling are investigated and integrated into the new modelling framework. They include: partial differential equation-based shape manipulation, generalized elliptic curve-driven shape manipulation, sketch assisted shape manipulation, and template-based shape manipulation. These new local modeling techniques have both global and local shape control functions and are efficient in local shape manipulation. The final character models are represented with a collection of surfaces, which are modeled with two types of geometric entities: generalized elliptic curves (GECs) and partial differential equation-based surfaces. Our experiments indicate that the proposed modeling approach can build detailed and realistic character models easily and quickly.

     

TCN-KT：个人基础与遗忘融合的时间卷积知识追踪模型

智慧教育的热门领域知识追踪（KT）被视为序列建模任务,其主要关注点和解决方式都集中在循环神经网络（RNN）上。但RNN通常会面临梯度消失或者梯度爆炸等问题,且训练时间和设备要求都过于严苛,针对以上问题,提出融合学习者个人先验基础和遗忘因素的时间卷积知识追踪模型（TCN-KT）。首先利用RNN模型计算得到学生个人先验基础,然后使用梯度稳定、内存占用率更低的时间卷积网络（TCN）预测学生下一题正误的初始概率,最后融合基于学生基础的遗忘因素得到最终结果。实验验证,TCN-KT预测性能最佳并减少了计算时间。     

A comparative study of neural network and Box-Jenkins ARIMA modeling in time series prediction

This paper aims to investigate suitable time series models for repairable system failure analysis. A comparative study of the Box-Jenkins autoregressive integrated moving average (ARIMA) models and the artificial neural network models in predicting failures are carried out. The neural network architectures evaluated are the multi-layer feed-forward network and the recurrent network. Simulation results on a set of compressor failures showed that in modeling the stochastic nature of reliability data, both the ARIMA and the recurrent neural network (RNN) models outperform the feed-forward model; in terms of lower predictive errors and higher percentage of correct reversal detection. However, both models perform better with short term forecasting. The effect of varying the damped feedback weights in the recurrent net is also investigated and it was found that RNN at the optimal weighting factor gives satisfactory performances compared to the ARIMA model.     

Extracting finite-state representations from recurrent neuralnetworks trained on chaotic symbolic sequences

Concerns neural-based modeling of symbolic chaotic time series. We investigate the knowledge induction process associated with training recurrent mural nets (RNN) on single long chaotic symbolic sequences. Even though training RNN to predict the next symbol leaves the standard performance measures such as the mean square error on the network output virtually unchanged, the nets extract a lot of knowledge. We monitor the knowledge extraction process by considering the nets stochastic sources and letting them generate sequences which are then confronted with the training sequence via information theoretic entropy and cross-entropy measures. We also study the possibility of reformulating the knowledge gained by RNN in a compact easy-to-analyze form of finite-state stochastic machines. The experiments are performed on two sequences with different complexities measured by the size and state transition structure of the induced Crutchfield's epsilon-machines (1991, 1994). The extracted machines can achieve comparable or even better entropy and cross-entropy performance. They reflect the training sequence complexity in their dynamical state representations that can be reformulated using finite-state means. The findings are confirmed by a much more detailed analysis of model generated sequences. We also introduce a visual representation of allowed block structure in the studied sequences that allows for an illustrative insight into both RNN training and finite-state stochastic machine extraction processes.     

Robust Recurrent Neural Network Control of Biped Robot

In this paper, a recurrent neural network (RNN) control scheme is proposed for a biped robot trajectory tracking system. An adaptive online training algorithm is optimized to improve the transient response of the network via so-called conic sector theorem. Furthermore, L ₂-stability of weight estimation error of RNN is guaranteed such that the robustness of the controller is ensured in the presence of uncertainties. In consideration of practical applications, the algorithm is developed in the discrete-time domain. Simulations for a seven-link robot model are presented to justify the advantage of the proposed approach. We give comparisons between the standard PD control and the proposed RNN compensation method.     

Recurrent neuro-fuzzy networks for nonlinear process modeling

A type of recurrent neuro-fuzzy network is proposed in this paper to build long-term prediction models for nonlinear processes. The process operation is partitioned into several fuzzy operating regions. Within each region, a local linear model is used to model the process. The global model output is obtained through the centre of gravity defuzzification which is essentially the interpolation of local model outputs. This modeling strategy utilizes both process knowledge and process input/output data. Process knowledge is used to initially divide the process operation into several fuzzy operating regions and to set up the initial fuzzification layer weights. Process I/O data are used to train the network. Network weights are such trained so that the long-term prediction errors are minimized. Through training, membership functions of fuzzy operating regions are refined and local models are learnt. Based on the recurrent neuro-fuzzy network model, a novel type of nonlinear model-based long range predictive controller can be developed and it consists of several local linear model-based predictive controllers. Local controllers are constructed based on the corresponding local linear models and their outputs are combined to form a global control action by using their membership functions. This control strategy has the advantage that control actions can be calculated analytically avoiding the time consuming nonlinear programming procedures required in conventional nonlinear model-based predictive control. The techniques have been successfully applied to the modeling and control of a neutralization process.     

Static, Dynamic, and Hybrid Neural Networks in Forecasting Inflation

The back-propagation neural network (BPN) model has been the most popular form of artificial neural network model used for forecasting, particularly in economics and finance. It is a static (feed-forward) model which has a learning process in both hidden and output layers. In this paper we compare the performance of the BPN model with that of two other neural network models, viz., the radial basis function network (RBFN) model and the recurrent neural network (RNN) model, in the context of forecasting inflation. The RBFN model is a hybrid model with a learning process that is much faster than the BPN model and that is able to generate almost the same results as the BPN model. The RNN model is a dynamic model which allows feedback from other layers to the input layer, enabling it to capture the dynamic behavior of the series. The results of the ANN models are also compared with those of the econometric time series models.     

基于隐式描述符的三维模型对应关系计算

三维模型对应关系计算在自动驾驶、虚拟现实、智能交通等领域得到广泛关注与应用。三维模型在几何结构和尺度发生很大变化时，低层次几何信息描述符所提取的特征将不足，从而使得对应关系计算结果准确率不高。为此，提出一种通过引入先验知识来完成三维模型对应关系计算的方法。利用深度学习网络模仿人类计算先验知识，以对模型各部分之间的几何相似性进行编码，解决模型在各部分发生显著变化时无法应用低层次几何信息计算模型间对应关系的问题。使用多视图卷积神经网络对模型各部分相应的视图进行预分割并标记，根据模型对应表面点之间的相似度隐式地计算数据驱动描述符，在数据驱动描述符的指导下计算最终的三维模型对应关系。实验结果表明，相较基于先验知识的计算方法，该方法能提高三维模型对应关系计算结果的准确率，且可有效降低测地误差。     

Gray-box modeling and control of polymer molecular weight distribution using orthogonal polynomial neural networks

A method of modeling and control on polymer molecular weight distribution (MWD) is presented in this paper. An orthogonal polynomial feedforward neural network (OPFNN) and a recurrent neural network (RNN) are combined to model the shape of MWD. In this combined neural networks, the weights of OPFNN are equivalent to moments of MWD through a linear transformation, when the polynomial used as the basis function of OPFNN satisfies some requirements. So the weights are given practical feature, and terms the neural network model a gray-box model. Then the requirements of polynomial are deduced. After modeling, an optimal control scheme is discussed on tracking the desired MWD during the polymerization process. The modeling error is added into the performance function to improve the control effect. The modeling and control method is tested on styrene polymerization reacted in CSTR, and simulation results proved the effectiveness of the method.     

Artificial intelligence-enhanced seismic response prediction of reinforced concrete frames

Existing physics-based modeling approaches do not have a good compromise between performance and computational efficiency in predicting the seismic response of reinforced concrete (RC) frames, where high-fidelity models (e.g., fiber-based modeling method) have reasonable predictive performance but are computationally demanding, while more simplified models (e.g., shear building model) are the opposite. This paper proposes a novel artificial intelligence (AI)-enhanced computational method for seismic response prediction of RC frames which can remedy these problems. The proposed AI-enhanced method incorporates an AI technique with a shear building model, where the AI technique can directly utilize the real-world experimental data of RC columns to determine the lateral stiffness of each column in the target RC frame while the structural stiffness matrix is efficiently formulated via the shear building model. Therefore, this scheme can enhance prediction accuracy due to the use of real-world data while maintaining high computational efficiency due to the incorporation of the shear building model. Two data-driven seismic response solvers are developed to implement the proposed approach based on a database including 272 RC column specimens. Numerical results demonstrate that compared to the experimental data, the proposed method outperforms the fiber-based modeling approach in both prediction capability and computational efficiency and is a promising tool for accurate and efficient seismic response prediction of structural systems.     

Application of a neural network to the generation of a robot arm trajectory

We propose a neural network model generating a robot arm trajectory. The developed neural network model is based on a recurrent-type neural network (RNN) model calculating the proper arm trajectory based on data acquired by evaluation functions of human operations as the training data. A self-learning function has been added to the RNN model. The proposed method is applied to a 2-DOF robot arm, and laboratory experiments were executed to show the effectiveness of the proposed method. Through experiments, it is verified that the proposed model can reproduce the arm trajectory generated by a human. Further, the trajectory of a robot arm is successfully modified to avoid collisions with obstacles by a self-learning function.This work was presented, in part, at the 9th International Symposium on Artificial Life and Robotics, Oita, Japan, January 28–30, 2004     

基于遗传算法改进的一阶滞后滤波和长短期记忆网络的蓝藻水华预测方法

河湖藻类水华形成过程中所具有的突发性和不确定性,导致对藻类水华爆发预测准确性不高。为解决此问题,以叶绿素a的浓度值作为蓝藻水华演化过程表征指标,提出基于长短期记忆（LSTM）循环神经网络（RNN）蓝藻水华预测模型。首先,用遗传算法改进的一阶滞后滤波（GF）优化算法对数据进行平滑滤波处理;然后,搭建GF-LSTM网络的蓝藻水华预测模型,实现对水华发生的精准预测;最后,以太湖水域梅梁湖区域的采样数据为样本,对预测模型进行检验,并与传统的RNN和LSTM网络进行对比。仿真结果表明,提出的GF-LSTM网络模型平均相对误差控制在16%~18%,而RNN模型的预测平均相对误差为28%~32%,LSTM网络模型的平均相对误差为19%~22%,对采用数据的平滑性处理效果较好,预测精度更高,对样本具有更好的适应性,克服了传统RNN模型在长期训练时出现的梯度消失与梯度爆炸缺点。