基于图片问答的静态重启随机梯度下降算法

图片问答是计算机视觉与自然语言处理交叉的多模态学习任务.为了解决该任务,研究人员提出堆叠注意力网络(stacked attention networks, SANs).研究发现该模型易陷入不好的局部最优解,引发较高的问答错误率.为了解决该问题,提出基于图片问答的静态重启随机梯度下降算法.实验结果和分析表明：它的准确率比基准算法提高0.29%,但其收敛速度慢于基准算法.为了验证改善性能的显著性,对实验结果进行统计假设检验.T检验结果证明它的改善性能是极其显著的.为了验证它在同类算法中的有效性,将该算法和当前最好的一阶优化算法进行有效性实验,实验结果和分析证明它更有效.为了验证它的泛化性能和推广价值,在经典的Cifar-10数据集上进行图像识别实验.实验结果和T检验结果证明:它具有良好的泛化性能和较好的推广价值.     

Effective and Efficient Video Compression by the Deep Learning Techniques

Deep learning has reached many successes in Video Processing. Video has become a growing important part of our daily digital interactions. The advancement of better resolution content and the large volume offers serious challenges to the goal of receiving, distributing, compressing and revealing high-quality video content. In this paper we propose a novel Effective and Efficient video compression by the Deep Learning framework based on the flask, which creatively combines the Deep Learning Techniques on Convolutional Neural Networks (CNN) and Generative Adversarial Networks (GAN). The video compression method involves the layers are divided into different groups for data processing, using CNN to remove the duplicate frames, repeating the single image instead of the duplicate images by recognizing and detecting minute changes using GAN and recorded with Long Short-Term Memory (LSTM). Instead of the complete image, the small changes generated using GAN are substituted, which helps with frame-level compression. Pixel wise comparison is performed using K-nearest Neighbours (KNN) over the frame, clustered with K-means and Singular Value Decomposition (SVD) is applied for every frame in the video for all three colour channels [Red, Green, Blue] to decrease the dimension of the utility matrix [R, G, B] by extracting its latent factors. Video frames are packed with parameters with the aid of a codec and converted to video format and the results are compared with the original video. Repeated experiments on several videos with different sizes, duration, Frames per second (FPS), and quality results demonstrated a significant resampling rate. On normal, the outcome delivered had around a 10% deviation in quality and over half in size when contrasted, and the original video.     

一种随机配置网络的模型与数据混合并行学习方法

随机配置网络(Stochastic configuration networks, SCNs)在增量构建过程引入监督机制来分配隐含层参数以确保其无限逼近特性, 具有易于实现、收敛速度快、泛化性能好等优点. 然而, 随着数据量的不断扩大, SCNs的建模任务面临一定的挑战性. 为了提高神经网络算法在大数据建模中的综合性能, 本文提出了一种混合并行随机配置网络(Hybrid parallel stochastic configuration networks, HPSCNs)架构, 即: 模型与数据混合并行的增量学习方法. 所提方法由不同构建方式的左右两个SCNs模型组成, 以快速准确地确定最佳隐含层节点, 其中左侧采用点增量网络(PSCN), 右侧采用块增量网络(BSCN); 同时每个模型建立样本数据的动态分块方法, 从而加快候选“节点池”的建立、降低计算量. 所提方法首先通过大规模基准数据集进行了对比实验, 然后应用在一个实际工业案例上, 表明其有效性.     

一种减小方差求解非光滑问题的随机优化算法

随机优化算法是求解大规模机器学习问题的高效方法之一.随机学习算法使用随机抽取的单个样本梯度代替全梯度,有效节省了计算量,但却会导致较大的方差.近期的研究结果表明:在光滑损失优化问题中使用减小方差策略,能够有效提高随机梯度算法的收敛速率.考虑求解非光滑损失问题随机优化算法COMID(compositeobjective mirror descent)的方差减小问题.首先证明了COMID具有方差形式的O(1/√T+σ²/√T)收敛速率,其中,T是迭代步数,σ²是方差.该收敛速率保证了减小方差的有效性,进而在COMID中引入减小方差的策略,得到一种随机优化算法α-MDVR(mirror descent with variance reduction).不同于Prox-SVRG(proximal stochastic variance reduced gradient),α-MDVR收敛速率不依赖于样本数目,每次迭代只使用部分样本来修正梯度.对比实验验证了α-MDVR既减小了方差,又节省了计算时间.     

融合隐语义模型与门控循环单元的推荐算法

在传统的推荐算法中, 往往缺乏对用户长短期兴趣偏好问题的考虑, 而随着深度学习在推荐算法中应用的不断深入, 这一问题能够得到很好的解决. 本文针对该问题提出一种融合隐语义模型与门控循环单元的长短期推荐算法(recommendation algorithm based on long short-term, RA_LST), 以实现对用户长短期偏好的分别捕捉, 有效解决了因用户兴趣随时间变化而导致推荐效果下降的问题. 最终的实验结果表明, 本文提出的算法在不同的数据集上都表现出了推荐准确性的提升.     

XCSG在多机器人强化学习中的应用

XCS分类器在解决机器人强化学习方面已显示出较强的能力,但在多机器人领域仅局限于MDP环境,只能解决环境空间较小的学习问题。提出了XCSG来解决多机器人的强化学习问题。XCSG建立低维的逼近函数,梯度下降技术利用在线知识建立稳定的逼近函数,使Q-表格一直保持在稳定低维状态。逼近函数Q不仅所需的存储空间更小,而且允许机器人在线对已获得的知识进行归纳一般化。仿真实验表明,XCSG算法很好地解决了多机器人学习空间大、学习速度慢、学习效果不确定等问题。     

深度学习批归一化及其相关算法研究进展

深度学习已经广泛应用到各个领域, 如计算机视觉和自然语言处理等, 并都取得了明显优于早期机器学习算法的效果. 在信息技术飞速发展的今天, 训练数据逐渐趋于大数据集, 深度神经网络不断趋于大型化, 导致训练越来越困难, 速度和精度都有待提升. 2013年, Ioffe等指出训练深度神经网络过程中存在一个严重问题: 中间协变量迁移(Internal covariate shift), 使网络训练过程对参数初值敏感、收敛速度变慢, 并提出了批归一化(Batch normalization, BN)方法, 以减少中间协变量迁移问题, 加快神经网络训练过程收敛速度. 目前很多网络都将BN作为一种加速网络训练的重要手段, 鉴于BN的应用价值, 本文系统综述了BN及其相关算法的研究进展. 首先对BN的原理进行了详细分析. BN虽然简单实用, 但也存在一些问题, 如依赖于小批量数据集的大小、训练和推理过程对数据处理方式不同等, 于是很多学者相继提出了BN的各种相关结构与算法, 本文对这些结构和算法的原理、优势和可以解决的主要问题进行了分析与归纳. 然后对BN在各个神经网络领域的应用方法进行了概括总结, 并且对其他常用于提升神经网络训练性能的手段进行了归纳. 最后进行了总结, 并对BN的未来研究方向进行了展望.     

基于改进并行回火算法的RBM网络训练研究

目前受限玻尔兹曼机网络训练算法主要是基于采样的算法.当用采样算法进行梯度计算时,得到的采样梯度是真实梯度的近似值,采样梯度和真实梯度之间存在较大的误差,这严重影响了网络的训练效果.针对该问题,本文首先分析了采样梯度和真实梯度之间的数值误差和方向误差,以及它们对网络训练性能的影响,然后从马尔科夫采样的角度对以上问题进行了理论分析,并建立了梯度修正模型,通过修正梯度对采样梯度进行数值和方向的调节,并提出了基于改进并行回火算法的训练算法,即GFPT（Gradient fixing parallel tempering）算法.最后给出GFPT算法与现有算法的对比实验,仿真结果表明,GFPT算法可以极大地减小采样梯度和真实梯度之间的误差,大幅度提升受限玻尔兹曼机网络的训练效果.     

分批处理的K-means算法并行实现

为解决K-means 算法计算量大、收敛缓慢、运算耗时长等问题,给出一种新的K-means算法的并行实现方法。在通用计算图形处理器架构上,使用统一计算设备架构(CUDA)加速K-means算法。采用分批原则,更合理地运用CUDA提供的各种存储器,避免访问冲突,同时减少对数据集的访问次数,以提高算法效率。在大规模数据集中的实验结果表明,该算法具有较快的聚类速度。     

A PID-incorporated Latent Factorization of Tensors Approach to Dynamically Weighted Directed Network Analysis

A large-scale dynamically weighted directed network(DWDN) involving numerous entities and massive dynamic interaction is an essential data source in many big-data-related applications, like in a terminal interaction pattern analysis system(TIPAS). It can be represented by a high-dimensional and incomplete(HDI) tensor whose entries are mostly unknown. Yet such an HDI tensor contains a wealth knowledge regarding various desired patterns like potential links in a DWDN. A latent factorization-of-ten...     

Estimating time and score uncertainty in generating successful learning paths under time constraints

This paper addresses the problem of course (path) generation when a learner's available time is not enough to follow the complete course. We propose a method to recommend successful paths regarding a learner's available time and his/her knowledge background. Our recommender is an instance of long term goal recommender systems (LTRS). This method, after locating a target learner in a course graph, applies a depth‐first search algorithm to find all paths for the learner given a time limitation. In addition, our method estimates learning time and score for all paths. It also indicates the probability of error for the estimated time and score for each path. Finally, our method recommends a path that satisfies the learner's time restriction while maximizing expected learning score. In order to evaluate our proposals for time and score estimation, we used the mean absolute error and average MAE. We have evaluated time and score estimation methods, including one proposed in the literature, on two E‐learning datasets.     

Automatic recognition of epileptic EEG patterns via Extreme Learning Machine and multiresolution feature extraction

Epilepsy is one of the most common neurological disorders- approximately one in every 100 people worldwide are suffering from it. In this paper, a novel pattern recognition model is presented for automatic epilepsy diagnosis. Wavelet transform is investigated to decompose EEG into five EEG frequency bands which approximate to delta (δ), theta (θ), alpha (α), beta (β), and gamma (γ) bands. Complexity based features such as permutation entropy (PE), sample entropy (SampEn), and the Hurst exponent (HE) are extracted from both the original EEG signals and each of the frequency bands. The wavelet-based methodology separates the alterations in PE, SampEn, and HE in specific frequency bands of the EEG. The effectiveness of these complexity based measures in discriminating between normal brain state and brain state during the absence of seizures is evaluated using the Extreme Learning Machine (ELM). It is discovered that although there exists no significant differences in the feature values extracted from the original EEG signals, differences can be recognized when the features are examined within specific EEG frequency bands. A genetic algorithm (GA) is developed to choose feature subsets that are effective for enhancing the recognition performance. The GA is also examined for weight alteration for both sensitivity and specificity. The results show that the abnormal EEG diagnosis rate of the model without the involvement of the genetic algorithm is 85.9%. However, the diagnosis rate of the model increases to 94.2% when the genetic algorithm is integrated as a feature selector.     

PRAM和LARPBS模型上的近似串匹配并行算法

近似串匹配技术在网络信息搜索、数字图书馆、模式识别、文本挖掘、IP路由查找、网络入侵检测、生物信息学、音乐研究计算等领域具有广泛的应用.基于CREW-PRAM(parallel random access machine with concurrent read and exclusive write)模型,采用波前式并行推进的方法直接计算编辑距离矩阵D,设计了一个允许k-差别的近似串匹配动态规划并行算法,该算法使用(m+1)个处理器,时间复杂度为O(n),算法理论上达到线性加速;采取水平和斜向双并行计算编辑距离矩阵D的方法,设计了一个使用((m+1)个处理器和O(n/(+m)时间的、可伸缩的、允许k-差别的近似串匹配动态规划并行算法,.基于分治策略,通过灵活拆分总线和合并子总线动态重构光总线系统,并充分利用光总线的消息播送技术和并行计算前缀和的方法,实现了汉明距离的并行计算,设计了两个基于LARPBS(linear arrays with reconfigurable pipelined bus system)模型的通信高效、可扩放的允许k-误配的近似串匹配并行算法,其中一个算法使用n个处理器,时间为O(m);另一个为常数时间算法,使用mn个处理器.     

Proximal Alternating-Direction-Method-of- Multipliers-Incorporated Nonnegative Latent Factor Analysis

High-dimensional and incomplete (HDI) data subject to the nonnegativity constraints are commonly encountered in a big data-related application concerning the interactions among numerous nodes. A nonnegative latent factor analysis (NLFA) model can perform representation learning to HDI data efficiently. However, existing NLFA models suffer from either slow convergence rate or representation accuracy loss. To address this issue, this paper proposes a proximal alternating-direction-method-of-multipliers-based nonnegative latent factor analysis (PAN) model with two-fold ideas: 1) adopting the principle of alternating-direction-method-of-multipliers to implement an efficient learning scheme for fast convergence and high computational efficiency; and 2) incorporating the proximal regularization into the learning scheme to suppress the optimization fluctuation for high representation learning accuracy to HDI data. Theoretical studies verify that PAN converges to a Karush-Kuhn-Tucker (KKT) stationary point of its nonnegativity-constrained learning objective with its learning scheme. Experimental results on eight HDI matrices from real applications demonstrate that the proposed PAN model outperforms several state-of-the-art models in both estimation accuracy for missing data of an HDI matrix and computational efficiency.      

A newly-designed fault diagnostic method for transformers via improved empirical wavelet transform and kernel extreme learning machine

In order to solve the problems of unsatisfactory diagnosis performance and unstable model of conventional fault diagnosis methods for transformers, a new approach based on improved empirical wavelet transform (IEWT) and salp swarm algorithm (SSA) optimized kernel extreme learning machine (KELM) is proposed in this study. Firstly, IEWT is used to adaptively decompose the vibration signal to obtain a set of empirical wavelet functions (EWFs). Secondly, the first n-order components with high correlation coefficient are collected. Thirdly, the mean value, variance, kurtosis, refine composite multiscale entropy (RCMSE), and time-frequency entropy(TFE) of these n-order components are calculated to construct a fusion feature vector. Finally, a two-level diagnostic model based on SSA-KELM is established. The first-level of it is applied to identify normal and abnormal states, and the second-level is selected to identify fault categories in the abnormal states. The proposed method can effectively diagnose the existing fault categories in the training set and accurately identify the unknown categories of faults. Experimental results show that the proposed method can efficiently extract features of different vibration signals and identify the faults, with an average classification accuracy of 96.25%. It is better than other methods, such as wavelet packet energy spectrum analysis-KELM and EWT-fisher.     

Mixtures of Factor Analysers. Bayesian Estimation and Inference by Stochastic Simulation

Factor Analysis (FA) is a well established probabilistic approach to unsupervised learning for complex systems involving correlated variables in high-dimensional spaces. FA aims principally to reduce the dimensionality of the data by projecting high-dimensional vectors on to lower-dimensional spaces. However, because of its inherent linearity, the generic FA model is essentially unable to capture data complexity when the input space is nonhomogeneous. A finite Mixture of Factor Analysers (MFA) is a globally nonlinear and therefore more flexible extension of the basic FA model that overcomes the above limitation by combining the local factor analysers of each cluster of the heterogeneous input space. The structure of the MFA model offers the potential to model the density of high-dimensional observations adequately while also allowing both clustering and local dimensionality reduction. Many aspects of the MFA model have recently come under close scrutiny, from both the likelihood-based and the Bayesian perspectives. In this paper, we adopt a Bayesian approach, and more specifically a treatment that bases estimation and inference on the stochastic simulation of the posterior distributions of interest. We first treat the case where the number of mixture components and the number of common factors are known and fixed, and we derive an efficient Markov Chain Monte Carlo (MCMC) algorithm based on Data Augmentation to perform inference and estimation. We also consider the more general setting where there is uncertainty about the dimensionalities of the latent spaces (number of mixture components and number of common factors unknown), and we estimate the complexity of the model by using the sample paths of an ergodic Markov chain obtained through the simulation of a continuous-time stochastic birth-and-death point process. The main strengths of our algorithms are that they are both efficient (our algorithms are all based on familiar and standard distributions that are easy to sample from, and many characteristics of interest are by-products of the same process) and easy to interpret. Moreover, they are straightforward to implement and offer the possibility of assessing the goodness of the results obtained. Experimental results on both artificial and real data reveal that our approach performs well, and can therefore be envisaged as an alternative to the other approaches used for this model.     

面向机器学习的分布式并行计算关键技术及应用

当前机器学习等算法的计算、迭代过程日趋复杂, 充足的算力是保障人工智能应用落地效果的关键。本文首先提出一种适应倾斜数据的分布式异构环境下的任务时空调度算法,有效提升机器学习模型训练等任务的平均效率;其次,提出分布式异构环境下高效的资源管理系统与节能调度算法,实现分布式异构环境下基于动态预测的跨域计算资源迁移及电压/频率的动态调节,节省了系统的整体能耗;然后构建了适应于机器学习/深度学习算法迭代的分布式异构优化环境,提出了面向机器学习/图迭代算法的分布式并行优化基本方法。最后,本文研发了面向领域应用的智能分析系统,并在制造、交通、教育、医疗等领域推广应用,解决了在高效数据采集、存储、清洗、融合与智能分析等过程中普遍存在的性能瓶颈问题。