A review of Nyström methods for large-scale machine learning

Generating a low-rank matrix approximation is very important in large-scale machine learning applications. The standard Nyström method is one of the state-of-the-art techniques to generate such an approximation. It has got rapid developments since being applied to Gaussian process regression. Several enhanced Nyström methods such as ensemble Nyström, modified Nyström and SS-Nyström have been proposed. In addition, many sampling methods have been developed. In this paper, we review the Nyström methods for large-scale machine learning. First, we introduce various Nyström methods. Second, we review different sampling methods for the Nyström methods and summarize them from the perspectives of both theoretical analysis and practical performance. Then, we list several typical machine learning applications that utilize the Nyström methods. Finally, we make our conclusions after discussing some open machine learning problems related to Nyström methods.     

基于Nyström方法的偏好特征提取

针对电影评分中特征提取效率较低的问题,提出了与QR分解相结合的Nyström方法。首先,利用自适应方法进行采样,然后对内部矩阵进行QR分解,将分解后的矩阵与内部矩阵进行重新组合并进行特征分解。Nyström方法的近似过程与标志点选取的数量以及选取标志点的过程密切相关,选取一系列具有标志性的点来保证采样后的近似性,自适应的采样方法能够保证近似的精度。QR分解能够保证矩阵的稳定性,提高偏好特征提取的精度。偏好特征提取的精度越高,推荐系统的稳定性就会越高,推荐的精度也会提高。最后在真实的观众对电影评分的数据集上进行了特征提取的实验,该电影数据集中包含480189个用户,17770部电影,实验结果表明,提取相同数目的标志点时,该算法的精度和效率都有了一定程度的提高：相对于采样前,时间复杂度由原来的O（n³）减少为O（nc²）（c<<n）;与标准的Nyström相比,误差控制在25%以下。     

Generalized iterative RELIEF for supervised distance metric learning

The RELIEF algorithm is a popular approach for feature weighting. Many extensions of the RELIEF algorithm are developed, and I-RELIEF is one of the famous extensions. In this paper, I-RELIEF is generalized for supervised distance metric learning to yield a Mahananobis distance function. The proposed approach is justified by showing that the objective function of the generalized I-RELIEF is closely related to the expected leave-one-out nearest-neighbor classification rate. In addition, the relationships among the generalized I-RELIEF, the neighbourhood components analysis, and graph embedding are also pointed out. Experimental results on various data sets all demonstrate the superiority of the proposed approach.     

基于奇异值分解的宽基线图像匹配算法

图像匹配是计算机视觉中许多应用研究的基础.窄基线匹配技术虽然较为成熟,但是解决能力有限,不能处理较大的尺度、旋转、亮度以及仿射变化引起的宽基线图像序列的匹配.针对宽基线图像序列匹配的特点,在分析传统SVD匹配算法不足的基础上,引入具有尺度和旋转不变性的特征,改进邻近矩阵的度量方式,设计了一种新的基于奇异值分解的宽基线自动匹配算法.通过对比实验表明,该算法性能优于基于SIFT距离的匹配器和原SVD匹配算法,对于存在较大的尺度、旋转、亮度等宽基线变化的图像序列,能够自动获得更多的正确匹配点对和较高的准确性,鲁棒性强,甚至对视角和仿射变换也有一定的适应性.     

Numerical solution of nonlinear two-dimensional Fredholm integral equations of the second kind using Sinc Nyström method

In this paper, a double-exponential (DE) Sinc Nyström method is utilized to solve nonlinear two-dimensional Fredholm integral equations of the second kind. Using the DE transformation, the Sinc quadrature rule for a definite integral is extended to double integral over a rectangular region. Therefore, a nonlinear Fredholm integral equation is reduced to a system of nonlinear algebraic equations, which is solved using the Newton iteration method. Convergence analysis shows that the proposed method can converge exponentially. Several numerical examples are provided to demonstrate the high efficiency and accuracy of the proposed method.     

Embedded implicit Runge–Kutta Nyström method for solving second-order differential equations

An embedded diagonally implicit Runge–Kutta Nyström (RKN) method is constructed for the integration of initial-value problems for second-order ordinary differential equations possessing oscillatory solutions. This embedded method is derived using a three-stage diagonally implicit RKN method of order four within which a third-order three stage diagonally implicit RKN method is embedded. We demonstrate how this system can be solved, and by an appropriate choice of free parameters, we obtain an optimized RKN(4,3) embedded algorithm. We also examine the intervals of stability and show that the method is strongly stable within an appropriate region of stability and is thus suitable for oscillatory problems by applying the method to the test equation y″=?ω² y, ω>0. Necessary and sufficient conditions are given for this method to possess non-vanishing intervals of periodicity, for the fourth-order method. Finally, we present the coefficients of the method optimized for small truncation errors. This new scheme is likely to be efficient for the numerical integration of second-order differential equations with periodic solutions, using adaptive step size.     

Numerical results for linear Fredholm integral equations of the first kind over surfaces in three dimensions

Linear Fredholm integral equations of the first kind over surfaces are less familiar than those of the second kind, although they arise in many applications like computer tomography, heat conduction and inverse scattering. This article emphasizes their numerical treatment, since discretization usually leads to ill-conditioned linear systems. Strictly speaking, the matrix is nearly singular and ordinary numerical methods fail. However, there exists a numerical regularization method – the Tikhonov method – to deal with this ill-conditioning and to obtain accurate numerical results.     

Symmetric and symplectic exponentially fitted Runge–Kutta–Nyström methods for Hamiltonian problems

The construction of symmetric and symplectic exponentially fitted modified Runge–Kutta–Nyström (SSEFRKN) methods is considered. Based on the symmetry, symplecticity, and exponentially fitted conditions, new explicit modified RKN integrators with FSAL property are obtained. The new integrators integrate exactly differential systems whose solutions can be expressed as linear combinations of functions from the set { exp(± iωt)}, ω > 0, i² = −1, or equivalently from the set { cos(ωt), sin(ωt)}. The phase properties of the new integrators are examined and their periodicity regions are obtained. Numerical experiments are accompanied to show the high efficiency and competence of the new SSEFRKN methods compared with some highly efficient nonsymmetric symplecti EFRKN methods in the literature.     

一种融合语义距离的最近邻图像标注方法

传统的基于最近邻的图像标注方法效果不佳,主要原因在于提取图像视觉特征时,损失了很多有价值的信息.提出了一种改进的最近邻分类模型.首先利用距离测度学习方法,引入图像的语义类别信息进行训练,生成新的语义距离;然后利用该距离对每一类图像进行聚类,生成多个类内的聚类中心;最后通过计算图像到各个聚类中心的语义距离来构建最近邻分类模型.在构建最近邻分类模型的整个过程中,都使用训练得到的语义距离来计算,这可以有效减少相同图像类内的变动和不同图像类之间的相似所造成的语义鸿沟.在ImageCLEF2012图像标注数据库上进行了实验,将本方法与传统分类模型和最新的方法进行了比较,验证了本方法的有效性.     

New optimized symmetric and symplectic trigonometrically fitted RKN methods for second-order oscillatory differential equations

Symmetric, symplectic and trigonometrically fitted Runge–Kutta–Nyström (SSTFRKN) methods for second-order differential equations with oscillatory solutions are investigated. Symmetry, symplecticity and trigonometric fitting conditions for modified Runge–Kutta–Nyström (RKN) methods are presented. Order conditions for modified RKN methods are derived via the special Nyström tree theory. Two explicit SSTFRKN methods with variable nodes are derived. The two new methods are zero-dissipative due to symplecticity. Their dispersion orders are analysed and their periodicity regions are obtained. The results of numerical experiments show the robustness and competence of the new SSTFRKN methods compared with some highly efficient codes in the recent literature.     

无需特征分解的快速谱聚类算法

为了解决样本数较大时,传统谱聚类算法执行特征分解消耗时间过大的问题,提出了一种无需特征分解的快速谱聚类算法,通过乘法更新迭代来降低时间开销。首先,利用Nyström方法进行随机采样,建立了采样矩阵和原始矩阵之间的关系;其次,基于乘法更新原理实现矩阵指示器矩阵的迭代更新;最后,在理论上对所设计算法进行了正确性和收敛性分析。在广泛使用的五个真实数据集和三个人工合成数据集上进行测试。实验结果表明,在真实数据集上,所提算法的标准互信息（NMI）平均值为0.45,与k-means聚类算法相比提高了12.50%;运行时间为61.73 s,与传统谱聚类算法相比减少了61.13%;而且表现性能优于层次聚类算法,验证了该算法的有效性。     

基于最优解耦的鲁棒故障检测方法研究

系统地研究了线性系统在模型参数变动和存在外部扰动下的控制系统故障的鲁棒故障检测方法。分析了在与干扰信号不能完全解耦的情况下,如何设计残差信号使其对模型参数变化及噪声不敏感,而对故障敏感,由此得出了对未知输入具有鲁棒性的条件方程式,并基于最优解耦理论,通过矩阵的奇异值分解实现了最优解耦。在给出的仿真实例中,验证了该文提出方法的有效性。     

Fast convergence pirkn-type pc methods with adams-type predictors ∗

This paper discusses predictor–corrector iteration schemes (PC iteration schemes) based on direct collocatio–based Runge–Kutt–Nyström corrector methods (RKN corrector methods) for solving nonstiff initial-value problems (IVPs) for systems of special second-order differential equations y′′(t) = f(y(t)) Our approach is to regard the well-known parallel-iterated RKN methods (PIRKN methods) as PC iteration processes in which the simple, low-order last step value predictors are replaced with the high-order Adams-type predictors. Moreover, the param-eters of the new direct collocation-based RKN corrector methods are chosen in such a way that the convergence rate of the considered PC iteration processes is optimized. In this way, we obtain parallel PC methods with fast convergence and high-accurate predictions. Application of the resulting parallel PC methods to a few widely-used test problems reveals that the sequential costs are very much reduced when compared with the parallel and sequential explicit RKN methods from the literature.     

无需特征分解的快速谱聚类算法

为了解决样本数较大时,传统谱聚类算法执行特征分解消耗时间过大的问题,提出了一种无需特征分解的快速谱聚类算法,通过乘法更新迭代来降低时间开销.首先,利用Nystr?m方法进行随机采样,建立了采样矩阵和原始矩阵之间的关系;其次,基于乘法更新原理实现矩阵指示器矩阵的迭代更新;最后,在理论上对所设计算法进行了正确性和收敛性分析...     

Bootstrap confidence intervals for principal response curves

The principal response curve (PRC) model is of use to analyse multivariate data resulting from experiments involving repeated sampling in time. The time-dependent treatment effects are represented by PRCs, which are functional in nature. The sample PRCs can be estimated using a raw approach, or the newly proposed smooth approach. The generalisability of the sample PRCs can be judged using confidence bands. The quality of various bootstrap strategies to estimate such confidence bands for PRCs is evaluated. The best coverage was obtained with BC_a intervals using a non-parametric bootstrap. The coverage appeared to be generally good, except for the case of exactly zero population PRCs for all conditions. Then, the behaviour is irregular, which is caused by the sign indeterminacy of the PRCs. The insights obtained into the optimal bootstrap strategy are useful to apply in the PRC model, and more generally for estimating confidence intervals in singular value decomposition based methods.     

Capturing the symptoms of malicious code in electronic documents by file’s entropy signal combined with machine learning

Email cyber-attacks based on malicious documents have become popular techniques in today’s sophisticated attacks. Persistent efforts have been made to detect such attacks, but there are still some common defects in the existing methods, including the inability to capture unknown attacks, high overhead of resource and time, and only can be used to detect specific formats of documents. This study proposes a new method named Entropy Signal Reflects the Malicious Document (ESRMD), which can identify malicious documents based on the entropy distribution of the file. ESRMD is a machine learning classifier, which differ from the traditional approaches in that ESRMD extracts both global and structural entropy features from the entropy sequence, enduring it the ability to deal with various formats documents and fight against the parser-confusion and obfuscated attacks. To assess the validity of the proposed model, we conducted extensive experiments on a collected dataset which contains 10,381 samples, including malware (51.47%) and benign (48.53%) samples. Through extensive experiments, ESRMD showed its superiority comparing with some leading anti-virus engines and prevalent tools, achieving good performance on the true positive rate and ROC with the value of 96.00% and 99.2% respectively.     

基于奇异值分解和径向基函数神经网络的人脸识别算法研究

提出一种基于奇异值分解和径向基函数神经网络的人脸特征提取与识别方法,来解决人脸识别中的高维、小样本问题。该方法采用奇异值分解、奇异值降维压缩、奇异值矢量标准化和奇异值矢量排序,最后得到用于识别的奇异值特征矢量。运用基于径向基函数神经网络分类器进行人脸分类识别。在ORL数据库上进行实验和数据分析表明,该方法无论是在分类的错误率上还是在学习的效率上都能表现出极好的性能。     

Distance metric learning by knowledge embedding

This paper presents an algorithm which learns a distance metric from a data set by knowledge embedding and uses the new distance metric to solve nonlinear pattern recognition problems such a clustering.     

基于特征变换和度量网络的小样本学习算法

在小样本分类任务中, 每个类别可供训练的样本数量非常有限. 因此在特征空间中同类样本分布稀疏, 异类样本间边界模糊. 提出一种新的基于特征变换和度量网络(Feature transformation and metric networks, FTMN)的小样本学习算法用于小样本分类任务. 算法通过嵌入函数将样本映射到特征空间, 并计算输入该样本与所属类别中心的特征残差. 构造一个特征变换函数对该残差进行学习, 使特征空间内的样本特征经过该函数后向同类样本中心靠拢. 利用变换后的样本特征更新类别中心, 使各类别中心间的距离增大. 算法进一步构造了一种新的度量函数, 对样本特征中每个局部特征点的度量距离进行联合表达, 该函数能够同时对样本特征间的夹角和欧氏距离进行优化. 算法在小样本分类任务常用数据集上的优秀表现证明了算法的有效性和泛化性.