Discriminative learning quadratic discriminant function for handwriting recognition

In character string recognition integrating segmentation and classification, high classification accuracy and resistance to noncharacters are desired to the underlying classifier. In a previous evaluation study, the modified quadratic discriminant function (MQDF) proposed by Kimura et al. was shown to be superior in noncharacter resistance but inferior in classification accuracy to neural networks. This paper proposes a discriminative learning algorithm to optimize the parameters of MQDF with aim to improve the classification accuracy while preserving the superior noncharacter resistance. We refer to the resulting classifier as discriminative learning QDF (DLQDF). The parameters of DLQDF adhere to the structure of MQDF under the Gaussian density assumption and are optimized under the minimum classification error (MCE) criterion. The promise of DLQDF is justified in handwritten digit recognition and numeral string recognition, where the performance of DLQDF is comparable to or superior to that of neural classifiers. The results are also competitive to the best ones reported in the literature.     

A genetic framework using contextual knowledge for segmentation and recognition of handwritten numeral strings

For the first time, a genetic framework using contextual knowledge is proposed for segmentation and recognition of unconstrained handwritten numeral strings. New algorithms have been developed to locate feature points on the string image, and to generate possible segmentation hypotheses. A genetic representation scheme is utilized to show the space of all segmentation hypotheses (chromosomes). For the evaluation of segmentation hypotheses, a novel evaluation scheme is introduced, in order to improve the outlier resistance of the system. Our genetic algorithm tries to search and evolve the population of segmentation hypotheses, and to find the one with the highest segmentation/recognition confidence. The NIST NSTRING SD19 and CENPARMI databases were used to evaluate the performance of our proposed method. Our experiments showed that proper use of contextual knowledge in segmentation, evaluation and search greatly improves the overall performance of the system. On average, our system was able to obtain correct recognition rates of 95.28% and 96.42% on handwritten numeral strings using neural network and support vector classifiers, respectively. These results compare favorably with the ones reported in the literature.     

Class-specific feature polynomial classifier for pattern classification and its application to handwritten numeral recognition

The polynomial classifier (PC) that takes the binomial terms of reduced subspace features as inputs has shown superior performance to multilayer neural networks in pattern classification. In this paper, we propose a class-specific feature polynomial classifier (CFPC) that extracts class-specific features from class-specific subspaces, unlike the ordinary PC that uses a class-independent subspace. The CFPC can be viewed as a hybrid of ordinary PC and projection distance method. The class-specific features better separate one class from the others, and the incorporation of class-specific projection distance further improves the separability. The connecting weights of CFPC are efficiently learned class-by-class to minimize the mean square error on training samples. To justify the promise of CFPC, we have conducted experiments of handwritten digit recognition and numeral string recognition on the NIST Special Database 19 (SD19). The digit recognition task was also benchmarked on two standard databases USPS and MNIST. The results show that the performance of CFPC is superior to that of ordinary PC, and is competitive with support vector classifiers (SVCs).     

Handwritten digit recognition: benchmarking of state-of-the-art techniques

This paper presents the results of handwritten digit recognition on well-known image databases using state-of-the-art feature extraction and classification techniques. The tested databases are CENPARMI, CEDAR, and MNIST. On the test data set of each database, 80 recognition accuracies are given by combining eight classifiers with ten feature vectors. The features include chaincode feature, gradient feature, profile structure feature, and peripheral direction contributivity. The gradient feature is extracted from either binary image or gray-scale image. The classifiers include the k-nearest neighbor classifier, three neural classifiers, a learning vector quantization classifier, a discriminative learning quadratic discriminant function (DLQDF) classifier, and two support vector classifiers (SVCs). All the classifiers and feature vectors give high recognition accuracies. Relatively, the chaincode feature and the gradient feature show advantage over other features, and the profile structure feature shows efficiency as a complementary feature. The SVC with RBF kernel (SVC-rbf) gives the highest accuracy in most cases but is extremely expensive in storage and computation. Among the non-SV classifiers, the polynomial classifier and DLQDF give the highest accuracies. The results of non-SV classifiers are competitive to the best ones previously reported on the same databases.     

基于识别的手写数字串分割

手写数字串的分割与字符识别密切相关.采用基于识别的分割方法,在分割过程中引入识别机制识别分割碎片,将识别结果经过差值运算后置为每个识别对象的识别可信度,利用动态规划找到最佳分割路径.在训练分类器时,使用反例样本估计分类器参数,得到了性能良好的分类器.实验数据表明,利用正例和反例样本结合训练的分类器比只经过正例样本训练的分类器的识别率要高很多.     

基于多分类器的无分割手写数字字符串识别算法

手写体数字字符串识别常用于邮件自动分拣、银行票据和财务报表的录入中,针对其分割识别算法复杂度较高、准确率较低的问题,提出一种多分类器下无分割手写数字字符串识别算法。该算法的核心是采用四个分类器实现粘连字符串的无分割识别;将残差结构应用于LeNet-5网络,以增加网络深度,提高识别准确率,加快收敛速度;使用动态选择策略,以避免长度分类器误分类对识别结果的影响。实验结果表明,在NIST SD19一位数字和Synthetic数据集训练网络下,使用NIST SD19上长度为2、3、4、5、6的字符串验证网络,其识别准确率分别为99.3%、98.5%、98.1%、96.6%和97.2%。     

The recognition of handwritten numeral strings using a two-stage HMM-based method

In this paper, a two-stage HMM-based recognition method allows us to compensate for the possible loss in terms of recognition performance caused by the necessary trade-off between segmentation and recognition in an implicit segmentation-based strategy. The first stage consists of an implicit segmentation process that takes into account some contextual information to provide multiple segmentation-recognition hypotheses for a given preprocessed string. These hypotheses are verified and re-ranked in a second stage by using an isolated digit classifier. This method enables the use of two sets of features and numeral models: one taking into account both the segmentation and recognition aspects in an implicit segmentation-based strategy, and the other considering just the recognition aspects of isolated digits. These two stages have been shown to be complementary, in the sense that the verification stage compensates for the loss in terms of recognition performance brought about by the necessary tradeoff between segmentation and recognition carried out in the first stage. The experiments on 12,802 handwritten numeral strings of different lengths have shown that the use of a two-stage recognition strategy is a promising idea. The verification stage brought about an average improvement of 9.9% on the string recognition rates. On touching digit pairs, the method achieved a recognition rate of 89.6%. Received June 28, 2002 / Revised July 03, 2002     

A combination of CSP-based method with soft margin SVM classifier and generalized RBF kernel for imagery-based brain computer interface applications

Several methods utilizing common spatial pattern (CSP) algorithm have been presented for improving the identification of imagery movement patterns for brain computer interface applications. The present study focuses on improving a CSP-based algorithm for detecting the motor imagery movement patterns. A discriminative filter bank of CSP method using a discriminative sensitive learning vector quantization (DFBCSP-DSLVQ) system is implemented. Four algorithms are then combined to form three methods for improving the efficiency of the DFBCSP-DSLVQ method, namely the kernel linear discriminant analysis (KLDA), the kernel principal component analysis (KPCA), the soft margin support vector machine (SSVM) classifier and the generalized radial bases functions (GRBF) kernel. The GRBF is used as a kernel for the KLDA, the KPCA feature selection algorithms and the SSVM classifier. In addition, three types of classifiers, namely K-nearest neighbor (K-NN), neural network (NN) and traditional support vector machine (SVM), are employed to evaluate the efficiency of the classifiers. Results show that the best algorithm is the combination of the DFBCSP-DSLVQ method using the SSVM classifier with GRBF kernel (SSVM-GRBF), in which the best average accuracy, attained are 92.70% and 83.21%, respectively. Results of the Repeated Measures ANOVA shows the statistically significant dominance of this method at p <?0.05. The presented algorithms are then compared with the base algorithm of this study i.e. the DFBCSP-DSLVQ with the SVM-RBF classifier. It is concluded that the algorithms, which are based on the SSVM-GRBF classifier and the KLDA with the SSVM-GRBF classifiers give sufficient accuracy and reliable results.

     

Lexicon-driven segmentation and recognition of handwritten character strings for Japanese address reading

This paper describes a handwritten character string recognition system for Japanese mail address reading on a very large vocabulary. The address phrases are recognized as a whole because there is no extra space between words. The lexicon contains 111,349 address phrases, which are stored in a trie structure. In recognition, the text line image is matched with the lexicon entries (phrases) to obtain reliable segmentation and retrieve valid address phrases. The paper first introduces some effective techniques for text line image preprocessing and presegmentation. In presegmentation, the text line image is separated into primitive segments by connected component analysis and touching pattern splitting based on contour shape analysis. In lexicon matching, consecutive segments are dynamically combined into candidate character patterns. An accurate character classifier is embedded in lexicon matching to select characters matched with a candidate pattern from a dynamic category set. A beam search strategy is used to control the lexicon matching so as to achieve real-time recognition. In experiments on 3,589 live mail images, the proposed method achieved correct rate of 83.68 percent while the error rate is less than 1 percent.     

Performance evaluation of pattern classifiers for handwritten character recognition

This paper describes a performance evaluation study in which some efficient classifiers are tested in handwritten digit recognition. The evaluated classifiers include a statistical classifier (modified quadratic discriminant function, MQDF), three neural classifiers, and an LVQ (learning vector quantization) classifier. They are efficient in that high accuracies can be achieved at moderate memory space and computation cost. The performance is measured in terms of classification accuracy, sensitivity to training sample size, ambiguity rejection, and outlier resistance. The outlier resistance of neural classifiers is enhanced by training with synthesized outlier data. The classifiers are tested on a large data set extracted from NIST SD19. As results, the test accuracies of the evaluated classifiers are comparable to or higher than those of the nearest neighbor (1-NN) rule and regularized discriminant analysis (RDA). It is shown that neural classifiers are more susceptible to small sample size than MQDF, although they yield higher accuracies on large sample size. As a neural classifier, the polynomial classifier (PC) gives the highest accuracy and performs best in ambiguity rejection. On the other hand, MQDF is superior in outlier rejection even though it is not trained with outlier data. The results indicate that pattern classifiers have complementary advantages and they should be appropriately combined to achieve higher performance. Received: July 18, 2001 / Accepted: September 28, 2001     

Help-Training for semi-supervised support vector machines

In this paper, we propose to reinforce the Self-Training strategy in semi-supervised mode by using a generative classifier that may help to train the main discriminative classifier to label the unlabeled data. We call this semi-supervised strategy Help-Training and apply it to training kernel machine classifiers as support vector machines (SVMs) and as least squares support vector machines. In addition, we propose a model selection strategy for semi-supervised training. Experimental results on both artificial and real problems demonstrate that Help-Training outperforms significantly the standard Self-Training. Moreover, compared to other semi-supervised methods developed for SVMs, our Help-Training strategy often gives the lowest error rate.     

Approximate Test Risk Bound Minimization Through Soft Margin Estimation

Inspired by the great success of margin-based classifiers, there is a trend to incorporate the margin concept into hidden Markov modeling for speech recognition. Several attempts based on margin maximization were proposed recently. In this paper, a new discriminative learning framework, called soft margin estimation (SME), is proposed for estimating the parameters of continuous-density hidden Markov models. The proposed method makes direct use of the successful ideas of soft margin in support vector machines to improve generalization capability and decision feedback learning in minimum classification error training to enhance model separation in classifier design. SME is illustrated from a perspective of statistical learning theory. By including a margin in formulating the SME objective function, SME is capable of directly minimizing an approximate test risk bound. Frame selection, utterance selection, and discriminative separation are unified into a single objective function that can be optimized using the generalized probabilistic descent algorithm. Tested on the TIDIGITS connected digit recognition task, the proposed SME approach achieves a string accuracy of 99.43%. On the 5 k-word Wall Street Journal task, SME obtains relative word error rate reductions of about 10% over our best baseline results in different experimental configurations. We believe this is the first attempt to show the effectiveness of margin-based acoustic modeling for large vocabulary continuous speech recognition in a hidden Markov model framework. Further improvements are expected because the approximate test risk bound minimization principle offers a flexible and rigorous framework to facilitate incorporation of new margin-based optimization criteria into hidden Markov model training.     

Chinese calligraphic style representation for recognition

Chinese calligraphy draws a lot of attention for its beauty and elegance. The various styles of calligraphic characters make calligraphy even more charming. But it is not always easy to recognize the calligraphic style correctly, especially for beginners. In this paper, an automatic character styles representation for recognition method is proposed. Three kinds of features are extracted to represent the calligraphic characters. Two of them are typical hand-designed features: the global feature, GIST and the local feature, scale invariant feature transform. The left one is deep feature which is extracted by a deep convolutional neural network (CNN). The state-of-the-art classifier modified quadratic discriminant function was employed to perform recognition. We evaluated our method on two calligraphic character datasets, the unconstraint real-world calligraphic character dataset (CCD) and SCL (the standard calligraphic character library). And we also compare MQDF with other two classifiers, support vector machine and neural network, to perform recognition. In our experiments, all three kinds of feature are evaluated with all three classifiers, respectively, finding that deep feature is the best feature for calligraphic style recognition. We also fine-tune the deep CNN (alex-net) in Krizhevsky et al. (Advances in Neural Information Processing Systems, pp. 1097–1105, 2012) to perform calligraphic style recognition. It turns out our method achieves about equal accuracy comparing with the fine-tuned alex-net but with much less training time. Furthermore, the algorithm style discrimination evaluation is developed to evaluate the discriminative style quantitatively.     

So near and yet so far: New insight into properties of some well-known classifier paradigms

This article provides some new insight into the properties of four well-established classifier paradigms, namely support vector machines (SVM), classifiers based on mixture density models (CMM), fuzzy classifiers (FCL), and radial basis function neural networks (RBF). It will be shown that these classifiers can be formulated in a way such that they are functionally equivalent or at least highly similar. The interpretation of a specific classifier as being an SVM, CMM, FCL, or RBF then only depends on the objective function and the optimization algorithm used to adjust the parameters. The properties of these four paradigms, however, are very different: a discriminative classifier such as an SVM is expected to have optimal generalization capabilities on new data, a generative classifier such as a CMM also aims at modeling the processes from which the observed data originate, and a comprehensible classifier such as an FCL is intended to be parameterized and understood by human domain experts. We will discuss the advantages and disadvantages of these properties and show how they can be measured numerically in order to compare these classifiers. In such a way, the article aims at supporting a practitioner in assessing the properties of classifier paradigms and in selecting or combining certain paradigms for a given application problem.     

A hybrid discriminative/generative approach to protein fold recognition

There are two standard approaches to the classification task: generative, which use training data to estimate a probability model for each class, and discriminative, which try to construct flexible decision boundaries between the classes. An ideal classifier should combine these two approaches. In this paper a classifier combining the well-known support vector machine (SVM) classifier with regularized discriminant analysis (RDA) classifier is presented. The hybrid classifier is used for protein structure prediction which is one of the most important goals pursued by bioinformatics. The obtained results are promising, the hybrid classifier achieves better result than the SVM or RDA classifiers alone. The proposed method achieves higher recognition ratio than other methods described in the literature.     

极限学习机集成在骨髓细胞分类中的应用

骨髓细胞的分类有重要的医学诊断意义。先对骨髓细胞图像分割和特征提取,用提取出来的训练集对极限学习机训练,再用该分类器对未知样本识别。针对单个分类器性能的不稳定,提出基于元胞自动机的极限学习机集成算法。通过元胞自动机抽样策略构建差异大的训练子集,多个分类器并行学习,多数投票法联合决策。实验结果表明,与BP、支持向量机比较,该算法基本无参数调整,学习速度快,分类精度高能达到97.33%,且有效克服了神经网络分类器不稳定的缺点。     

Neural-network classifiers for recognizing totally unconstrainedhandwritten numerals

Artificial neural networks have been recognized as a powerful tool for pattern classification problems, but a number of researchers have also suggested that straightforward neural-network approaches to pattern recognition are largely inadequate for difficult problems such as handwritten numeral recognition. In this paper, we present three sophisticated neural-network classifiers to solve complex pattern recognition problems: multiple multilayer perceptron (MLP) classifier, hidden Markov model (HMM)/MLP hybrid classifier, and structure-adaptive self-organizing map (SOM) classifier. In order to verify the superiority of the proposed classifiers, experiments were performed with the unconstrained handwritten numeral database of Concordia University, Montreal, Canada. The three methods have produced 97.35%, 96.55%, and 96.05% of the recognition rates, respectively, which are better than those of several previous methods reported in the literature on the same database.     

基于多配置特征包的目标检测

针对目标检测中的特征失配问题,提出多配置特征包的概念,刻画同一特征可能出现的不同失配情况。目标分类器学习时,利用Boosting算法学习出最具鉴别力特征包,每个特征包对应一个单特征和它的失配情况,目标分类器是最优特征包分类器的线性组合。进一步地,引入多示例学习思想,有效评估特征包鉴别力、学习特征包分类器。在人脸数据集上的实验表明,较之传统方法,考虑特征失配后,文中算法能获得更好的检测性能。同时,与固定包生成方式相比,多配置特征包能较好拟合特征失配情况,在提高检测率的同时获得更小的检测器尺寸。     

Localized Generalization Error of Gaussian-based Classifiers and Visualization of Decision Boundaries

In pattern classification problem, one trains a classifier to recognize future unseen samples using a training dataset. Practically, one should not expect the trained classifier could correctly recognize samples dissimilar to the training dataset. Therefore, finding the generalization capability of a classifier for those unseen samples may not help in improving the classifiers accuracy. The localized generalization error model was proposed to bound above the generalization mean square error for those unseen samples similar to the training dataset only. This error model is derived based on the stochastic sensitivity measure(ST-SM)of the classifiers. We present the ST-SMS for various Gaussian based classifiers: radial basis function neural networks and support vector machine in this paper. At the end of this work, we compare the decision boundaries visualization using the training samples yielding the largest sensitivity measures and the one using support vectors in the input space.     

液晶字符识别的CNN和SVM组合分类器

针对仪表液晶显示字符识别问题,提出一种结合了卷积神经网络(CNN)和支持向量机(SVM)的字符识别方法.分别采用具有并联结构的CNN模型和基于梯度方向直方图(HOG)特征的SVM方法构建基本分类器,当2个分类器的结果存在冲突时,利用CNN的softmax输出最大值判决最终结果,当其大于设定阈值时采用CNN分类器的结果,...