Effectiveness of error correcting output coding methods in ensemble and monolithic learning machines

Abstract Error Correcting Output Coding (ECOC) methods for multiclass classification present several open problems ranging from the trade-off between their error recovering capabilities and the learnability of the induced dichotomies to the selection of proper base learners and to the design of well-separated codes for a given multiclass problem. We experimentally analyse some of the main factors affecting the effectiveness of ECOC methods. We show that the architecture of ECOC learning machines influences the accuracy of the ECOC classifier, highlighting that ensembles of parallel and independent dichotomic Multi-Layer Perceptrons are well-suited to implement ECOC methods. We quantitatively evaluate the dependence among codeword bit errors using mutual information based measures, experimentally showing that a low dependence enhances the generalisation capabilities of ECOC. Moreover we show that the proper selection of the base learner and the decoding function of the reconstruction stage significantly affects the performance of the ECOC ensemble. The analysis of the relationships between the error recovering power, the accuracy of the base learners, and the dependence among codeword bits show that all these factors concur to the effectiveness of ECOC methods in a not straightforward way, very likely dependent on the distribution and complexity of the data.An erratum to this article can be found at     

基于KNN模型的层次纠错输出编码算法

纠错输出编码是一种解决多类分类问题的有效方法,但其编码矩阵只对类进行编码且都采用事先构造出来的统一形式,适应性较差。为此,提出一种新颖的层次纠错输出编码算法。该算法在训练阶段先通过KNN模型算法在数据集上构建多个同类簇,选取各类中最具代表性的簇形成层次编码矩阵,然后再根据编码矩阵进行单分类器训练。在测试阶段,该算法通过模型融合进一步发挥KNN模型和纠错输出编码各自的优点。在UCI公共数据集上的实验结果表明,新方法的性能优于KNN模型算法和纠错输出编码算法。     

基于最小二乘和纠错输出编码的多类分类

多类分类是目标识别中必须面对的一个关键问题,现有分类器大都为二分器,无法满足对多类目标进行分类,为此,提出利用纠错输出编码方法对多类问题进行分解,即把多类问题转化成二类问题;同时讨论一种基于最小二乘法对二分器结果进行融合的策略。实验分别对UCI数据集和三种一维距离像数据集进行测试,结果表明与经典的多分类器相比,提出的多类分类策略有较高的分类正确率。     

适用于非平衡数据的多关系多分类模型

针对多关系多分类的非平衡数据,提出一种分类模型。在预处理阶段,建立目标类纠错输出编码(ECOC)、目标关系与背景关系间的虚拟连接并完成属性聚集处理,进而划分训练集和验证集。在训练阶段,依据一对多划分思想,结合CrossMine算法构造多个子分类器,采用AUC法评估验证各子分类器。在验证阶段,比较目标类ECOC与各子分类器分类结果连接字的海明距离,选择最小海明距离的目标类为最终分类。经合成和真实数据的实验,验证了模型有效性及分类效果。     

Face verification via error correcting output codes

We propose a novel approach to face verification based on the Error Correcting Output Coding (ECOC) classifier design concept. In the training phase, the client set is repeatedly divided into two ECOC specified sub-sets (super-classes) to train a set of binary classifiers. The output of the classifiers defines the ECOC feature space, in which it is easier to separate transformed patterns representing clients and impostors. As a matching score in this space, we propose the average first order Minkowski distance between the probe and gallery images. The proposed method exhibits superior verification performance on the well known XM2VTS data set as compared with previously reported results.     

Subclass problem-dependent design for error-correcting output codes

A common way to model multi-class classification problems is by means of Error-Correcting Output Codes (ECOC). Given a multi-class problem, the ECOC technique designs a code word for each class, where each position of the code identifies the membership of the class for a given binary problem. A classification decision is obtained by assigning the label of the class with the closest code. One of the main requirements of the ECOC design is that the base classifier is capable of splitting each sub-group of classes from each binary problem. However, we can not guarantee that a linear classifier model convex regions. Furthermore, non-linear classifiers also fail to manage some type of surfaces. In this paper, we present a novel strategy to model multi-class classification problems using sub-class information in the ECOC framework. Complex problems are solved by splitting the original set of classes into sub-classes, and embedding the binary problems in a problem-dependent ECOC design. Experimental results show that the proposed splitting procedure yields a better performance when the class overlap or the distribution of the training objects conceil the decision boundaries for the base classifier. The results are even more significant when one has a sufficiently large training size.     

Discriminant error correcting output codes based on spectral clustering

Error correcting output codes (ECOCs) is a powerful framework to solve the multi-class problems. Finding the optimal partitions with maximum class discrimination efficiently is a key point to improve its performance. In this paper, we propose an alternative and efficient approach to obtain the partitions which are discriminative in the class space. The main idea of the proposed method is to transform the partition in the class space into the cut for an undirected graph using spectral clustering. In addition to measuring the class similarity, the confusion matrix with a pre-classifier is used. Our method is compared with the classical ECOC and DECOC over a synthetic dataset, a set of UCI machine learning repository datasets and one face recognition application. The results show that our proposal is able to obtain comparable or even better classification accuracy while reducing the computational complexity in comparison with the state-of-the-art coding methods.     

基于改进LPP和ECOC-SVMS的离线签名识别方法

提出一种基于改进LPP和ECOC-SVMS的离线签名识别方法。针对预处理后的签名图像,选择多种有效特征构建高维特征向量,引入一种改进的保局投影方法进行特征提取并同时实现高效降维;签名识别方面,使用基于Hadamard纠错编码方法的ECOC支持向量机多类分类方法,并引入近似概率对ECOC解码进行改进,以提升多类分类器的性能。实验结果表明此方法的可行性和有效性。     

基于纠错输出编码的人脸表情识别

多分类问题一直是模式识别领域的一个热点,提出了一种基于纠错输出编码和支持向量机的多分类器算法。根据通信编码理论设计纠错输出编码矩阵;按照该编码矩阵设计若干个互不相关的子支持向量机,根据编码原理将它们融合为一个多分类器。为了验证本分类器的有效性,采用Gabor小波提取人脸表情特征,应用二元主成分（2DPCA）分析法对提取的特征进行降维处理,应用该分类器进行了人脸表情的识别。实验结果表明,提出的方法能有效提高人脸表情的识别率,并具有极好的鲁棒性。     

Thinned-ECOC ensemble based on sequential code shrinking

Error-correcting output coding (ECOC) is a strategy to create classifier ensembles which reduces a multi-class problem into some binary sub-problems. A key issue in designing any ECOC classifier refers to defining optimal codematrix having maximum discrimination power and minimum number of columns. This paper proposes a heuristic method for application-dependent design of optimal ECOC matrix based on a thinning algorithm. The main idea of the proposed Thinned-ECOC method is to successively remove some redundant and unnecessary columns of any initial codematrix based on a metric defined for each column. As a result, computational cost of the ensemble is reduced while preserving its accuracy. Proposed method has been validated using the UCI machine learning database and further applied to a couple of real-world pattern recognition problems (the face recognition and gene expression based cancer classification). Experimental results emphasize the robustness of Thinned-ECOC in comparison with existing state-of-the-art code generation methods.     

Hierarchical classification method based on selective learning of slacked hierarchy for activity recognition systems

Physical activity recognition using wearable sensors has gained significant interest from researchers working in the field of ambient intelligence and human behavior analysis. The problem of multi-class classification is an important issue in the applications which naturally has more than two classes. A well-known strategy to convert a multi-class classification problem into binary sub-problems is the error-correcting output coding (ECOC) method. Since existing methods use a single classifier with ECOC without considering the dependency among multiple classifiers, it often fails to generalize the performance and parameters in a real-life application, where different numbers of devices, sensors and sampling rates are used. To address this problem, we propose a unique hierarchical classification model based on the combination of two base binary classifiers using selective learning of slacked hierarchy and integrating the training of binary classifiers into a unified objective function. Our method maps the multi-class classification problem to multi-level classification. A multi-tier voting scheme has been introduced to provide a final classification label at each level of the solicited model. The proposed method is evaluated on two publicly available datasets and compared with independent base classifiers. Furthermore, it has also been tested on real-life sensor readings for 3 different subjects to recognize four activities i.e. Walking, Standing, Jogging and Sitting. The presented method uses same hierarchical levels and parameters to achieve better performance on all three datasets having different number of devices, sensors and sampling rates. The average accuracies on publicly available dataset and real-life sensor readings were recorded to be 95% and 85%, respectively. The experimental results validate the effectiveness and generality of the proposed method in terms of performance and parameters.     

ECOC-DRF: Discriminative random fields based on error correcting output codes

We present ECOC-DRF, a framework where potential functions for Discriminative Random Fields are formulated as an ensemble of classifiers. We introduce the label trick, a technique to express transitions in the pairwise potential as meta-classes. This allows to independently learn any possible transition between labels without assuming any pre-defined model. The Error Correcting Output Codes matrix is used as ensemble framework for the combination of margin classifiers. We apply ECOC-DRF to a large set of classification problems, covering synthetic, natural and medical images for binary and multi-class cases, outperforming state-of-the art in almost all the experiments.     

Intelligent GPGPU Classification in Volume Visualization: A framework based on Error‐Correcting Output Codes

In volume visualization, the definition of the regions of interest is inherently an iterative trial‐and‐error process finding out the best parameters to classify and render the final image. Generally, the user requires a lot of expertise to analyze and edit these parameters through multi‐dimensional transfer functions. In this paper, we present a framework of intelligent methods to label on‐demand multiple regions of interest. These methods can be split into a two‐level GPU‐based labelling algorithm that computes in time of rendering a set of labelled structures using the Machine Learning Error‐Correcting Output Codes (ECOC) framework. In a pre‐processing step, ECOC trains a set of Adaboost binary classifiers from a reduced pre‐labelled data set. Then, at the testing stage, each classifier is independently applied on the features of a set of unlabelled samples and combined to perform multi‐class labelling. We also propose an alternative representation of these classifiers that allows to highly parallelize the testing stage. To exploit that parallelism we implemented the testing stage in GPU‐OpenCL. The empirical results on different data sets for several volume structures shows high computational performance and classification accuracy.     

Coding and decoding strategies for multi-class learning problems

It is known that the error correcting output code (ECOC) technique, when applied to multi-class learning problems, can improve generalisation performance. One reason for the improvement is its ability to decompose the original problem into complementary two-class problems. Binary classifiers trained on the sub-problems are diverse and can benefit from combining using a simple distance-based strategy. However there is some discussion about why ECOC performs as well as it does, particularly with respect to the significance of the coding/decoding strategy. In this paper we consider the binary (0,1) code matrix conditions necessary for reduction of error in the ECOC framework, and demonstrate the desirability of equidistant codes. It is shown that equidistant codes can be generated by using properties related to the number of 1’s in each row and between any pair of rows. Experimental results on synthetic data and a few popular benchmark problems show how performance deteriorates as code length is reduced for six decoding strategies.     

On the design of an ECOC-Compliant Genetic Algorithm

Genetic Algorithms (GA) have been previously applied to Error-Correcting Output Codes (ECOC) in state-of-the-art works in order to find a suitable coding matrix. Nevertheless, none of the presented techniques directly take into account the properties of the ECOC matrix. As a result the considered search space is unnecessarily large. In this paper, a novel Genetic strategy to optimize the ECOC coding step is presented. This novel strategy redefines the usual crossover and mutation operators in order to take into account the theoretical properties of the ECOC framework. Thus, it reduces the search space and lets the algorithm to converge faster. In addition, a novel operator that is able to enlarge the code in a smart way is introduced. The novel methodology is tested on several UCI datasets and four challenging computer vision problems. Furthermore, the analysis of the results done in terms of performance, code length and number of Support Vectors shows that the optimization process is able to find very efficient codes, in terms of the trade-off between classification performance and the number of classifiers. Finally, classification performance per dichotomizer results shows that the novel proposal is able to obtain similar or even better results while defining a more compact number of dichotomies and SVs compared to state-of-the-art approaches.     

Nearest neighbor classification from multiple feature subsets

Combining multiple classifiers is an effective technique for improving accuracy. There are many general combining algorithms, such as Bagging, Boosting, or Error Correcting Output Coding, that significantly improve classifiers like decision trees, rule learners, or neural networks. Unfortunately, these combining methods do not improve the nearest neighbor classifier. In this paper, we present MFS, a combining algorithm designed to improve the accuracy of the nearest neighbor (NN) classifier. MFS combines multiple NN classifiers each using only a random subset of features. The experimental results are encouraging: On 25 datasets from the UCI repository, MFS significantly outperformed several standard NN variants and was competitive with boosted decision trees. In additional experiments, we show that MFS is robust to irrelevant features, and is able to reduce both bias and variance components of error.     

Research on the unbiased probability estimation of error-correcting output coding

Supervised classification based on error-correcting output codes (ECOC) is an efficient method to solve the problem of multi-class classification, and how to get the accurate probability estimation via ECOC is also an attractive research direction. This paper proposed three kinds of ECOC to get unbiased probability estimates, and investigated the corresponding classification performance in depth at the same time. Two evaluating criterions for ECOC that has better classification performance were concluded, which are Bayes consistence and unbiasedness of probability estimation. Experimental results on artificial data sets and UCI data sets validate the correctness of our conclusion.     

Online error correcting output codes

This article proposes a general extension of the error correcting output codes framework to the online learning scenario. As a result, the final classifier handles the addition of new classes independently of the base classifier used. In particular, this extension supports the use of both online example incremental and batch classifiers as base learners. The extension of the traditional problem independent codings one-versus-all and one-versus-one is introduced. Furthermore, two new codings are proposed, unbalanced online ECOC and a problem dependent online ECOC. This last online coding technique takes advantage of the problem data for minimizing the number of dichotomizers used in the ECOC framework while preserving a high accuracy. These techniques are validated on an online setting of 11 data sets from UCI database and applied to two real machine vision applications: traffic sign recognition and face recognition. As a result, the online ECOC techniques proposed provide a feasible and robust way for handling new classes using any base classifier.     

Applying error correction codes to achieve security and dependability

We apply Linear Error Correction (LEC) code to a novel encoding scheme to assure two fundamental requirements for transmission channels and storage units: security and dependability. Our design has the capacity to adapt itself to different applications and their various characteristics such as availability, error rate, and vulnerabilities. Based on simple logic operations, our scheme affords fast encryption, scalability (dual or more column erasures), and flexibility (LEC encoder employed as a front end to any conventional compression scheme). Performance results are very promising: Experiments on dual erasures outperform conventional compression algorithms including Arithmetic Coding, Huffman, and LZ77.     

Traffic sign recognition system with β -correction

Traffic sign classification represents a classical application of multi-object recognition processing in uncontrolled adverse environments. Lack of visibility, illumination changes, and partial occlusions are just a few problems. In this paper, we introduce a novel system for multi-class classification of traffic signs based on error correcting output codes (ECOC). ECOC is based on an ensemble of binary classifiers that are trained on bi-partition of classes. We classify a wide set of traffic signs types using robust error correcting codings. Moreover, we introduce the novel β-correction decoding strategy that outperforms the state-of-the-art decoding techniques, classifying a high number of classes with great success.