Accurate estimates of the generalization ability for symmetric set of predictors and randomized learning algorithms

The main issue of the combinatorial approach to overfitting is to obtain computationally efficient formulas for overfitting probabilities. A group-theoretical approach is proposed to simplify derivation of such formulas when the set of predictors has a certain group of symmetries. Examples of the sets are given. The general estimate of overfitting probability is proved for the randomized learning algorithm. It is applied to four model sets of predictors—a layer of the Boolean cube, the Boolean cube, the unimodal chain, and a bundle of monotonic chains.     

Splitting and similarity phenomena in the sets of classifiers and their effect on the probability of overfitting

It is shown that computationally tight bounds for the probability of overfitting can be obtained only by simultaneous consideration of the following two properties of classifier sets: splitting into error levels and similarity of classifiers. For a set consisting of only two classifiers, an exact bound is obtained for the probability of overfitting. This is the simplest learning task that exhibits overfitting and the effects of splitting and similarity, which reduce the probability of overfitting. For a more complex case—a chain of classifiers—an experiment is carried out in which the effects of splitting and similarity are estimated separately. It is shown that reasonably low probabilities of overfitting can be obtained only for the sets that possess both properties.     

A neural network method of density estimation for univariate unimodal data

In real world statistical applications a large class of univariate probability distributions are unimodal. This paper presents a neural network approach to probability density estimation for univariate unimodal data. This approach is superior to the traditional statistical density estimation methods in that neither of the forms of probability function (as in the parametric methods) or the kernel functions (as in the non-parametric methods) is needed to form an a prioriassumption. An example is used to demonstrate that the proposed approach is an effective method for estimating univariate unimodal probability distributions.     

Exact estimates of the probability of overfitting for multidimensional modeling families of algorithms

Exact estimates of the probability of overfitting are obtained for three modeling families of algorithms having properties of demixing, connectivity, and dimension. For obtaining the given estimates, a generalization of the method of generating and inhibitory sets is supposed and a reduction formalism is developed. The correctness of the estimates obtained is verified experimentally using a synthetic sampling.     

缓存敏感的封闭冰山立方体计算

数据立方体计算通常会产生大量的输出结果,冰山立方体和封闭立方体是解决这个问题的比较流行的两种策略,二者可以结合使用.鉴于封闭冰山立方体(closed iceberg cube)的重要性和实用性,如何高效地计算封闭冰山立方体是一个值得研究的问题.提出一种缓存敏感(cache-conscious)的计算封闭冰山立方体的方法,在自底向上对数据进行聚集的同时,寻找覆盖聚集单元的封闭单元,将其输出,使用两种策略进行剪枝,去掉不必要的递归,同时使用Apriori剪枝技术,支持冰山立方体(iceberg cube)的计算.为了减少与内存相关的延迟,快速得到聚集结果,对多个维进行预排序,并将软件预取技术引入到数据扫描中.在模拟数据和真实数据上进行了详细而全面的实验研究,结果表明,封闭冰山立方体的计算方法是快速、有效的.     

关于Trivium-型序列密码代数次数估计的研究

对于序列密码,输出密钥流比特可以视为关于密钥变元和Ⅳ变元的布尔函数,而该布尔函数的代数次数是影响密码算法安全性的重要因素;当代数次数偏低时,密码算法抵抗代数攻击、立方攻击和积分攻击的能力比较弱.目前,针对Trivium-型序列密码算法,最有效的代数次数估计方法是数值映射方法和基于MILP的可分性质方法.本文通过分析两种典型方法的特点,结合两种方法的优势,对Trivium-型算法的代数次数估计进行了改进.我们利用改进后的方法对大量随机选取的Ⅳ变量集进行了实验.实验结果表明,对于Trivium-型算法,改进后的方法能够给出比数值映射方法更紧的代数次数上界.特别地,针对Trivium算法,当输入变元为全密钥变元和全Ⅳ变元时,即80个密钥变元和80个Ⅳ变元,输出比特代数次数未达到160的最大轮数从907轮提高到912轮,这是目前已知的全变元情形下的最优代数次数估计结果.     

Uncertainty analysis of rule-based expert systems with Dempster-Shafer mass assignments

This article extends Dempster-Shafer Theory (DST) mass probability assignments to Boolean algebra and considers how such probabilities can propagate through a system of Boolean equations, which form the basis for both rule-based expert systems and fault trees. the advantage of DST mass assignments over classical probability methods is the ability to accommodate when necessary uncommitted probability belief. This paper also examines rules in the context of a probabilistic logic, where a given rule itself may be true with some probability in the interval [0,1]. When expert system knowledge bases contain rules which may not always hold, or rules that occasionally must be operated upon with imprecise information, the DST mass assignment formalism is shown to be a suitable methodology for calculating probability assignments throughout the system.     

一种新型逻辑函数化简方法——立体化简法

文章在卡诺图化简法的思想基础上设计了一种新型的逻辑函数化简方法——“立体化简法”。用逻辑函数立方体代替卡诺图来表示逻辑函数,在三维立体空间进行逻辑函数的化简,既保持了卡诺图化简法方便、直观、容易掌握的优点,又使得可以方便化简的逻辑函数变量增加至六个;如果采用达到卡诺图化简法五、六变量逻辑函数化简的难易程度的方法,可使化简的逻辑函数变量增加至九个。这种新型的逻辑化简方法使得五、六变量逻辑函数的化简变得非常简单、方便,也使得九变量以内的逻辑函数的化简变得直观、可行。     

一种求解混合约束问题的快速完备算法

布尔与数值变量相混合的约束问题有着广泛盼应用,但是当约束中的数值变量间存在非线性关系时该问题求解起来十分困难．目前的许多求解方法都是不完备的,即这些方法不能完全肯定某些包含非线性数值表达式的约束是否能够成立．针对这种问题,提出了数值与区间分析相结合进行数值约束求解的方法．已经实现了一个基于此方法的原型工具．实验结果表明。该方法能够有效、快速、完备地求解非线性混合约束问题．     

基于Dropout的改进卷积神经网络模型平均方法

针对深度卷积神经网络（CNN）中的过拟合问题，提出一种基于Dropout改进CNN的模型预测平均方法。首先，训练阶段在池化层引入Dropout，使得池化层单元值具有稀疏性；然后，在测试阶段将训练时池化层Dropout选择单元值的概率与池化区域各单元值所占概率相乘作为双重概率；最后，将提出的双重概率加权的模型平均方法应用于测试阶段，使得训练阶段池化层Dropout的稀疏效果能够更好地反映到测试阶段池化层上，从而使测试错误率达到与训练的较低错误率相近的结果。在给定大小的网络中所提方法在MNIST和CIFAR-10数据集上的测试错误率分别为0.31%和11.23%。实验结果表明：仅考虑池化层对结果的影响，所提方法与Prob.weighted pooling和Stochastic Pooling方法相比具有更低的错误率，表明池化层Dropout使得模型更具泛化性，并且池化单元值对于模型泛化具有一定帮助，能够更有效避免过拟合。     

A systematic review on overfitting control in shallow and deep neural networks

Shallow neural networks process the features directly, while deep networks extract features automatically along with the training. Both models suffer from overfitting or poor generalization in many cases. Deep networks include more hyper-parameters than shallow ones that increase the overfitting probability. This paper states a systematic review of the overfit controlling methods and categorizes them into passive, active, and semi-active subsets. A passive method designs a neural network before training, while an active method adapts a neural network along with the training process. A semi-active method redesigns a neural network when the training performance is poor. This review includes the theoretical and experimental backgrounds of these methods, their strengths and weaknesses, and the emerging techniques for overfitting detection. The adaptation of model complexity to the data complexity is another point in this review. The relation between overfitting control, regularization, network compression, and network simplification is also stated. The paper ends with some concluding lessons from the literature.

     

Local and interweight spectra of completely regular codes and of perfect colorings

We introduce notions of local and interweight spectra of an arbitrary coloring of a Boolean cube, which generalize the notion of a weight spectrum. The main objects of our research are colorings that are called perfect. We establish an interrelation of local spectra of such a coloring in two orthogonal faces of a Boolean cube and study properties of the interweight spectrum. Based on this, we prove a new metric property of perfect colorings, namely, their strong distance invariance. As a consequence, we obtain an analogous property of an arbitrary completely regular code, which, together with his neighborhoods, forms a perfect coloring.     

Binary feature fusion for discriminative and secure multi-biometric cryptosystems

Biometric cryptosystem has been proven to be a promising approach for template protection. Cryptosystems such as fuzzy extractor and fuzzy commitment require discriminative and informative binary biometric input to offer accurate and secure recognition. In multi-modal biometric recognition, binary features can be produced via fusing the real-valued unimodal features and binarizing the fused features. However, when the extracted features of certain modality are represented in binary and the extraction parameters are not known, real-valued features of other modalities need to be binarized and the feature fusion needs to be carried out at the binary level. In this paper, we propose a binary feature fusion method that extracts a set of fused binary features with high discriminability (small intra-user and large inter-user variations) and entropy (weak dependency among bits and high bit uniformity) from multiple sets of binary unimodal features. Unlike existing fusion methods that mainly focus on discriminability, the proposed method focuses on both feature discriminability and system security: The proposed method 1) extracts a set of weakly dependent feature groups from the multiple unimodal features; and 2) fuses each group to a bit using a mapping that minimizes the intra-user variations and maximizes the inter-user variations and uniformity of the fused bit. Experimental results on three multi-modal databases show that fused binary feature of the proposed method has both higher discriminability and higher entropy compared to the unimodal features and the fused features generated from the state-of-the-art binary fusion approaches.     

A partial order for the M-of-N rule-extraction algorithm.

We present a method to unify the rules obtained by the M-of-N rule-extraction technique. The rules extracted from a perceptron by the M-of-N algorithm are in correspondence with sets of minimal Boolean vectors with respect to the classical partial order defined on vectors. Our method relies on a simple characterization of another partial order defined on Boolean vectors. We show that there exists also a correspondence between sets of minimal Boolean vectors with respect to this order and M-of-N rules equivalent to a perceptron. The gain is that fewer rules are generated with the second order. Independently, we prove that deciding whether a perceptron is symmetric with respect to two variables is NP-complete.     

优势关系下基于浓缩布尔矩阵的属性约简方法

在优势关系粗糙集方法(DRSA)的框架下,针对不协调的目标信息系统求属性约简。基于优势矩阵的方法是最常用的一类约简方法,但矩阵中不是所有的元素都有效。浓缩优势矩阵只保留对求约简有用的最小属性集,因而可以明显降低约简过程中的计算量。进一步地,浓缩布尔矩阵通过布尔代数的形式有效地弥补了优势矩阵生成效率低的缺点。文中将等价关系上的浓缩布尔矩阵属性约简方法扩展到优势关系上,针对优势矩阵提出了浓缩布尔矩阵的概念,建立了相应的高效约简方法,使效率得到明显提高。最后采用9组UCI数据进行实验,结果验证了所提方法的有效性。     

Efficient Venn predictors using random forests

Successful use of probabilistic classification requires well-calibrated probability estimates, i.e., the predicted class probabilities must correspond to the true probabilities. In addition, a probabilistic classifier must, of course, also be as accurate as possible. In this paper, Venn predictors, and its special case Venn-Abers predictors, are evaluated for probabilistic classification, using random forests as the underlying models. Venn predictors output multiple probabilities for each label, i.e., the predicted label is associated with a probability interval. Since all Venn predictors are valid in the long run, the size of the probability intervals is very important, with tighter intervals being more informative. The standard solution when calibrating a classifier is to employ an additional step, transforming the outputs from a classifier into probability estimates, using a labeled data set not employed for training of the models. For random forests, and other bagged ensembles, it is, however, possible to use the out-of-bag instances for calibration, making all training data available for both model learning and calibration. This procedure has previously been successfully applied to conformal prediction, but was here evaluated for the first time for Venn predictors. The empirical investigation, using 22 publicly available data sets, showed that all four versions of the Venn predictors were better calibrated than both the raw estimates from the random forest, and the standard techniques Platt scaling and isotonic regression. Regarding both informativeness and accuracy, the standard Venn predictor calibrated on out-of-bag instances was the best setup evaluated. Most importantly, calibrating on out-of-bag instances, instead of using a separate calibration set, resulted in tighter intervals and more accurate models on every data set, for both the Venn predictors and the Venn-Abers predictors.

     

用平滑方法改进多关系朴素贝叶斯分类

为消除朴素贝叶斯分类时的零概率以及过度拟合问题,分析了各种概率平滑方法,给出了基于M估计的多关系朴素贝叶斯分类方法（MRNBC-M）和基于Laplace估计的多关系朴素贝叶斯分类方法（MRNBC-L）,分析探讨了M平滑和Laplace平滑方法对多关系分类的影响情况,为进一步优化分类,方法基于扩展互信息标准对数据进行属性过滤。多关系标准数据集上的实验显示,MRNBC-M可以有效改进分类性能。     

基于BDD的关联故障树定量分析法研究

利用二元决策图(BDD)中的B—割集和节点概率,提出了基于BDD的关联故障树定量分析新算法。推导了故障树顶事件发生概率和绝对失效强度算法公式,将故障树顶事件发生概率和绝对失效强度计算结合起来,避免了复杂的最小割集和不交化求解过程,显著地减少了布尔代数运算量,给出了算法递归步骤。大量的实例分析验证了此算法的工程实用性。     

基于先验概率的性能降级链路推断方法

研究基于布尔网络断层扫描技术辨识网络中性能出现严重降级的链路。将辨识问题归结为最大概率解释问题,提出一种利用链路状态先验概率分布的推断性能严重降级链路的方法。通过计算每种候选链路指派的后验概率,从中选取具有最大概率的指派作为推断结果。理论分析证明了该方法的可行性,仿真实验验证了该方法的有效性和准确性。