Bayesian nonlinear model selection and neural networks: a conjugateprior approach

In order to select the best predictive neural-network architecture in a set of several candidate networks, we propose a general Bayesian nonlinear regression model comparison procedure, based on the maximization of an expected utility criterion. This criterion selects the model under which the training set achieves the highest level of internal consistency, through the predictive probability distribution of each model. The density of this distribution is computed as the model posterior predictive density and is asymptotically approximated from the assumed Gaussian likelihood of the data set and the related conjugate prior density of the parameters. The use of such a conjugate prior allows the analytic calculation of the parameter posterior and predictive posterior densities, in an empirical Bayes-like approach. This Bayesian selection procedure allows us to compare general nonlinear regression models and in particular feedforward neural networks, in addition to embedded models as usual with asymptotic comparison tests.     

Bayes网络学习的MCMC方法

基于Bayes统计理论, 提出了一种从数据样本中学习Bayes网络的Markov链Monte Carlo(MCMC)方法. 首先通过先验概率和数据样本的结合得到未归一化的后验概率, 然后使用此后验概率指导随机搜索算法寻找“好”的网络结构模型. 通过对Alarm网络的学习表明了本算法具有较好的性能.     

Bayesian network model of overall print quality: Construction and structural optimisation

Prediction of overall visual quality based on instrumental measurements is a challenging task. Despite the several proposed models and methods, there exists a gap between the instrumental measurements of print and human visual assessment of natural images. In this work, a computational model for representing and quantifying the overall visual quality of prints is proposed. The computed overall quality should correspond to the human visual quality perception when viewing the printed images. The proposed model is a Bayesian network which connects the objective instrumental measurements to the subjective opinion distribution of human observers. This relationship can be used to score printed images, and additionally, to computationally study the connections of the attributes. A novel graphical learning approach using an iterative evolve-estimate-simulate loop learning the quality model based on psychometric data and instrumental measurements is suggested. The network structure is optimised by applying evolutionary computation (evolve). The estimation of the Bayesian network parameters is within the evolutionary loop. In this loop, the maximum likelihood approach is used (estimate). The stochastic learning process is guided by priors devised from the psychometric subjective experiments (performance through simulation). The model reveals and represents the explanatory factors between its elements providing insight to the psychophysical phenomenon of how observers perceive visual quality and which measurable entities affect the quality perception. By using true data, the design choices are demonstrated. It is also shown that the best-performing network establishes a clear and intuitively correct structure between the objective measurements and psychometric data.     

基于D-vine Copula理论的贝叶斯分类器设计

高斯判别分析、朴素贝叶斯等传统贝叶斯分类方法在构建变量的联合概率分布时,往往会对变量间的相关性进行简化处理,从而使得贝叶斯决策理论中类条件概率密度的估计与实际数据之间存在一定的偏差.对此,结合Copula函数研究特征变量之间的相关性优化问题,设计基于D-vine Copula理论的贝叶斯分类器,主要目的是为了提高类条件概率密度估计的准确性.将变量的联合概率分布分解为一系列二元Copula函数与边缘概率密度函数的乘积,采用核函数方法对边缘概率密度进行估计 ,通过极大似然估计对二元Copula函数的参数分别进行优化,进而得到类条件概率密度函数的形式.将基于D-vine Copula理论的贝叶斯分类器应用到生物电信号的分类问题上,并对分类效果进行分析和验证.结果表明,所提出的方法在各项分类指标上均具备良好的性能.     

Bayesian system identification

In Bayesian statistics the concept of probability is interpreted as a rational measure of belief which is used to describe mathematically the uncertain relation between the statistician and the external world. The statistical inference is understood as a correction of prior subjective probability distribution by objective data. The paper shows that on this Bayesian basis it is possible to build a consistent theory of system identification. The following problems are considered: one-shot and real-time identification, estimation and prediction in closed control loop, redundant and unidentifiable parameters, time-varying parameters and adaptivity.     

基于共轭先验分布的贝叶斯网络分类模型

针对贝叶斯网络后验概率需计算样本边际分布,计算代价大的问题,将共轭先验分布思想引入贝叶斯分类,提出了基于共轭先验分布的贝叶斯网络分类模型.针对非区间离散样本,提出一种自适应的样本离散方法,将小波包提取模拟电路故障特征离散化作为分类模型属性.仿真验证表明,模型分类效果较好,算法运行速度得以提高,也可应用于连续样本和多分类的情况,扩展了贝叶斯网络分类的应用范围.     

基于混合概率模型的无监督离散化算法

现实应用中常常涉及许多连续的数值属性，而且前许多机器学习算法则要求所处理的属性取离散值，根据在对数值属性的离散化过程中，是否考虑相关类别属性的值，离散化算法可分为有监督算法和无监督算法两类。基于混合概率模型，该文提出了一种理论严格的无监督离散化算法，它能够在无先验知识，无类别是属性的前提下，将数值属性的值域划分为若干子区间，再通过贝叶斯信息准则自动地寻求最佳的子区间数目和区间划分方法。     

Parameter learning in hybrid Bayesian networks using prior knowledge

Mixtures of truncated basis functions have been recently proposed as a generalisation of mixtures of truncated exponentials and mixtures of polynomials for modelling univariate and conditional distributions in hybrid Bayesian networks. In this paper we analyse the problem of learning the parameters of marginal and conditional MoTBF densities when both prior knowledge and data are available. Incorporating prior knowledge provide a valuable tool for obtaining useful models, especially in domains of applications where data are costly or scarce, and prior knowledge is available from practitioners. We explore scenarios where the prior knowledge can be expressed as an MoTBF density that is afterwards combined with another MoTBF density estimated from the available data. The resulting model remains within the MoTBF class which is a convenient property from the point of view of inference in hybrid Bayesian networks. The performance of the proposed method is tested in a series of experiments carried out over synthetic and real data.     

Prediction of survival probabilities with Bayesian Decision Trees

Practitioners use Trauma and Injury Severity Score (TRISS) models for predicting the survival probability of an injured patient. The accuracy of TRISS predictions is acceptable for patients with up to three typical injuries, but unacceptable for patients with a larger number of injuries or with atypical injuries. Based on a regression model, the TRISS methodology does not provide the predictive density required for accurate assessment of risk. Moreover, the regression model is difficult to interpret. We therefore consider Bayesian inference for estimating the predictive distribution of survival. The inference is based on decision tree models which recursively split data along explanatory variables, and so practitioners can understand these models. We propose the Bayesian method for estimating the predictive density and show that it outperforms the TRISS method in terms of both goodness-of-fit and classification accuracy. The developed method has been made available for evaluation purposes as a stand-alone application.     

Heterogeneous data analysis using a mixture of Laplace models with conjugate priors

The development of flexible parametric classes of probability models in Bayesian analysis is a very popular approach. This study is designed for heterogeneous population for a two-component mixture of the Laplace probability distribution. When a process initially starts, the researcher expects that the failure components will be very high but after some improvement/inspection it is assumed that the failure components will decrease sufficiently. That is why in such situation the Laplace model is more suitable as compared to the normal distribution due to its fatter tails behaviour. We considered the derivation of the posterior distribution for censored data assuming different conjugate informative priors. Various kinds of loss functions are used to derive these Bayes estimators and their posterior risks. A method of elicitation of hyperparameter is discussed based on a prior predictive approach. The results are also compared with the non-informative priors. To examine the performance of these estimators we have evaluated their properties for different sample sizes, censoring rates and proportions of the component of the mixture through the simulation study. To highlight the practical significance we have included an illustrative application example based on real-life mixture data.     

Bayesian estimation of beta mixture models with variational inference

Bayesian estimation of the parameters in beta mixture models (BMM) is analytically intractable. The numerical solutions to simulate the posterior distribution are available, but incur high computational cost. In this paper, we introduce an approximation to the prior/posterior distribution of the parameters in the beta distribution and propose an analytically tractable (closed form) Bayesian approach to the parameter estimation. The approach is based on the variational inference (VI) framework. Following the principles of the VI framework and utilizing the relative convexity bound, the extended factorized approximation method is applied to approximate the distribution of the parameters in BMM. In a fully Bayesian model where all of the parameters of the BMM are considered as variables and assigned proper distributions, our approach can asymptotically find the optimal estimate of the parameters posterior distribution. Also, the model complexity can be determined based on the data. The closed-form solution is proposed so that no iterative numerical calculation is required. Meanwhile, our approach avoids the drawback of overfitting in the conventional expectation maximization algorithm. The good performance of this approach is verified by experiments with both synthetic and real data.     

Efficient reliability analysis based on Bayesian framework under input variable and metamodel uncertainties

In the reliability analysis, input variables as well as the metamodel uncertainties are often encountered in practice. The input uncertainty includes the statistical uncertainty of the distribution parameters due to the lack of knowledge or insufficient data. Metamodel uncertainty arises when the response function is approximated by a surrogate function using a finite number of responses to reduce the costly computations. In this study, a reliability analysis procedure is proposed based on a Bayesian framework that can incorporate these uncertainties in an integrated manner into the form of posterior PDF. The PDF, often expressed by arbitrary functions, is evaluated via Markov Chain Monte Carlo (MCMC) method, which is an efficient simulation method to draw random samples that follow the distribution. In order to avoid the nested computation in the full Bayesian approach, a posterior predictive approach is employed, which requires only a single loop of reliability analysis. Gaussian process model is employed for the metamodel. Mathematical and engineering examples are used to demonstrate the proposed method. In the results, comparing with the full Bayesian approach, the predictive approach provides much less information, i.e., only a point estimate of the probability. Nevertheless, the predictive approach adequately accounts for the uncertainties with much less computation, which is more advantageous in the design practice. The smaller the data are provided, the higher the statistical uncertainty, leading to the higher (or lower) failure probability (or reliability).     

基于BOA-ELM的混凝土抗压强度预测研究

为控制控制混凝土生产成本,在混凝土拌和期限制抗压强度不足的缺陷构建产出,可以有效降低原料的浪费,是节能降耗的关键方法之一。针对混凝土抗压强度的传统测量方法严重滞后的问题,提出了基于贝叶斯优化极限学习机(BOA-ELM)的混凝土抗压强度预测方法。首先,分析了混凝土拌和过程中对抗压强度预测值实时获得的需求。以各物料的用量为分析基础,28天标准养护后混凝土抗压强度值为预测目标,设计了基于极限学习机的强度预测模型。其次,为进一步提高模型的稳定性以及准确行,提出基于贝叶斯优化的极限学习机模型,根据模型超参数的分布特征,以高斯过程作为超参的先验分布,预测误差最小化作为目标,寻找最优的模型超参。最后,在实际施工产生的C50标号混凝土数据集上测试文中模型,并对比分析了其他预测模型和寻优算法。结果表明,结合了贝叶斯优化的极限学习机预测模型相较于经典算法具有更高的预测准确性和模型训练的高效性。     

Bayesian multioutput feedforward neural networks comparison: a conjugate prior approach

A Bayesian method for the comparison and selection of multioutput feedforward neural network topology, based on the predictive capability, is proposed. As a measure of the prediction fitness potential, an expected utility criterion is considered which is consistently estimated by a sample-reuse computation. As opposed to classic point-prediction-based cross-validation methods, this expected utility is defined from the logarithmic score of the neural model predictive probability density. It is shown how the advocated choice of a conjugate probability distribution as prior for the parameters of a competing network, allows a consistent approximation of the network posterior predictive density. A comparison of the performances of the proposed method with the performances of usual selection procedures based on classic cross-validation and information-theoretic criteria, is performed first on a simulated case study, and then on a well known food analysis dataset.     

基于概率模型的动态分层强化学习

为解决大规模强化学习中的＂维度灾难＂问题,克服以往学习算法的性能高度依赖于先验知识的局限性,本文提出一种基于概率模型的动态分层强化学习方法.首先基于贝叶斯学习对状态转移概率进行建模,建立基于概率参数的关键状态识别方法,进而通过聚类动态生成若干状态子空间和学习分层结构下的最优策略.仿真结果表明该算法能显著提高复杂环境下智能体的学习效率,适用于未知环境中的大规模学习.     

航班备降概率分布预测模型研究

对航班备降问题的小样本特点进行了分析,提出了基于观察学习的航班备降概率分布预测模型。该模型利用松弛属性约束思想抽取数据子集,三次样条插值方法构建基学习器,并结合虚拟数据生成策略促使各基学习器达成一致。并在此基础上,对信任度参数进行优化,进一步完善了预测模型。在航班备降数据集的实验表明,在大样本下,该预测模型的预测精度高于朴素贝叶斯方法和贝叶斯网方法;在小样本数据集上分析了航班不同备降次数下的置信度,为相关部门提供决策支持。     

Learning Bayesian network parameters under incomplete data with domain knowledge

Bayesian networks (BNs) have gained increasing attention in recent years. One key issue in Bayesian networks is parameter learning. When training data is incomplete or sparse or when multiple hidden nodes exist, learning parameters in Bayesian networks becomes extremely difficult. Under these circumstances, the learning algorithms are required to operate in a high-dimensional search space and they could easily get trapped among copious local maxima. This paper presents a learning algorithm to incorporate domain knowledge into the learning to regularize the otherwise ill-posed problem, to limit the search space, and to avoid local optima. Unlike the conventional approaches that typically exploit the quantitative domain knowledge such as prior probability distribution, our method systematically incorporates qualitative constraints on some of the parameters into the learning process. Specifically, the problem is formulated as a constrained optimization problem, where an objective function is defined as a combination of the likelihood function and penalty functions constructed from the qualitative domain knowledge. Then, a gradient-descent procedure is systematically integrated with the E-step and M-step of the EM algorithm, to estimate the parameters iteratively until it converges. The experiments with both synthetic data and real data for facial action recognition show our algorithm improves the accuracy of the learned BN parameters significantly over the conventional EM algorithm.     

基于贝叶斯方法的神经网络非线性模型辨识

研究了基于贝叶斯推理的多层前向神经网络训练算法,以提高网络的泛化性能。在网络目标函数中引入表示网络结构复杂性的惩罚项,以便能够在训练优化过程中降低网络结构的复杂性,达到避免网络过拟合的目的。训练过程中使用显式的概率分布假设对模型进行分析和推断,根据融入先验分布的假设和依据,获取网络参数和正则化参数的后验条件概率,并基于后验分布的贝叶斯推理得出最优化参数。利用上述算法训练前向网络,对一个微型锅炉对象进行了模型辨识,通过测试,证明所辨识出的对象模型能够较好地表现出对象的动态行为,且具有较好的泛化性能。     

STATISTICAL INFERENCES BASED ON A SECOND-ORDER POSSIBILITY DISTRIBUTION

Abstract

A new, general method of statistical inference is proposed. It encompasses all the coherent forms of statistical inference that can be derived from a Bayesian prior distribution, Bayesian sensitivity analysis or upper and lower prior probabilities. The method is to model prior uncertainty about statistical parameters in terms of a second-order possibility distribution (a special type of upper probability) which measures the plausibility of each conceivable prior probability distribution. This defines an imprecise hierarchical model. Two,applications are studied: the problem of robustifying Bayesian analyses by forming a neighbourhood of a Bayesian prior distribution, and the problem of combining prior opinions from different sources.