SimilarityExplorer: A Visual Inter‐Comparison Tool for Multifaceted Climate Data

Inter‐comparison and similarity analysis to gauge consensus among multiple simulation models is a critical visualization problem for understanding climate change patterns. Climate models, specifically, Terrestrial Biosphere Models (TBM) represent time and space variable ecosystem processes, like, simulations of photosynthesis and respiration, using algorithms and driving variables such as climate and land use. While it is widely accepted that interactive visualization can enable scientists to better explore model similarity from different perspectives and different granularity of space and time, currently there is a lack of such visualization tools. In this paper we present three main contributions. First, we propose a domain characterization for the TBM community by systematically defining the domain‐specific intents for analyzing model similarity and characterizing the different facets of the data. Second, we define a classification scheme for combining visualization tasks and multiple facets of climate model data in one integrated framework, which can be leveraged for translating the tasks into the visualization design. Finally, we present SimilarityExplorer, an exploratory visualization tool that facilitates similarity comparison tasks across both space and time through a set of coordinated multiple views. We present two case studies from three climate scientists, who used our tool for a month for gaining scientific insights into model similarity. Their experience and results validate the effectiveness of our tool.     

HyperMoVal: Interactive Visual Validation of Regression Models for Real‐Time Simulation

During the development of car engines, regression models that are based on machine learning techniques are increasingly important for tasks which require a prediction of results in real‐time. While the validation of a model is a key part of its identification process, existing computation‐ or visualization‐based techniques do not adequately support all aspects of model validation. The main contribution of this paper is an interactive approach called HyperMoVal that is designed to support multiple tasks related to model validation: 1) comparing known and predicted results, 2) analyzing regions with a bad fit, 3) assessing the physical plausibility of models also outside regions covered by validation data, and 4) comparing multiple models. The key idea is to visually relate one or more n‐dimensional scalar functions to known validation data within a combined visualization. HyperMoVal lays out multiple 2D and 3D sub‐projections of the n‐dimensional function space around a focal point. We describe how linking HyperMoVal to other views further extends the possibilities for model validation. Based on this integration, we discuss steps towards supporting the entire workflow of identifying regression models. An evaluation illustrates a typical workflow in the application context of car‐engine design and reports general feedback of domain experts and users of our approach. These results indicate that our approach significantly accelerates the identification of regression models and increases the confidence in the overall engineering process.     

基于样本个体差异性的深度神经网络训练方法

深度神经网络目前在许多任务中的表现已经达到甚至超越了人类的水平,但是其泛化能力和人类相比还是相去甚远.如何提高网络的泛化性,一直是重要的研究方向之一.围绕这个方向开展的大量卓有成效的研究,从扩展增强训练数据、通过正则化抑制模型复杂度、优化训练策略等角度,提出了很多行之有效的方法.这些方法对于训练数据集来说都是某种全局性质的策略,每一个样本数据都会被平等的对待.但是,每一个样本数据由于其携带的信息量、噪声等的不同,在训练过程中,对模型的拟合性能和泛化性能的影响也应该是有差异性的.针对是否一些样本在反复的迭代训练中更倾向于使得模型过度拟合,如何找到这些样本,是否可以通过对不同的样本采用差异化的抗过拟合策略使得模型获得更好的泛化性能等问题,提出了一种依据样本数据的差异性来训练深度神经网络的方法,首先使用预训练模型对每一个训练样本进行评估,判断每个样本对该模型的拟合效果;然后依据评估结果将训练集分为易使得模型过拟合的样本和普通的样本两个子集;最后,再使用两个子集的数据对模型进行交替训练,过程中对易使得模型过拟合的子集采用更强有力的抗过拟合策略.通过在不同的数据集上对多种深度模型进行的一系列实验...     

低秩特征选择多输出回归算法

针对现有回归算法没有考虑利用特征与输出的关系,各输出之间的关系,以及样本之间的关系来处理高维数据的多输出回归问题易输出不稳定的模型,提出一种新的低秩特征选择多输出回归方法。该方法采用低秩约束去构建低秩回归模型来获取多输出变量之间的关联结构;同时创新地在该低秩回归模型上使用[L2,p]-范数来进行样本选择,合理地去除噪音和离群点的干扰;并且使用[L2,p]-范数正则化项惩罚回归系数矩阵进行特征选择,有效地处理特征与输出的关系和避免“维灾难”的影响。通过实际数据集的实验结果表明,提出的方法在处理高维数据的多输出回归分析中能获得非常好的效果。     

Learning from Batched Data: Model Combination Versus Data Combination

Combining models learned from multiple batches of data provide an alternative to the common practice of learning one model from all the available data (i.e. the data combination approach). This paper empirically examines the base-line behavior of the model combination approach in this multiple-data-batches scenario. We find that model combination can lead to better performance even if the disjoint batches of data are drawn randomly from a larger sample, and relate the relative performance of the two approaches to the learning curve of the classifier used. In the beginning of the curve, model combination has higher bias and variance than data combination and thus a higher error rate. As training data increases, model combination has either a lower error rate than or a comparable performance to data combination because the former achieves larger variance reduction. We also show that this result is not sensitive to the methods of model combination employed. Another interesting result is that we empirically show that the near-asymptotic performance of a single model in some classification tasks can be significantly improved by combining multiple models (derived from the same algorithm) in the multiple-data-batches scenario.     

Structured feature selection and task relationship inference for multi-task learning

Multi-task learning (MTL) aims to enhance the generalization performance of supervised regression or classification by learning multiple related tasks simultaneously. In this paper, we aim to extend the current MTL techniques to high dimensional data sets with structured input and structured output (SISO), where the SI means the input features are structured and the SO means the tasks are structured. We investigate a completely ignored problem in MTL with SISO data: the interplay of structured feature selection and task relationship modeling. We hypothesize that combining the structure information of features and task relationship inference enables us to build more accurate MTL models. Based on the hypothesis, we have designed an efficient learning algorithm, in which we utilize a task covariance matrix related to the model parameters to capture the task relationship. In addition, we design a regularization formulation for incorporating the structured input features in MTL. We have developed an efficient iterative optimization algorithm to solve the corresponding optimization problem. Our algorithm is based on the accelerated first order gradient method in conjunction with the projected gradient scheme. Using two real-world data sets, we demonstrate the utility of the proposed learning methods.     

Meta-learning meets the Internet of Things: Graph prototypical models for sensor-based human activity recognition

With the rapid growth of the Internet of Things (IoT), smart systems and applications are equipped with an increasing number of wearable sensors and mobile devices. These sensors are used not only to collect data but, more importantly, to assist in tracking and analyzing the daily human activities. Sensor-based human activity recognition is a hotspot and starts to employ deep learning approaches to supersede traditional shallow learning that rely on hand-crafted features. Although many successful methods have been proposed, there are three challenges to overcome: (1) deep model’s performance overly depends on the data size; (2) deep model cannot explicitly capture abundant sample distribution characteristics; (3) deep model cannot jointly consider sample features, sample distribution characteristics, and the relationship between the two. To address these issues, we propose a meta-learning-based graph prototypical model with priority attention mechanism for sensor-based human activity recognition. This approach learns not only sample features and sample distribution characteristics via meta-learning-based graph prototypical model, but also the embeddings derived from priority attention mechanism that mines and utilizes relations between sample features and sample distribution characteristics. What is more, the knowledge learned through our approach can be seen as a priori applicable to improve the performance for other general reasoning tasks. Experimental results on fourteen datasets demonstrate that the proposed approach significantly outperforms other state-of-the-art methods. On the other hand, experiments of applying our model to two other tasks show that our model effectively supports other recognition tasks related to human activity and improves performance on the datasets of these tasks.     

Disclosing incoherent sparse and low-rank patterns inside homologous GPCR tasks for better modelling of ligand bioactivities

There are many new and potential drug targets in G protein-coupled receptors (GPCRs) without sufficient ligand associations, and accurately predicting and interpreting ligand bioactivities is vital for screening and optimizing hit compounds targeting these GPCRs. To efficiently address the lack of labeled training samples, we proposed a multi-task regression learning with incoherent sparse and low-rank patterns (MTR-ISLR) to model ligand bioactivities and identify their key substructures associated with these GPCRs targets. That is, MTR-ISLR intends to enhance the performance and interpretability of models under a small size of available training data by introducing homologous GPCR tasks. Meanwhile, the low-rank constraint term encourages to catch the underlying relationship among homologous GPCR tasks for greater model generalization, and the entry-wise sparse regularization term ensures to recognize essential discriminative substructures from each task for explanative modeling. We examined MTR-ISLR on a set of 31 important human GPCRs datasets from 9 subfamilies, each with less than 400 ligand associations. The results show that MTR-ISLR reaches better performance when compared with traditional single-task learning, deep multi-task learning and multi-task learning with joint feature learning-based models on most cases, where MTR-ISLR obtains an average improvement of 7% in correlation coefficient (r²) and 12% in root mean square error (RMSE) against the runner-up predictors. The MTR-ISLR web server appends freely all source codes and data for academic usages. ^①     

Satellite-based carbon stock estimation for bamboo forest with a non-linear partial least square regression technique

This article explores a non-linear partial least square (NLPLS) regression method for bamboo forest carbon stock estimation based on Landsat Thematic Mapper (TM) data. Two schemes, leave-one-out (LOO) cross validation (scheme 1) and split sample validation (scheme 2), are used to build models. For each scheme, the NLPLS model is compared to a linear partial least square (LPLS) regression model and multivariant linear model based on ordinary least square (LOLS). This research indicates that an optimized NLPLS regression mode can substantially improve the estimation accuracy of Moso bamboo (Phyllostachys heterocycla var. pubescens) carbon stock, and it provides a new method for estimating biophysical variables by using remotely sensed data.     

Smooth relevance vector machine: a smoothness prior extension of the RVM

Enforcing sparsity constraints has been shown to be an effective and efficient way to obtain state-of-the-art results in regression and classification tasks. Unlike the support vector machine (SVM) the relevance vector machine (RVM) explicitly encodes the criterion of model sparsity as a prior over the model weights. However the lack of an explicit prior structure over the weight variances means that the degree of sparsity is to a large extent controlled by the choice of kernel (and kernel parameters). This can lead to severe overfitting or oversmoothing—possibly even both at the same time (e.g. for the multiscale Doppler data). We detail an efficient scheme to control sparsity in Bayesian regression by incorporating a flexible noise-dependent smoothness prior into the RVM. We present an empirical evaluation of the effects of choice of prior structure on a selection of popular data sets and elucidate the link between Bayesian wavelet shrinkage and RVM regression. Our model encompasses the original RVM as a special case, but our empirical results show that we can surpass RVM performance in terms of goodness of fit and achieved sparsity as well as computational performance in many cases. The code is freely available. Action Editor: Dale Schuurmans.     

Imbalanced regression and extreme value prediction

Research in imbalanced domain learning has almost exclusively focused on solving classification tasks for accurate prediction of cases labelled with a rare class. Approaches for addressing such problems in regression tasks are still scarce due to two main factors. First, standard regression tasks assume each domain value as equally important. Second, standard evaluation metrics focus on assessing the performance of models on the most common values of data distributions. In this paper, we present an approach to tackle imbalanced regression tasks where the objective is to predict extreme (rare) values. We propose an approach to formalise such tasks and to optimise/evaluate predictive models, overcoming the factors mentioned and issues in related work. We present an automatic and non-parametric method to obtain relevance functions, building on the concept of relevance as the mapping of target values into non-uniform domain preferences. Then, we propose SERA, a new evaluation metric capable of assessing the effectiveness and of optimising models towards the prediction of extreme values while penalising severe model bias. An experimental study demonstrates how SERA provides valid and useful insights into the performance of models in imbalanced regression tasks.

     

先验知识与基于核函数的回归方法的融合

In some sample based regression tasks, the observed samples are quite few or not informative enough. As a result, the conflict between the number of samples and the model complexity emerges, and the regression method will confront the dilemma whether to choose a complex model or not. Incorporating the prior knowledge is a potential solution for this dilemma. In this paper, a sort of the prior knowledge is investigated and a novel method to incorporate it into the kernel based regression scheme is proposed. The proposed prior knowledge based kernel regression (PKBKR) method includes two subproblems: representing the prior knowledge in the function space, and combining this representation and the training samples to obtain the regression function. A greedy algorithm for the representing step and a weighted loss function for the incorporation step are proposed. Finally, experiments are performed to validate the proposed PKBKR method, wherein the results show that the proposed method can achieve relatively high regression performance with appropriate model complexity, especially when the number of samples is small or the observation noise is large.     

Learning multi-tasks with inconsistent labels by using auxiliary big task

Multi-task learning is to improve the performance of the model by transferring and exploiting common knowledge among tasks. Existing MTL works mainly focus on the scenario where label sets among multiple tasks (MTs) are usually the same, thus they can be utilized for learning across the tasks. However, the real world has more general scenarios in which each task has only a small number of training samples and their label sets are just partially overlapped or even not. Learning such MTs is more challenging because of less correlation information available among these tasks. For this, we propose a framework to learn these tasks by jointly leveraging both abundant information from a learnt auxiliary big task with sufficiently many classes to cover those of all these tasks and the information shared among those partially-overlapped tasks. In our implementation of using the same neural network architecture of the learnt auxiliary task to learn individual tasks, the key idea is to utilize available label information to adaptively prune the hidden layer neurons of the auxiliary network to construct corresponding network for each task, while accompanying a joint learning across individual tasks. Extensive experimental results demonstrate that our proposed method is significantly competitive compared to state-of-the-art methods.     

A flexible coefficient smooth transition time series model

We consider a flexible smooth transition autoregressive (STAR) model with multiple regimes and multiple transition variables. This formulation can be interpreted as a time varying linear model where the coefficients are the outputs of a single hidden layer feedforward neural network. This proposal has the major advantage of nesting several nonlinear models, such as, the self-exciting threshold autoregressive (SETAR), the autoregressive neural network (AR-NN), and the logistic STAR models. Furthermore, if the neural network is interpreted as a nonparametric universal approximation to any Borel measurable function, our formulation is directly comparable to the functional coefficient autoregressive (FAR) and the single-index coefficient regression models. A model building procedure is developed based on statistical inference arguments. A Monte Carlo experiment showed that the procedure works in small samples, and its performance improves, as it should, in medium size samples. Several real examples are also addressed.     

Multi-block synchronous prediction of concrete dam displacements using MIMO machine learning paradigm

The integration of numerous monitoring points poses a significant challenge to the efficient modeling of dam displacement behavior, and multi-point synchronous prediction is an effective solution. However, traditional approaches usually construct site-specific data-driven models for each monitoring point individually, which focus on single-target regression and discard the underlying spatial correlation among different displacement monitoring points. This study therefore proposes a multi-input multi-output (MIMO) machine learning (ML) paradigm based on support vector machine (SVM) for synchronous modeling and prediction of multi-point displacements from various dam blocks. In this method, a novel multi-output data-driven model, termed as multi-target SVM (MSVM), is formulated through a deep hybridization of classical SVM architecture and multi-target regression. During the initialization of MSVM, the intercorrelation of multiple target variables is fully exploited by decomposing and regulating the weight vectors. The proposed MSVM is designed to capture the complex MIMO mapping from influential factors to multi-block displacements, while taking into account the correlation between multi-block displacement outputs. Additionally, in order to avoid obtaining the unreliable prediction results due to the empirical selection of parameters, an efficient optimization strategy based on the parallel multi-population Jaya (PMP-Jaya) algorithm is used to adaptively tune the hyperparameters involved in MSVM, which contains no algorithm-specific parameters and is easy to implement. The effectiveness of the proposed model is verified using monitoring data collected from a real concrete gravity dam, where its performance is compared with conventional single-target SVM (SSVM)-based models and state-of-the-art ML-based models. The results indicate that our proposed MSVM is much more promising than the SSVM-based models because only one prediction model is required, rather than constructing multiple site-specific SSVM-based models for different dam blocks. Moreover, MSVM can achieve better performance than other ML-based models in most cases, which provides an innovative modeling tool for dam multi-block behavior monitoring.     

Gaussian process motion graph models for smooth transitions among multiple actions

We propose a unified model for human motion prior with multiple actions. Our model is generated from sample pose sequences of the multiple actions, each of which is recorded from real human motion. The sample sequences are connected to each other by synthesizing a variety of possible transitions among the different actions. For kinematically-realistic transitions, our model integrates nonlinear probabilistic latent modeling of the samples and interpolation-based synthesis of the transition paths. While naive interpolation makes unexpected poses, our model rejects them (1) by searching for smooth and short transition paths by employing the good properties of the observation and latent spaces and (2) by avoiding using samples that unexpectedly synthesize the nonsmooth interpolation. The effectiveness of the model is demonstrated with real data and its application to human pose tracking.     

Bayesian sample steered discriminative regression for biometric image classification

Regression techniques, such as ridge regression (RR) and logistic regression (LR), have been widely used in supervised learning for pattern classification. However, these methods mainly exploit the class label information for linear mapping function learning. They will become less effective when the number of training samples per class is small. In visual classification tasks such as face recognition, the appearance of the training sample images also conveys important discriminative information. This paper proposes a novel regression based classification model, namely Bayesian sample steered discriminative regression (BSDR), which simultaneously exploits the sample class label and the sample appearance for linear mapping function learning by virtue of the Bayesian formula. BSDR learns a linear mapping for each class to extract the image class label features, and classification can be simply done by nearest neighbor classifier. The proposed BSDR method has advantages such as small number of mappings, insensitiveness to input feature dimensionality and robustness to small sample size. Extensive experiments on several biometric databases also demonstrate the promising classification performance of our method.     

GPU-accelerated and parallelized ELM ensembles for large-scale regression

The paper presents an approach for performing regression on large data sets in reasonable time, using an ensemble of extreme learning machines (ELMs). The main purpose and contribution of this paper are to explore how the evaluation of this ensemble of ELMs can be accelerated in three distinct ways: (1) training and model structure selection of the individual ELMs are accelerated by performing these steps on the graphics processing unit (GPU), instead of the processor (CPU); (2) the training of ELM is performed in such a way that computed results can be reused in the model structure selection, making training plus model structure selection more efficient; (3) the modularity of the ensemble model is exploited and the process of model training and model structure selection is parallelized across multiple GPU and CPU cores, such that multiple models can be built at the same time. The experiments show that competitive performance is obtained on the regression tasks, and that the GPU-accelerated and parallelized ELM ensemble achieves attractive speedups over using a single CPU. Furthermore, the proposed approach is not limited to a specific type of ELM and can be employed for a large variety of ELMs.     

On the impact of disproportional samples in credit scoring models: An application to a Brazilian bank data

Statistical methods have been widely employed to assess the capabilities of credit scoring classification models in order to reduce the risk of wrong decisions when granting credit facilities to clients. The predictive quality of a classification model can be evaluated based on measures such as sensitivity, specificity, predictive values, accuracy, correlation coefficients and information theoretical measures, such as relative entropy and mutual information. In this paper we analyze the performance of a naive logistic regression model (Hosmer & Lemeshow, 1989) and a logistic regression with state-dependent sample selection model (Cramer, 2004) applied to simulated data. Also, as a case study, the methodology is illustrated on a data set extracted from a Brazilian bank portfolio. Our simulation results so far revealed that there is no statistically significant difference in terms of predictive capacity between the naive logistic regression models and the logistic regression with state-dependent sample selection models. However, there is strong difference between the distributions of the estimated default probabilities from these two statistical modeling techniques, with the naive logistic regression models always underestimating such probabilities, particularly in the presence of balanced samples.     

Statistical prediction of air pollution levels using non-physical models

This paper treats the prediction problem of air pollution levels at a short range by non-physical models. Main results are given as follows: (i) The prediction accuracy of the pollution levels by time series models is compared by evaluating three performance indices, and it is shown that the multiple linear regression model already proposed is better than the auto-regressive model, the Box-Jenkins' model and the persistence model. (ii) The multiple linear regression model is more improved if the model is classified by weather. (iii) The modeling accuracy is discussed for various sample sizes, and an appropriate sample size is determined from the experiment. (iv) The confidence intervals of the predicted means at a fixed time are calculated, and the combinations of the measurement times and the measured factors that improve the prediction accuracy are chosen. (v) A revised GMDH is proposed and the accuracy by this method is more improved than those by the time series models already presented. (vi) The Kalman filtering method is applied to the prediction of pollution levels, and the measured factors that improve the prediction accuracy are chosen.