Learning long-term dependencies with gradient descent is difficult

Recurrent neural networks can be used to map input sequences to output sequences, such as for recognition, production or prediction problems. However, practical difficulties have been reported in training recurrent neural networks to perform tasks in which the temporal contingencies present in the input/output sequences span long intervals. We show why gradient based learning algorithms face an increasingly difficult problem as the duration of the dependencies to be captured increases. These results expose a trade-off between efficient learning by gradient descent and latching on information for long periods. Based on an understanding of this problem, alternatives to standard gradient descent are considered.     

Differential Effects of STCH and Stress-Inducible Hsp70 on the Stability and Maturation of NKCC2

Mutations in the Na-K-2Cl co-transporter NKCC2 lead to type I Bartter syndrome, a life-threatening kidney disease. We previously showed that export from the ER constitutes the limiting step in NKCC2 maturation and cell surface expression. Yet, the molecular mechanisms involved in this process remain obscure. Here, we report the identification of chaperone stress 70 protein (STCH) and the stress-inducible heat shock protein 70 (Hsp70), as two novel binding partners of the ER-resident form of NKCC2. STCH knock-down increased total NKCC2 expression whereas Hsp70 knock-down or its inhibition by YM-01 had the opposite effect. Accordingly, overexpressing of STCH and Hsp70 exerted opposite actions on total protein abundance of NKCC2 and its folding mutants. Cycloheximide chase assay showed that in cells over-expressing STCH, NKCC2 stability and maturation are heavily impaired. In contrast to STCH, Hsp70 co-expression increased NKCC2 maturation. Interestingly, treatment by protein degradation inhibitors revealed that in addition to the proteasome, the ER associated degradation (ERAD) of NKCC2 mediated by STCH, involves also the ER-to-lysosome-associated degradation pathway. In summary, our data are consistent with STCH and Hsp70 having differential and antagonistic effects with regard to NKCC2 biogenesis. These findings may have an impact on our understanding and potential treatment of diseases related to aberrant NKCC2 trafficking and expression.     

Representational power of restricted boltzmann machines and deep belief networks

Deep belief networks (DBN) are generative neural network models with many layers of hidden explanatory factors, recently introduced by Hinton, Osindero, and Teh (2006) along with a greedy layer-wise unsupervised learning algorithm. The building block of a DBN is a probabilistic model called a restricted Boltzmann machine (RBM), used to represent one layer of the model. Restricted Boltzmann machines are interesting because inference is easy in them and because they have been successfully used as building blocks for training deeper models. We first prove that adding hidden units yields strictly improved modeling power, while a second theorem shows that RBMs are universal approximators of discrete distributions. We then study the question of whether DBNs with more layers are strictly more powerful in terms of representational power. This suggests a new and less greedy criterion for training RBMs within DBNs.     

Synchronized switch harvesting applied to self-powered smart systems: Piezoactive microgenerators for autonomous wireless receivers

This paper introduces the conceptual architecture of a fully integrated, truly self-powered structural health monitoring (SHM) scheme. The challenge here is to power an array of numerous distributed actuators and sensors as well as wireless data transmission modules without recurring to heavy and costly wiring. Based on microgenerators which directly convert ambient mechanical energy into electrical energy, using the synchronized switch harvesting (SSH) method, the proposed solution allows avoiding the periodic replacement or reloading of batteries. This addresses environmental and economic issues at the same time, knowing that such elements are heavy, polluting and might be installed in rather inaccessible locations. Indeed, especially in airborne structures saving weight and maintenance cost is of priority importance.Previous work showed that such microgenerators provide a stand-alone power source, whose performances meet the requirements of autonomous wireless transmitters (AWTs) that comprise an acoustic Lamb wave's actuator and a radio frequency (RF) emitter (D. Guyomar, Y. Jayet, L. Petit, E. Lefeuvre, T. Monnier, C. Richard, M. Lallart, Synchronized switch harvesting applied to self-powered smart systems: Piezoactive microgenerators for autonomous wireless transmitters, Sens Actuators A: Phys. 138 (1) (2007) 151–160, doi:10.1016/j.sna.2007.04.009). Following this work, the present contribution presents a further step towards the integration of the SHM technique. It shows the ability of our microgenerators to provide enough energy to give logical autonomy to each self-powered sensing node, named autonomous wireless receiver (AWR), and thus to provide some local (decentralized) pre-processing ability to the SHM system.A preliminary design of the device using off-the-shelf electronics and surface mounted piezoelectric patches will be presented. Since the existence of a positive energy balance between the harvesting capabilities of the SSH technique and the energy requirements of the proposed device will be proved, the system formed by the combination of the AWR with the previously developed AWT, is a proof of concept of truly self-powered smart systems for damage detection in simple structures, setting apart application-specific optimization or miniaturization concerns that will be addressed in future works.     

A critical discussion on rebar electrical continuity and usual interpretation thresholds in the field of half-cell potential measurements in steel reinforced concrete

Corrosion of steel in reinforced concrete structures is a recurrent problem affecting civil engineering structures and costing the world billions of dollars per year. This physical phenomenon mainly results from chloride ingress or concrete carbonation. Corrosion can be diagnosed through a nondestructive method such as half-cell potential measurements. The present paper studies this method on a reinforced concrete wall containing eighteen unconnected steel bars and subjected to chloride-induced macrocell corrosion. Three corrosion systems with different configurations of connections between the steel bars are generated, involving three different anode-to-cathode surface ratios. Then, half-cell potential variations are observed versus macrocell corrosion current. The results lead to a critical discussion regarding the physical relevance of the usual potential threshold method to detect corroding rebars in reinforced concrete structures. In addition, the experiments demonstrate that electrical continuity between reinforcing steel bars is not necessary to get meaningful information about the macrocell corrosion system. At last, the paper show that the electric field (potential gradient) relative to a macrocell corrosion system may be measured by connecting the measurement system (reference electrode?+?voltmeter) to any electrochemical system in electrolytic contact with the concrete.     

Cost functions and model combination for VaR-based asset allocationusing neural networks

We introduce an asset-allocation framework based on the active control of the value-at-risk of the portfolio. Within this framework, we compare two paradigms for making the allocation using neural networks. The first one uses the network to make a forecast of asset behavior, in conjunction with a traditional mean-variance allocator for constructing the portfolio. The second paradigm uses the network to directly make the portfolio allocation decisions. We consider a method for performing soft input variable selection, and show its considerable utility. We use model combination (committee) methods to systematize the choice of hyperparameters during training. We show that committees using both paradigms are significantly outperforming the benchmark market performance.     

Adaptive importance sampling to accelerate training of a neural probabilistic language model.

Previous work on statistical language modeling has shown that it is possible to train a feedforward neural network to approximate probabilities over sequences of words, resulting in significant error reduction when compared to standard baseline models based on n-grams. However, training the neural network model with the maximum-likelihood criterion requires computations proportional to the number of words in the vocabulary. In this paper, we introduce adaptive importance sampling as a way to accelerate training of the model. The idea is to use an adaptive n-gram model to track the conditional distributions produced by the neural network. We show that a very significant speedup can be obtained on standard problems.     

Taking on the curse of dimensionality in joint distributions usingneural networks

The curse of dimensionality is severe when modeling high-dimensional discrete data: the number of possible combinations of the variables explodes exponentially. We propose an architecture for modeling high-dimensional data that requires resources (parameters and computations) that grow at most as the square of the number of variables, using a multilayer neural network to represent the joint distribution of the variables as the product of conditional distributions. The neural network can be interpreted as a graphical model without hidden random variables, but in which the conditional distributions are tied through the hidden units. The connectivity of the neural network can be pruned by using dependency tests between the variables (thus reducing significantly the number of parameters). Experiments on modeling the distribution of several discrete data sets show statistically significant improvements over other methods such as naive Bayes and comparable Bayesian networks and show that significant improvements can be obtained by pruning the network.     

Bias learning, knowledge sharing

Biasing properly the hypothesis space of a learner has been shown to improve generalization performance. Methods for achieving this goal have been proposed, that range from designing and introducing a bias into a learner to automatically learning the bias. Multitask learning methods fall into the latter category. When several related tasks derived from the same domain are available, these methods use the domain-related knowledge coded in the training examples of all the tasks as a source of bias. We extend some of the ideas presented in this field and describe a new approach that identifies a family of hypotheses, represented by a manifold in hypothesis space, that embodies domain-related knowledge. This family is learned using training examples sampled from a group of related tasks. Learning models trained on these tasks are only allowed to select hypotheses that belong to the family. We show that the new approach encompasses a large variety of families which can be learned. A statistical analysis on a class of related tasks is performed that shows significantly improved performances when using this approach.