Noisy recurrent neural networks: the continuous-time case

The classical stochastic analog of the deterministic linear system in engineering is the linear system driven by white noise. As the promise of artificial neural networks in modeling nonlinear systems continues to grow, the need for a stochastic analog with quantitative foundations for analysis and synthesis will increase. This paper presents recent work in this direction, examining recurrent neural nets (RNN driven by white noise. We examine the effect of noise on the typical continuous-time RNN model. First, we perform qualitative analysis establishing uniform boundedness of moments of the neuron states over time. To enable practical application, however, it is necessary to relate these properties to useful measures that can be estimated. We thus subsequently derive bias and variance measures for the noisy RNN with respect to the corresponding deterministic RNN. This has significant practical implications, since net design is nonminimal in the sense that several nets can solve the same problem. The results allow the user to evaluate given RNN for noise performance. The designer can use these results to constrain the design space so that the design satisfies performance specifications whenever possible. An example is provided using the measures derived in this paper to predetermine the best among several RNN designs for a given problem.