Optimal interpolating and smoothing functional artificial neural networks (FANNs) based on a generalized Fock space framework

Functional artificial neural networks (FANNs) are artificial neural networks (ANNs) in which the synaptic weights are "functions" rather than numbers. Thus the signals in such networks are analog, and the action of a synapse on a signal passing through it takes place in the form of a scalar product in L-2 between the functional weight and the signal. In this paper, four classes of FANNs are introduced. They result from the solution of a nonparametric optimization problem in a generalized Fock space (GFS) of abstract Volterra series under interpolating or smoothing input-output training data constraints. Two of these classes of FANNs correspond to the interpolating case and are represented by what we call the (two-layer) optimal interpolating (OI) FANN and the optimal multilayer neural interpolating (OMNI) FANN. The remaining two classes correspond to the smoothing case. We name their representations as the (two-layer) optimal smoothing (OS) FANN and the optimal smoothing multilayer artificial neural (OSMAN) FANN. In addition to providing the background and the derivation of these FANNs, this paper presents a novel approach to their implementation. This approach does away with the computationally cumbersome use of functional weights, Instead, the effect of these weights is provided by linear time-invariant differential equation models of which those weights are impulse responses. These are implemented by a linear filter bank. This approach thus leads to simple and meaningful causal realizations of FANNs which we call Dynamical FANNs or simply D-FANNs.