LEARNING AND GENERALIZATION IN A 2-LAYER NEURAL-NETWORK - THE ROLE OF THE VAPNIK-CHERVONENKIS DIMENSION

Bounds for the generalization ability of neural networks based on Vapnik-Chervonenkis (VC) theory are compared with statistical mechanics results for the case of the parity machine. For fixed phase space dimension, the VC dimension grows arbitrarily by increasing the number K of hidden units. Generalization is impossible up to a critical number of training examples that grows with the VC dimension. The asymptotic decrease of the generalization error epsilon(G) comes out independent of K and the VC bounds strongly overestimate epsilon(G). This shows that phase space dimension and VC dimension can play independent and different roles for the generalization process.