A UNIVERSAL THEOREM ON LEARNING-CURVES

A learning curve shows how fast a learning machine improves its behavior as the number of training examples increases. This paper proves a universal asymptotic behavior of learning curves for general noiseless dichotomy machines, or neural networks. It is proved that irrespective of the architecture of a machine, the average predictive entropy or the information gain [e*(t)] converges to 0 as [e*(t)] approximately d/t as the number t of training examples increases, where d is the number of modifiable parameters of a machine.