DILUTION AND SPARSE CODING IN THRESHOLD-LINEAR NETS

The storage capacity of an autoassociative memory with extremely diluted connectivity and with threshold-linear elementary units is studied in its dependence on the graded structure and on the sparseness of the coding scheme, and on the form of the learning rule used. As the coding becomes sparse, more patterns can be stored, and the difference in capacity (measured for a given number of modifiable synapses per unit) between fully connected and highly diluted systems vanishes. Graded (non-binary) codings, especially when used with learning rules nonlinear in their post-synaptic factor, further increase the number of patterns that can be stored by making their retrieved representation even sparser.