Extraction of comprehensive symbolic rules from a multi-layer perceptron

This paper introduces a system that extracts comprehensible symbolic rules from a multi-layer perceptron. Once the network has been trained in the usual manner, the training set is presented again, and the actual activations of the units recorded. Logical rules, corresponding to the logical combinations of the incoming signals, are extracted at each activated unit. This procedure is used for all examples belonging to the training set. Thus a set of rules is obtained which accounts for all logical steps taken by the network to process all known input patterns. Furthermore, it is shown that if some symbolic meaning were associated with every input unit, then the hidden units, which have formed concepts in order to deal with recurrent features in the input data, possess some symbolic meaning too! The algorithm described in this paper allows the recognition or the understandability of these concepts: they are found to be reducible to conjunctions and negations of the human input concepts. The rules can also be recombined in different ways, thus constituting some limited bur sound generalization of the training set. Neural networks could learn concepts about domains where little theory was known but where many examples were available. Yet, because their knowledge was stored in the synaptic strengths under numerical form, it was difficult to comprehend what they had discovered. This system therefore provides some means of accessing the information contained inside the network. Copyright (C) 1996 Elsevier Science Ltd