Geoffrey Towell, Symbolic Knowledge and Neural Networks: Insertion, Refinement, and Extraction, Doctoral dissertation, Computer Science Department, University of Wisconsin, 1991.  Home/Search   Document Not in Database   Summary   Related Articles   Check

....challenged, or Nelson Marcos National University of Singapore Singapore nelson comp.nus.edu.sg left unsupported by all the evidence available. Certain new evidences that do not figure in any existing belief can be assimilated as a new belief. successive rule refinement in a hybrid system [11] [12] 13] The available evidence is in the form of raw data. These have to be converted into rule form so that they can be integrated with the existing beliefs about the domain. Converting evidence into rule form is done through a rule extraction system that trains a neural network using the ....

Geoffrey Towell, Symbolic Knowledge and Neural Networks': Insertion, Refinement, and Extraction, Doctoral dissertation, Computer Science Department, University of Wisconsin, 1991.

....combinations of inputs which make an output node active regardless of the value of other inputs to the output node. Some of the existing rule and decision tree extraction techniques using neural nets are explained below. 2.3.0. 1 MofN Extracting Re ned Rules From KBANN Towell and Shavlik [Towell, 1991, Towell and W.Shavlik, 1992] developed a method that produces about one MofN rule for each node in knowledge based networks (e.g. KBANN. Towell made two assumptions about the trained networks in the rule extraction methods he presented. Nodes have activation values near 0 or 1. The boolean ....

Towell, G. G. (1991). Symbolic Knowledge and Neural Networks: Insertion, Re nement and Extraction. PhD thesis, Department of Computer Sciences, University of Wisconsin-Madison. (Also appears as UW Technical Report 1072 [out of print].).

....algorithms explained below are search based, RuleNeg, Rule extraction as learning, TREPAN and DecText algorithms are learning based. Some of the existing rule and decision tree extraction techniques using neural nets are explained below. 2. 1 MofN Extracting Refined Rules From KBANN Towell [17, 19] developed a method that produces about one MofN rule for each node in a Knowledge Based Neural Networks (e.g. KBANN. Towell s method requires the nodes to have activation values near 0 or 1. The boolean constraint is achieved by using a steep sigmoid function. It also assumes that meaning of the ....

G. G. Towell. Symbolic Knowledge and Neural Networks: Insertion, Refinement and Extraction. PhD thesis, Department of Computer Sciences, University of Wisconsin-Madison, 1991. (Also appears as UW Technical Report 1072 [out of print].).

....Our intention is not to provide an exhaustive comparative analysis with other extraction methods. Such a comparison could be easily biased, depending on the application at hand, training parameters and testing methodology used. Nevertheless, in what follows, we also present the results reported in [10,30,34], when available. We have used three application domains in order to test the extraction algorithm: the MONK s problems [32] DNA sequences analysis [5,10,30,34] and Power Systems FAULT DIAGNOSIS [4,31] Briefly, the results obtained indicate that a very high fidelity between the network and the ....

....at hand, training parameters and testing methodology used. Nevertheless, in what follows, we also present the results reported in [10,30,34] when available. We have used three application domains in order to test the extraction algorithm: the MONK s problems [32] DNA sequences analysis [5,10,30,34], and Power Systems FAULT DIAGNOSIS [4,31] Briefly, the results obtained indicate that a very high fidelity between the network and the extracted set of rules can be achieved. They also indicate that a reduced readability is the price one has to pay for soundness. We will discuss this problem in ....

[Article contains additional citation context not shown here]

G.G. Towell, Symbolic Knowledge and Neural Networks: Insertion, Refinement and Extraction, Ph.D. Thesis, University of Wisconsin, Madison, WI, 1992.

....and the complexity of interpretation of relevant physiological information impose extra demands that preclude the applicability of most statistical and machine learning techniques developed so far. To overcome such constraints, we propose Knowledge Based Artificial Neural Networks (KBANNs) [1], a hybrid methodology that combines knowledge from a domain in the form of simple rules with connectionist learning. This combination allows the use of small sets of data (typical of medical diagnosis tasks) to train the network. The initial structure is set from the dependencies of a set ....

....that could lead to more robust, flexible approaches which are capable of making the most of any available source of knowledge to constrain the search space and, at the same time, to guide the search itself to improve generalization as much as possible. 4 This motivation led G. Towell [1] to implement Knowledge Based Artificial Neural Networks (KBANNs) as a hybrid methodology consisting of two approaches: symbolic and connectionist. The symbolic module is knowledge intensive. It contains the description of a domain theory in the form of a hierarchically structured set of ....

[Article contains additional citation context not shown here]

Towell, G. G. (1991). Symbolic Knowledge and Neural Networks: Insertion, Refinement and Extraction. Ph.D. thesis, University of Wisconsin, Madison.

....search that space. However, it differs from previous work in this area by starting from an existing scientific model and using heuristic search to revise the model in ways that improve its fit to observations. Although there exists some research on theory refinement (e.g. Ourston Mooney 1990; Towell, 1991), it has emphasized qualitative knowledge rather than quantitative models that relate continuous variables, which play a central role in many sciences. In the pages that follow, we describe an approach to revising quantitative models of complex systems. We believe that our approach is a general ....

....than revising existing ones. In contrast, another line of research has addressed the refinement of existing models to improve their fit to observations. For example, Ourston and Mooney (1990) developed a method that used training data to revise models stated as sets of propositional Horn clauses. Towell (1991) reports another approach that transforms such models into multilayer neural networks, then uses backpropagation to improve their fit to observations, much as we have done for numeric equations. Work in this paradigm has emphasized classification rather than regression tasks, but one can view our ....

Towell, G. (1991). Symbolic knowledge and neural networks: Insertion, refinement, and extraction. Doctoral dissertation, Computer Sciences Department, University of Wisconsin, Madison.

....given belief net structure [Hec95] We, however, are making discrete changes to the structure of the Horn theory. Similarly, belief revision systems [AGM85, Dal88, Gar88, KM91] take as input an 2 (1) However, we make no claims concerning the applicability of our techniques to systems like KBANN [Tow91], which use a completely different means of modifying a theory. 2) The companion paper [Gre99] considers yet other ways of modifying a theory, viz. by rearranging the order of its component rules or antecedents. The Complexity of Theory Revision 5 initial theory T 0 and a new assertion hq; i ....

....of hill climbing. Finally, as noted in the Introduction, we hope these results will help push researchers and developers to consider other approaches to revising a sub optimal theory perhaps by finding useful special cases, employing alternative approaches (possibly stochastic, or like KBANN [Tow91]) changing representations, or exploiting other types of information The Complexity of Theory Revision 24 present, in either the labeled queries, or the reviser s prior knowledge. A Proofs Theorem 1 (from [Vap82, Theorem 6.2] Given a class of theories T , and ffl; ffi 0, let T 2 T be the ....

Geoff Towell. Symbolic Knowledge and Neural Networks: Insertion, Refinement and Extraction. PhD thesis, University of Wisconsin, Madison, 1991.

....two directions: the use of symbolic AI methods and the use of sub symbolic connectionist methods (Artificial Neural Networks ANN) In this framework we have studied different ways to integrate these two approaches through Hybrid NeuroSymbolic Systems. Hybrid neuro symbolic systems, like KBANN [Towell 91], SYNHESYS [Giacometi 92] or INSS [Osrio 95] generally use a set of production rules from the propositional calculus of 0 or 0 order such as: IF premise And Or premise . THEN conclusion ; IF feature variable [ value And Or . THEN conclusion There are some known methods used ....

....use a set of production rules from the propositional calculus of 0 or 0 order such as: IF premise And Or premise . THEN conclusion ; IF feature variable [ value And Or . THEN conclusion There are some known methods used to insert and extract these rules to from ANNs [Towell 91]. However, this kind of rules has a limited capacity for knowledge representation. Indeed, it can not be used to represent relations like that: IF X Y THEN Y is less important related to the problem This type of rules, that we call high level rules , has as a main characteristic the strong ....

Towell, Geoffrey. Symbolic Knowledge and Neural Networks: Insertion, Refinement and Extraction. Ph.D. Thesis. Univ. of Wisconsin - Madison. USA, 1991.

....to explore more than just connectionist rule extraction, it is not surprising that the generality and efficiency of the extraction techniques was not the focus. Nevertheless, McMillan s work with RuleNet must be credited as laying down the basic ideas behind template based rule extraction. 4.2. Towell Towell (1991) and Towell and Shavlik (1991) have attempted to combine the merits of explanation based learning and empirical (inductive) learning. To carry out a learning task using Towell s hybrid approach requires both an initial domain theory a set of rules for the task and a set of labeled training ....

....curve, and each approach will have applications where it is most appropriate. Thus we conclude that both methods have their place in rule extraction tool kits. 4.3. Other research Other methods for the connectionist rule extraction have been suggested by Fu (1989, 1991) and Hayashi (1990) Towell (1991) also describes several alternatives to the n of m method described earlier. The work of Gallant (1988) was an important early example of integrating connectionist and rule based approaches. Other examples of integrated approaches include Dolan and Smolensky (1989) Touretzky and Hinton (1998) ....

[Article contains additional citation context not shown here]

Towell, G. (1991). Symbolic knowledge and neural networks: Insertion, refinement and extraction. Ph.D. Thesis. Department of Computer Science, University of Wisconsion, Madison, WI.

....which have been constructed from training data. Unlike the technique proposed in this paper, most of the approaches seek to assign semantic concepts to the individual hidden and output units of a network. Often they translate each hidden unit into a separate rule. For example, Towell and Shavlik [Towell, 1991] , Towell and Shavlik, 1992] describe a method which analyzes the weights and biases of a neural network in order to translate the network step by step into a set of rules with equivalent structure. In order to do so, the weights and biases of the network are truncated and discretized, resulting ....

....m of n rules) from trained networks. Craven and Shavlik managed, most notably, to extract rules from a reduced version of Sejnowski and Rosenberg s NETtalk domain [Sejnowski and Rosenberg, 1986] Note that the weight regularization term may replace the need for initial knowledge, as reported in [Towell, 1991] and [Towell and Shavlik, 1992] In both of these extraction schemes the effectiveness of the rule extraction mechanism, as well as the degree of correctness of the extracted rules, relies crucially on the particular training procedure invoked. VI Analysis differs from these approaches in that it ....

Geoffrey G. Towell. Symbolic Knowledge and Neural Networks: Insertion, Refinement and Extraction. PhD thesis, University of Wisconsin--Madison, 1991.

....Porter, Lowry, Dobbs, et.al. 1994) To explore the full potential of such a promising non invasive technique, it is necessary to develop a computational aid capable of classifying complex and limited data. KBANN s (Towell, Shavlik, Noordewier, 1990; Noordewier, Towell, Shavlik, 1991; Towell, 1991) are a reliable methodology that combines the strengths of both symbolic and connectionist approaches into a hybrid learning algorithm capable of learning from small data sets. This paper presents results to assess the potential of knowledge based over knowledge free artificial neural networks ....

....a hybrid methodology which combines explanation based and empirical learning techniques. Strengths of one methodology overcome the weaknesses of the other, thus the combination of both approaches has a more robust performance than either method alone (Towell et al. 1990; Noordewier et al. 1991; Towell, 1991). This approach leans on the fact that living beings use previous knowledge to ease the learning of new tasks (Thrun, 1995) KBANN s have two modules. The first part, the explanation based module, consists in a set of approximately correct rules contained in the knowledge base. These rules will ....

[Article contains additional citation context not shown here]

Towell, G. G. (1991). Symbolic Knowledge and Neural Networks: Insertion, Refinement and Extraction.

....the connectionist module and boundary and particular cases in the atypical memory zone. III.4.2 INSS, a hybrid system for compiling and extracting knowledge III.4.2. 1 Introduction In the classification domain, the neuro symbolic hybrid systems, as for example SYNHESYS [Giacometti 92] and KBANN [Towell 91] exploit their capacity to use at the same time theoretical knowledge (set of symbolic rules) and empirical knowledge (set of observed examples) These systems also allow knowledge transfer between the symbolic and connectionist modules. 23 Figure 3 Hybrid neuro symbolic systems and ....

- 39 - G. Towell(1991). Symbolic Knowledge and Neural Networks: Insertion, Refinement and Extraction. Ph.D. Thesis, University of Wisconsin-Madison - Computer. Science Dept.

....consequent. Certainty factors on the individual rules were set so that if all of the antecedents of the original Horn clause were true, the result is a total certainty of 0.9 for the consequent. Rapture and Rapture Kbann were run on this dataset and the results compared with Kbann s results from Towell (1991). Standard training and test runs were performed, resulting in the learning curves shown in Figure 2. For the two Rapture systems, this graph is a plot of average classification accuracy over 25 independent trials. A single trial consists of providing each system with an increasing number of ....

Towell, G. G. (1991). Symbolic Knowledge and Neural Networks: Insertion, Refinement, and Extraction.

....to be able to use together both aspects. The INSS System Incremental Neuro Symbolic System developed by F. Osorio [11] is an hybrid system that combines theoretical and practical models of knowledge (i.e. symbolic rules and pratical examples) Its base model (inspired by KBANN system [14] with an important improvement in the ANN learning algorithm) works with two independent modules, each one representing one of the considered aspects. Theoretical aspect is represented by a set of production rules that associate hypothesis conditions or facts to conclusion facts. This set of rules ....

....attempt to find logical relationships between these variables, through weights value carried by links between corresponding units. At least, rules are derived from these relationships. This type of method is represented by algorithms like SUBSet al..gorithm, created by L.M. Fu [5] or NofM algorithm [14]. On the other hand, pedagogical methods don t examine each unit of the net, rather considering it as a black box . These methods try to directly find the relationships between the inputs and the outputs of the network, through directed search into the space of inputs. Examples of this kind of ....

[Article contains additional citation context not shown here]

Towell, G. Symbolic Knowledge and Neural Networks: Insertion, Refinement and Extraction ; Ph.D Thesis, University of Wisconsin-Madison - Comp. Science Dept. 1991.

....is (with high probability) more accurate. We let B (DNF ) TR refer to this model (for DNF formulae) where the TR designates Theory Revision , corresponding to the many existing systems that perform essentially the same task, albeit in the framework of Horn clause based reasoning systems; cf. Tow91, WP93, MB88, OM90, LDRG94] After this, we consider theory revision for decision trees, B (DT ) TR . There are several obvious advantages to theory revision over the grow from scratch approach discussed in the previous section. First, notice from Theorem 2 that the number of samples required ....

Geoff Towell. Symbolic Knowledge and Neural Networks: Insertion, Refinement and Extraction. PhD thesis, University of Wisconsin, Madison, 1991.

....Table 1: Left column: General weaknesses of expert systems (ES) Right column: some solutions. well known, and are often the origin of fruitful research, so we indicate in this table solutions or research fields. Besides, some of these problems are not limited to expert systems (see for instance [11] for a similar study in the field of machine learning) Several researchers [5, 8, 6, 10] have pointed out that these weaknesses of symbolic expert systems actually correspond to strong points of connectionist systems due to the latter s inherent capabilities (learning, generalization, ....

G. G. Towell. Symbolic knowledge and neural networks : insertion refinement and extraction. Technical Report 1072, Univ. of Wisconsin-Madison, Computer Science Dept., January 1992.

....to the original rule base, much of the expertgiven domain knowledge will remain intact, enabling the rules to perform at satisfactory levels of accuracy on unseen examples. Ourston and Mooney (in press) illustrate the success of this heuristic with results in several domains. 2. 2 KBANN Kbann (Towell, 1991) is a revision system that combines a rule base with neural network learning. An expert supplied rule base is converted into a neural network, which is then trained using connectionist techniques. After training, the network is translated into symbolic rules. The conversion process into a neural ....

Towell, G. G. (1991). Symbolic Knowledge and Neural Networks: Insertion, Refinement, and Extraction. PhD thesis, University of Wisconsin, Madison, WI.

....When a KNN is missing terms that are required to express a concept, it will modify existing terms to cover the vocabulary shortfall. This often leads to large shifts in the meaning of terms (discussed above) In such cases is is necessary to augment the initial network with additional hidden units (Towell et al. 1991). However, adding hidden units opens many of the issues raised in the above discussion of extracting rules from networks that are not initialized with a domain theory. ffl The system is yet to be tested on a broad range of problems. As discussed previously, sequence analysis problems often ....

....useful for real world problems. The two problems also share aspects that are not related to sequence analysis. For example, the initial domain theory of both problems is overly specific. Empirical tests suggest that Kbann is slightly more effective when the domain theory is overly specific (Towell, 1991). Hence, tests on a broader range of datasets are needed to prove the generality of the method. Towell Shavlik Extracting Rules 28 In addition to extending Kbann and the NofM method to address these limitations, we are currently working on a rule extraction algorithm that operates during the ....

Towell, G. G. (1991). Symbolic Knowledge and Neural Networks: Insertion, Refinement, and Extraction. PhD thesis, University of Wisconsin, Madison, WI. In preparation.

....a system that is superior to either of its parts. Using both theory and data to learn categorization tasks, Kbann has been shown to be more effective at classifying examples not seen during training than a wide variety of machine learning algorithms (Towell et al. 1990; Noordewier et al. 1991; Towell, 1991). However, recent experiments (briefly described on the next page) point to weaknesses in the algorithm. In addition, neural learning techniques are commonly thought to be relatively weak at training networks that This research was partially supported by Office of Naval Research Grant ....

....domain specific inference rules (that define what is initially known about a topic) into a neural network. In so doing, the algorithm defines the topology and connection weights of the networks it creates. Detailed explanations of this rules to network translation appear in (Towell et al. 1990; Towell, 1991). As an example of the Kbann rules to network translation method, consider the small rule set in Figure 2a that defines membership in category A. Figure 2b represents the hierarchical structure of these rules: solid and dotted lines represent necessary and prohibitory dependencies, respectively. ....

[Article contains additional citation context not shown here]

Towell, G. G. 1991. Symbolic Knowledge and Neural Networks: Insertion, Refinement, and Extraction. PhD thesis, University of Wisconsin, Madison, WI.

....between the actual and desired activations of the output units. bias = A: B. A: C. A: D. A: E. B C D A E w w w w B C D A bias = E w w w w A : B, C, D, not(E) w 5 2 w 1 2 Figure 5: Translation of conjunctive and disjunctive rules into a Kbann net. a Kbann net are given by Towell et al. 1990) and Towell (1991); the following is a brief summary. Kbann translates a collection of rules into a neural network by individually translating each rule into a small subnetwork that accurately reproduces the behavior of the translated rule. These small subnetworks are then assembled to form a single neural network ....

....section briefly describes the two real world datasets from the domain of molecular biology used for testing Kbann. Both datasets have been previously used to demonstrate the usefulness of the Kbann algorithm (Noordewier et al. 1991; Towell et al. 1990) They are more thoroughly described by Towell (1991). 4.1 Background and notation The two biological datasets in this chapter are taken from the domain of DNA sequence analysis. DNA, the blueprint of almost all living organisms, is a linear sequence from the alphabet fA, G, T, Cg. Each of these characters is referred to as a nucleotide. Human ....

[Article contains additional citation context not shown here]

Towell, G. G., Symbolic Knowledge and Neural Networks: Insertion, Refinement, and Extraction, PhD thesis, Computer Sciences Department, University of Wisconsin, Madison, WI, 1991.

No context found.

Geoffrey Towell, Symbolic Knowledge and Neural Networks: Insertion, Refinement, and Extraction, Doctoral dissertation, Computer Science Department, University of Wisconsin, 1991.

No context found.

G.G. Towell (1991) Symbolic Knowledge and Neural Networks: Insertion, Refinement, and Extraction. PhD Thesis, University of Wisconsin - Madison.

No context found.

Towell G.G. (1991) Symbolic Knowledge and Neural Networks: Insertion, Refinement, and Extraction. PhD Thesis, University of Wisconsin - Madison.

No context found.

G. Towell, Symbolic Knowledge and Neural Networks: Insertion, Refinement, and Extraction. Doctoral dissertation, Computer Sciences Department, University of Wisconsin, Madison (1991).

No context found.

Towell, G "Symbolic Knowledge and Neural Networks: Insertion, Refinement and Extraction", Doctoral Dissertation, University of Wisconsin, 1992

First 50 documents

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC