HON 207: Project 3

Detecting symmetry with a neural network

Objectives

In this project, you will gain "hands on" experience with neural networks. You will create a network, give it training and test data, train it, and report your results. You will also explore different variations of the basic network to see how they affect the network's learning.

Background

Neural networks are modeled on the "hardware" and behavior of our own brains. They learn by adjusting weights between nodes which simulate the strength of connections between neurons (or clusters of neurons). One simple task for neural networks was reported in "Learning Representations by Back-Propagating Errors", by Rumelhart, Hinton, and Williams in Nature (1986). In this task, a string of six digits (0 or 1) is judged for "mirror symmetry". An output of 1 indicates that the first three digits, when reversed, match the last three digits. An output of 0, indicates that they do not match.

There are a number of different neural net simulators available, including QuikNet, and LENS. For this project, you will use a java-based simulator from the University of British Columbia that runs as an applet through a browser or it can be downloaded. To access this applet, follow the "Neural Networks" link on this page, and then click on "Start Applet". (It may take a little while to load.) (If you have time, check out the other applications that they have available.)

What to do

First, go to the CIspace web page, locate the Neural network application, and start the applet. To get a feel for how the system works, you should create a "starter network":

The Neural Applet starts in "Create" mode, with "Create Node" the default action.
Click somewhere on the canvas (large empty rectangle) to create a node. Select OK to choose the default parameters for the node.
Create several nodes in this manner.
Click on the "Create Edge" tool.
Click on a node that the edge (link) will go from.
Click on a node that the edge (link) will go to.
Node shapes change automatically to indicate that they are input (diamond), hidden (oval), or output (rectangle) nodes.

The steps above will give you a complete neural net EXCEPT for the training data. It is possible to create training data manually using the View/Edit Examples tool (but see the notes below about pasting data into a file). Traditionally, a small portion (10% or so) of the training data is reserved (i.e. kept out of the training process), in order to keep the network from "memorizing the training set" and to have some additional items to test the network's generalization. (One standard method for ensuring that a network learns well is called N-fold cross-validition. In this technique, a different 1/Nth of the training data is removed for each of N rounds of learning, and the performance is calculated based on the removed items. But you are not required to do this in this project.)

Setting up and training the basic network

Create a network (including nodes and edges) that follows the architecture of the Rumelhart et al example.
Select File > Save graph to save the graph to a file.
Edit the graph to add training data from this file between the line that says:
```
  	
```
and the line that says:
```
</EXAMPLES>
```

IMPORTANT NOTE: This example file assumes that you defined the nodes in the order that I did, namely that the input nodes are Nodes 0 - 5, and the output node is Node 8. You should either create the graph in the same way, or edit the data file to correspond to your graph.
Another important note: This file must be saved as text only or xml (i.e. not an MS Word .doc file). (I recommend using TextPad, or WordPad (or emacs or vi).

Save the file.
In the Neural Applet, use Load from File to load your graph with the training data. Note: the applet automatically moves things around, so it won't look exactly like it did when you saved it, but the critical info should be the same.
Get into Solve mode if you're not there already by clicking on "Solve".
Modify the training set as desired according to the discussion above using View/Edit Examples.
Click Randomize Parameters to give an initial weight to each edge.
Train by clicking Step to Target Error
Click on Show plot (even while training). Training should stop when the error on the Training set gets below 0.1.
If it doesn't stop by 2000 steps or so, or if the error on the training set gets completely flat, it's probably not going to find a solution. Just stop in, randomize the parameters again, and start it over. It may take quite a few trials until it successfully solves the problem. If it doesn't solve it within 10 trials, you may need to adjust the training set. You can also double check that your network is set up correctly, including node numbers and edge directions.

After training:

Look at the hidden nodes to view their bias and outgoing weights. How do they compare to those in the Rumelhart et al paper?
Click on Summary Statistics to see which items from the test set were (un)successfully classified.

Advanced experimentation

After you get the network to learn a solution to the basic problem, you should experiment with it to explore its limitations. For example, you could:

try different sets of training and test data, either by adding or deleting items,
change the structure of the network by adding additional hidden nodes, or by deleting one or both of the hidden nodes,
change the output function on the hidden, input, or output nodes (with Set properties in Create mode)

Do any of these modifications make the network learn better? worse?

Reflection questions

Think about these questions during and after the training of your network(s), and address them in your discussion section:

What is the role of the training set and test set?
In the basic training set, there are only 8 positive items (symmetric strings) and 56 negative items. What effect do you think this has on the training?
Are there any obvious differences between your network(s) (after training) and the one presented in the Rumelhart et al paper?
Why do you think that some runs will result in a "solution" and some won't?

What to hand in

Your team should submit one report that describes your experience and your thoughts about it. Your report should include these sections:

Introduction: In your own words, what is your objective in this project?
Description of your method (e.g. how did you structure your network and what kinds of "experiments" did you do with it? Be specific.)
Results, including

a printout of your graph after training (ensuring that all weights are visible, perhaps by rearranging the spatial layout),
a printout of the "Summary Statistics" to show the results on the test set,
a description of how long it took to reach a solution,
the results of your experiments, including graphs and summary statistics to illustrate interesting results.

Discussion of your results, including points that you thought were especially interesting and answers to the reflection questions.
Conclusions: Your final thoughts on the matter
Statement of work: Description of who did what on this project. (If all work was done jointly, say so.)

There is no specific requirement on the length of the paper. It should be long enough to make it clear to me what you were doing and what you thought about, but please don't confuse quantity with quality. In other words, it doesn't need to be long to be good.

Grading

This project is worth 10 points. It will be graded on how well you follow the instructions above, on the depth of your insights, and on your demonstrated understanding of the mechanics of neural networks. My emphasis will be on the discussion section, so you should be sure to focus your energies there.

Last modified: Mon Nov 9 17:03:27 CST 2009