HON 207, Project 2

(This project was borrowed from Randolph M. Jones of SoarTech.)

Exploring a model of subtraction bugs

The main points of this project are to:

• Use some production-rule models of a cognitive skill
• Explore how repair theory explains some of the mistakes people make when doing subtraction problems
• Design and implement your own small experiments with the models

The system you will use

The "subtraction world" simulator comes with a number of pieces:

• A simulated "blackboard" for setting up subtraction problems. This includes a graphical interface to observe the actions of the cognitive models while they solve problems.
• A set of "primitive" subtraction actions and goals that can be used by a number of different procedures for doing subtraction. These operators include various commands for interacting with the "blackboard", as well as an implementation of some of Brown and VanLehn's "repair strategies".
  • The operators are implemented within the Soar cognitive architecture, which represents its entire long-term memory as a set of production rules.
• Three separate "subtraction models", or correct procedures for doing multi-column subtraction problems. These models include different sets of conditions for proposing the various primitive actions and goals that we have provided for subtraction.
  • The vanlehn procedure closely resembles the procedure presented by Brown and VanLehn (1980) in their paper on repair theory.
  • The neo-vanlehn procedure is very similar to the vanlehn procedure, but has a slightly different method for borrowing through zero.
  • The neches procedure is inspired by a subtraction model proposed by Neches, Langley, and Klahr (1987). This procedure has a more "flat" goal hierarchy than the other two do.
• A "procedure editor" that allows you to delete particular operations from a subtraction procedure, in order to introduce bugs.
• A "repair dialog" that allows you to select from a number of possible repair strategies when the model reaches an impasse. As described by Brown and VanLehn, different types of repairs will lead to different types of bugs.

You will need to install the system on a Windows machine. You can download and unpack the subtract.zip file. If you don't have a way to "unzip" this file, I recommend downloading an evaluation copy of WinZip.

The activities you will engage in

• Explore bugs generated by two of the models.
  • Without running the program, study the Neches model and one of the other two models. Try to identify five or more of the bugs that Brown and VanLehn discovered (listed in the appendix of their paper), that you think each model could generate. Try to include some different bugs in each of your two lists.
  • Now run the program on each model, using subtraction problems that you think will duplicate the bugs you predicted. See if you can actually generate the bugs you predicted you could.
• Investigate how faulty knowledge can sometimes lead to correct answers
  • For one of the subtraction procedures, try to come up with pairs of deletions and repairs that lead to correct answers on some problems. This highlights some difficulties with bug diagnosis, because students can have faulty knowledge but still get correct answers if they happen to use the right repair (without really understanding what they are doing). (Note: Just deleting a line which isn't used for a particular problem doesn't address this issue. For example, if you deleted a line that invoked borrowing, and gave it a problem that doesn't require borrowing, it would get the right answer. But no teacher would be so silly as to assume that a student could do multiple-column subtraction without giving them some problems that require borrowing.)
• Write a report summarizing your experiences with this project, describing any problems you had completing the project, and discussing what you learned from doing the project. In particular, think about the following discussion questions:
  1. How do you think children learn procedures like these?
  2. Could other factors besides the completeness of the student's procedures affect their answers on particular problems?
  3. What is more important, the knowledge of which actions that they should take, or the knowledge of when to do those actions?
  4. Which of the procedures (vanlehn, neo-vanlehn, neches) do you think is "the best"? Why? Can you think of an alternative procedure for multi-column subtraction?
  5. Do you think Repair Theory is a good cognitive model of multi-column subtraction?

What you will turn in

Your project report should include the following sections:

• Objective: What you are trying to find out?
• Method: How did you approach the tasks?
• Results, including:
• Your list of predicted bugs for each of the two procedures you choose to study, along with explanations of your predictions.
• A detailed summary of the experiments you ran in order to generate the bugs you predicted.
• A detailed summary of the experiments you ran in order to generate correct answers with faulty knowledge.
(Suggestion: For the bugs, present your predictions and results in a table (or two) which contains: the bug name, example problem, why you thought it would be a bug, how you attempted to generate it, and whether or not you were successful.)
• Discussion: What did you learn from your experiments (including answers to the discussion questions above)?
• Conclusions: Your final reflections on what you learned from the project.
• Statement of work: Description of who did what on this project. (If all work was done jointly, say so.)

Grading

• The "high A" score on this project is 10 points, but if you do a great job and extra work, it is possible to earn more than 10 points.
• 2 points for your predictions for bugs submitted in the lab
• 2 points for your description of your experiments for generating bugs, and for producing the correct answers with faulty knowledge.
• 3 points for your discussion of the results.
• 3 points for your report
• 1 extra credit point to reward especially creative or industrious efforts on this project
• You will be graded on the quality of writing in your report, as well as the quality of the insights, information, and analysis you provide. Bland reports that shallowly summarize what you did will not receive full points.

My grading sheet for this project can be found here.

References

• Brown, J. S., & VanLehn, K. (1980). Repair theory: A generative theory of bugs in procedural skill. Cognitive Science, 4, 379-426.
• Neches, R., Langley, P., & Klahr, D. (1987). Learning, development, and production systems. In D. Klahr, P. Langley, & R. Neches (Eds.), Production system models of learning and development. Cambridge, MA: MIT Press.