J.F. Kelley's Professional Bio

Where did the usability term Wizard of Oz come from?

[ Wikipedia article on this topic | Jeff's Personal Page | Jeff's Professional Bio ]

 

The term Wizard of Oz  (originally OZ Paradigm) has come into common usage in the fields of Experimental Psychology, Human Factors, Ergonomics and Usability Engineering to describe a testing or iterative design methodology wherein an experimenter (the "Wizard"), in a laboratory setting, simulates the behavior of a theoretical intelligent computer application (often by going into another room and intercepting all communications between participant and system).  Sometimes this is done with the participant's a-priori knowledge and sometimes it is a low-level deceit employed to manage the participant's expectations and encourage natural behaviors (though always, I would hope, with appropriate disclosure during the debriefing part of the experiments).

 

I coined that term around 1980 to describe the methodology I developed during my dissertation work at The Johns Hopkins University (my dissertation advisor was Professor Alphonse Chapanis, the "Godfather of Human Factors and Engineering Psychology").  Amusingly enough, in addition to some one-way mirrors and such, there literally was a curtain separating me, as the "Wizard", from view by the participant during the study.

 

Here's an interesting tidbit that you may not find in the literature: I did originally have a definition for the "OZ" acronym (aside from the obvious parallels with the1900 book by L. Frank Baum).  "Offline Zero" was a reference to the fact that an experimenter (the "Wizard") was interpreting the users' inputs in real time during the simulation phase.  Folks laughed at this lily-gilding as an expression of my acronymophillia and I eventually dropped it.

 

The OZ methodology is very powerful.  In its original application, I was able to create a simple keyboard-input natural language recognition system that far exceeded the recognition rates of any of the far more complex systems of the day.  The key enabling factor was that the system was designed to work in a single context (calendar-keeping), which constrained the complexity of language encountered from users to the extent where a simple language processing model was sufficient to meet the goals of the application.  (The processing model was a two-pass keyword/keyphrase matching approach, based loosely on the algorithms employed in Weizenbaum's famous Eliza program).

 

OZ was important because it addressed the obvious criticism:

 

Who can afford to use an iterative methodology to build a separate natural language system (dictionaries, syntax) for each new context?

 

The answer turned out to be:

 

By using an empirical approach like OZ, anyone can afford to do this; my dictionary and syntax growth reached asymptote (better than 90% recognition) after only 16 experimental trials and my resulting program, with dictionaries, was less than 300k of code.

 

Here are some of the original references on the subject (the methodology has been picked up in may research areas, and there are numerous subsequent references). 

 

Summary of the technical aspects of the work:

Kelley, J.F., "CAL - A Natural Language program developed with the OZ Paradigm: Implications for Supercomputing Systems". First International Conference on Supercomputing Systems (St. Petersburg, Florida, 16-20 December 1985), New York: ACM, pp. 238-248

 

Brief desciption of the methodology:

Kelley, J.F., "An empirical methodology for writing user-friendly natural language computer applications". . Proceedings of ACM SIG-CHI '83 Human Factors in Computing systems (Boston, 12-15 December 1983), New York: ACM, pp. 193-196.

 

**The BEST DESCRIPTION of the methodology:

Kelley, J.F., "An iterative design methodology for user-friendly natural language office information applications" . ACM Transactions on Office Information Systems, March 1984, 2:1, pp. 26-41.

 

Another tidbit: in the above-referenced article, I published one of my two life-time-best puns: I refer to the fact that during one of the early phases of the OZ methodology, the experimenter simulates the system in toto.  ("in toto" is Latin for "altogether"; Toto was also the name of Dorothy's dog in the 1900 story.)

 

The unpublished dissertation itself:

Kelley, J.F., "Natural Language and computers: Six empirical steps for writing an easy-to-use computer application". Unpublished doctoral dissertation, The Johns Hopkins University, 1983. (Can be obtained from University Microfilms International; 300 North Zeeb Road; Ann Arbor, Michigan 48106.)

 

A separate task-analysis study, done in preparation for the application design:

Kelley, J.F. & Chapanis, A., "How professional persons keep their calendars: Implications for computerization". Journal of Occupational Psychology, 1982, 55, 241-256.