Computational Approaches to Natural Language Dialogue Modelling


 CS 599  Spring 2004


Meeting time: Tuesday 3:30 - 6:20pm THH203





Instructor: David Traum (Research Assistant Professor, CS)


Pre-requisite: One of the following courses or approval of instructor: CS 544 (Natural Language Processing) or CS 562 (Empirical Methods in Natural Language Processing) or EE619 (Advanced Topics in Speech Recognition.



Natural language dialogue involves extended communication (multiple utterances) using natural language) between two or more participants. Dialogue modelling concerns the coherence that 'glues' these utterances together: How can a new utterance be understood given the context of what has come before? How can one use context to predict what will come next? How can one use the context to decide what should come next (especially to produce a contribution by a dialogue participant)? Computational models can help formalize answers to these questions and also provide the basis of systems that can track and understand dialogue and participate in dialogue with a human user. Dialogue systems are both an old topic in AI/ Computer Science (with famous early examples such as Eliza, Lunar, and SHRDLU) and a topic of much current research. Moreover, dialogue systems have become a commercial reality, with companies such as SpeechWorks and Nuance as well as others fielding automated information systems.


This course is intended to introduce students to the basic phenomena in natural language dialogue, standard techniques for dialogue modeling in dialogue systems, and areas of current research in the field. The course will consist of 1/2 lectures by the instructor, and 1/2 group discussion of research papers, mostly led by students. Students should come away from the course with a basic understanding of the topic, and be able to:

-implement simple dialogue systems

-read and assess research papers in the area

-embark on new research in the area


Course Requirements:                                                                          Grading

[1] reading and reviewing assigned papers                                              10%

[2] participation in class discussions                                                        10%

[3] leading one discussion topic based on assigned readings                   20%

[4] 2-3 small assignments                                                                         20%

[5] main project                                                                                        40%



Tentative Schedule:

January 20
Lectures: Overview of the general topic: what is dialogue, dialogue

genres, dialogue system components, techniques for dialogue modelling

or dialogue management


Discourse and Dialogue. Grosz, Barbara; Scott, Donia; Kamp, Hans; Cohen, Phil; Giachin, Egidio. Chapter 6 of  Survey of the State of the Art in Human Language Technology, R. Cole, J. Mariani, H. Uszkoreit, A. Zaenen, & V. Zue (eds.), Cambridge University Press, 1997.

Dialogue and Conversational Agents. Daniel Jurafsky and James H. Martin. Chapter 19 of Speech and Language Processing, Prentice Hall, 2000.

January 27
Lectures: Simple Dialogue techniques: trees and finite state approaches,  key phrase reactive approaches, Voice XML

ELIZA--A Computer Program For the Study of Natural Language Communication Between Man and Machine Communications of the ACM Volume 9, Number 1 (January 1966): 36-45.

Vector-Based Natural Language Call Routing. Jennifer Chu-Carroll, Bob Carpenter. Journal of Computational Linguistics, 25(30), pp. 361-388, 1999.

Sutton, S., Novick, D.G., Cole, R., Vermeulen, P., de Villiers, J., Schalkwyk, J. and Fanty, M., "Building 10,000 Spoken-Dialogue Systems," Proceedings of the 1996 International Conference on Spoken Language Processing, Philadelphia, PA, 709-712, October, 1996.


RAD tutorial

Voice XML Introduction

February 3
Lectures: Frame-based Approaches, Information-state approaches

David Traum and Staffan Larsson, The Information State Approach to Dialogue Management in Current and New Directions in Discourse and Dialogue, Ed. Jan van Kuppevelt and Ronnie Smith, Kluwer, 2003, pp 325-354.

Colin Matheson, Massimo Poesio, and David Traum, Modelling Grounding and Discourse Obligations Using Update Rules, in Proceedings of the 1st Annual Meeting of the North American Association for Computational Linguistics (NAACL2000), May 2000.

Xu, W. and Rudnicky, A. Task-based dialog management using an agenda. ANLP/NAACL 2000 Workshop on Conversational Systems, May 2000, pp. 42-47.

D. Goddeau, H. Meng, J. Polifroni, S. Seneff, and S. Busayapongchai. 1996. A form-based dialogue manager for spoken language applications. In Proc. ICSLP, 1996 pp. 701--704.

V. Zue, et al., "JUPITER: A Telephone-Based Conversational Interface for Weather Information," IEEE Transactions on Speech and Audio Processing, Vol. 8 , No. 1, January 2000.
February 10
Lectures: Theorem Proving, Plan- and Agent-based approaches

Smith, D.R. Hipp, and A.W. Biermann. An Architecture for Voice Dialog Systems Based on Prolog-Style Theorem Proving. Computational Linguistics, 21:3, 1995.

Perrault and Allen A plan-based analysis of indirect speech acts. Computational Linguistics, 6:167-183, 1980

Defining a Conversational Agent. James Allen. Chapter 17 of Natural Language Understanding, Benjamin/Cummings Publishing Company, 1995.

Lecure Notes
February 17 Lectures: Theorem Proving, Plan- and Agent-based approaches

The TRAINS Project. James F. Allen et al. Journal of Experimental and Theoretical AI, 1995.

Effective Human-Computer Cooperative Spoken Dialogue: The Ags Demonstrator  (Make Corrections)  M.D. Sadek, A. Ferrieux, A. Cozannet, P. Bretier, F. Panaget, J. Simonin Proceedings of the Fourth International Conference on Spoken Language Processing (ICSLP'96)

Design Considerations on Dialgue Systems: From Theory to Technology - The Case of Artimis. David Sadek. ESCA Workshop ``Interactive Dialogue in Multi-modal Systems,'' pp. 173-187, 1999.

Rich, C.; Sidner, C.L.; Lesh, N.B., "COLLAGEN: Applying Collaborative Discourse Theory to Human-Computer Interaction", Artificial Intelligence Magazine, Winter 2001 (Vol 22, Issue 4, pps 15-25)

Rich, C.; Sidner, C.L., "COLLAGEN: A Collaboration Manager for Software Interface Agents", An International Journal: User Modeling and User-Adapted Interaction, Vol. 8, Issue 3/4, pps 315-350, 1998
February 24
Lectures:  Dialogue System Evaluation

Sikorski, T., and Allen, J. F. A task-based evaluation of the TRAINS-95 dialogue system. In Proceedings of the Workshop on Dialog Processing in Spoken Language Systems, ECAI-96 (Budapest, 1996).

Assessing Agreement on Classification Tasks: The Kappa Statistic. Jean Carletta. Computational Linguistics, 22(2):249-254, 1996.

Towards Developing General Models of Usability with PARADISE. Marilyn A. Walker, Diane J. Litman, Candace. A. Kamm and Alicia Abella. Natural Language Engineering, 2000.


Wieland Eckert, Esther Levin, and Roberto Pieraccini. Automatic evaluation of spoken dialogue systems. In TWLT13: Formal semantics and pragmatics of dialogue, pages 99--110, 1998.

M. Walker, R. Passonneau, J. Aberdeen, J. Boland, E. Bratt, J. Garofolo, L. Hirschman, A. Le, S. Lee, S. Narayanan, K. Papineni, B. Pellom, J. Polifroni, A. Potamianos, P. Prabhu, A. Rudnicky, G. Sanders, S. Seneff, D. Stallard, S. Whittaker. Cross-Site Evaluation in DARPA Communicator: The June 2000 Data Collection  Submitted to  Computer Speech and Language , 2002.


Quantitative and Qualitative Evaluation of Darpa Communicator Spoken Dialogue Systems. Marilyn Walker, Rebecca Passonneau and Julie E. Boland. Proceedings of the Association of Computational Linguistics (ACL), 2001.

Lecture Notes
March 2
Lectures:  Overview of specialized research topics

Demos: student presentations of  Assignment 2 systems
Lecture Notes

Anish: Rad, VXML, Antonio: outline of both, Sudeep: Rad
March 9
(Leader: Jens Stephan)
 Referring in Dialogue

Attention, Intentions, and the Structure of Discourse. Barbara J. Grosz, Candace L. Sidner. Computational Linguistics, volume 12, number 3, July-September 1986, pp. 175-204.

Clark, H. H. & Wilkes-Gibbs, D. (1992). Referring as a Collaborative Process. In H. H. Clark (Ed.), Arenas of Language Use, 107-143. Chicago: University Press.

Eckert, Miriam & Michael Strube (1999). Resolving discourse deictic anaphora in dialogues. In Proceedings of the 9th Conference of the European Chapter of the Association for Computational Linguistics, Bergen, Norway, 8-12 June 1999., pp. 37--44.

Donna K. Byron and James F. Allen. What's a Reference Resolution Module to do? Redefining the Role of Reference in Language Understanding Systems. In the proceedings of the 4th Discourse Anaphora and Anaphor Resolution Colloquium (DAARC2002)

Lecture Notes

March 9
(Leader: Antonio Roque)
Initiative in Dialogue

"Mixed Initiative in Dialogue: An Investigation into Discourse Segmentation" M Walker and S Whittaker  ACL 1990

Control and Initiative in Collaborative Problem Solving Dialogues
(1997)   Pamela Jordan, Barbara Di Eugenio in Working Notes of the AAAI Spring Symposium on Computational Models for Mixed Initiative

Jennifer Chu-Carroll Michael K. Brown  "Tracking Initiative in Collaborative Dialogue Interactions" ACL 1997

"Adaptive modeling of dialogue initiative,
'' P. Heeman and S. Strayer. Proceedings of the NAACL Workshop on Adaption in Dialogue Systems, pages 79--80, Pittsburgh, USA, June 2001

Lecture Notes
March 16
Spring Break

March 23 Lectures:  Dialogue Genres and Dialogue Act Taxonomies
David R. Traum, Speech Acts for Dialogue Agents, in Michael Wooldridge and Anand Rao, editors, "Foundations And Theories Of Rational Agents", Kluwer Academic Publishers, pages 169--201, 1999.

The Reliability of a Dialogue Structure Coding Scheme. Carletta, J. C., Isard, A., Isard, S., Kowtko, J., Doherty-Sneddon, G., & Anderson, A. Computational Linguistics, 23(1), 1997.

Coding Dialogs with the DAMSL Annotation Scheme. Mark Core, James Allen. AAAI Fall Symposium on Communicative Action, 1997.

An Empirical Investigation of Proposals in Collaborative Dialogues.
Barbara Di Eugenio, Pamela W. Jordan, Johanna D. Moore and Richmond H. Thomason. Proceedings of the 17th International Conference on Computational Linguistics and the 36th Meeting of the Association for Computational Linguistics (COLING-ACL).

David R. Traum, 20 Questions for Dialogue Act Taxonomies, in Journal of Semantics, 17(1):7--30, 2000.

Lecure Notes
March 23
(Leader: Viktor Rozgic) Learning Approaches to Dialogue

Levin, E., and Pieraccini, R. (1997). A stochastic model of computer-human interaction for learning dialogue strategies. In Proc. 5th European Conf. on Speech Communication and Technology.
Spoken Dialog Management Using Probabilistic Reasoning. N. Roy, J. Pineau & S. Thrun.  Proceedings of the Association for Computational Linguistics (ACL), 2000.

 MIMIC: An adaptive mixed initiative spoken dialogue system for information queries. Jennifer Chu-Carroll.  Proceedings of the 6th ACL Conference on Applied Natural Language Processing (ANLP), 2000.

 Optimizing Dialogue Management with Reinforcement Learning: Experiments with the NJFun System. Satinder Singh, Diane Litman, Michael Kearns and Marilyn Walker.  Journal of Artificial Intelligence Research (JAIR), Vol. 16, 2002.

 Designing and Evaluating an Adaptive Spoken Dialogue System. Diane J. Litman and Shimei Pan.  User Modeling and User-Adapted Interaction, to appear.
March 30
(Leader: Sudeep Gandhe)

Multi-party dialogue
David Traum, Issues in multi-party dialogues, in Advances in Agent Communication Ed. F. Dignum, Springer-Verlag LNAI 2922 pp 201-211, 2004.

 Ishizaki, M. and T. Kato (1998) Exploring the Characteristics of Multi-Party Dialogue, In Proc. of the 17th COLING-ACL'98, pp.583-589

 Carletta, J., Anderson, A. H., & Garrod, S. (2002). Seeing eye to eye: an account of grounding and understanding in work groups. Cognitive Studies: Bulletin of the Japanese Cognitive Science Society, 9(1), 1-20, March. Invited paper for the special issue entitled "Towards Sciences of Linguistic Communication". ISSN 13417924

CobotDS: A Spoken Dialogue System for Chat. Michael Kearns, Charles Isbell, Satinder Singh, Diane Litman, and Jessica Howe. AAAI 2002.
March 30
(Leader: Abe Kazemzadeh)

grounding and repair

David R. Traum Computational Models of Grounding in Collaborative Systems, in working notes of AAAI Fall Symposium on Psychological Models of Communication, p. 124-131, November, 1999.

Conversation as Action Under Uncertainty. T. Paek and E. Horvitz. Proceedings of the 16th Conference on Uncertainty in Artificial Intelligence (UAI-2000), Stanford, CA, June 2000.

Reducing Miscommunication in Spoken Human-Machine Dialogue. Dybkjaer, L., Bernsen, N.O. and Dybkjaer, H. Proc. of AAAI Workshop on Detecting, Repairing and Preventing Human-Machine Miscommunication. Portland OR, 1996.

 E. Krahmer, M. Swerts, M. Theune and M. Weegels. Error  Detection in Spoken Human-Machine Interaction. In: International Journal of Speech  Technology, 4(1):19-30, 2001.

Nakano, Y., Reinstein, G., Stocky, T., Cassell, J. (2003) "Towards a Model of Face-to-Face Grounding" Proceedings of the Annual Meeting of the Association for Computational Linguistics. July 7-12, Sapporo, Japan.
April 6
(Leader: Anish Nair)

Dialogue Act Recognition

Norbert Reithinger. Some Experiments in Speech Act Prediction. In AAAI 95 Spring Symposium on Empirical Methods in Discourse Interpretation and Generation, 1995.

Dialogue Act Tagging with Transformation-Based Learning. Ken Samuel, Sandra Carberry, and K. Vijay-Shanker. Proceedings of the 36th Meeting of the Association for Computational Linguistics (ACL), 1998.

Dialogue Act Modeling for Automatic Tagging and Recognition of Conversational Speech. Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Marie Meteer, and Carol Van Ess-Dykema. Computational Linguistics 26:3, 2000.

Wright-Hastie, H., Poesio, M., and Isard, S. 2002. Automatically predicting dialogue structure using prosodic features.  Speech Communication  , v. 36, n. 1-2, p. 63-79.
April 6
(Leader: Nishit Rathod)

dialogue tracking in other language-processing systems

 Jan Alexandersson. Plan Recognition in VERBMOBIL. Verbmobil-Report 81. DFKI GmbH. May 1995.

 Jan Alexandersson, Norbert Reithinger, Elisabeth Maier. Insights into the Dialogue Processing of VERBMOBIL.  Verbmobil-Report 191. DFKI GmbH Saarbrücken. March 1997. 

 Birte Schmitz. Collaboration in Automatic Dialogue Interpreting.  Verbmobil-Report 212. Technische Universität Berlin. August 1997.

Klaus Zechner, 2001. Automatic Generation of Concise Summaries of Spoken Dialogues in Unrestricted Domains. Proceedings of the 24th ACM-SIGIR International Conference on Research and Development in Information Retrieval, New Orleans, LA, September 2001

Alex Waibel, Michael Bett, Florian Metze, Klaus Ries, Thomas Schaaf, Tanja Schultz, Hagen Soltau, Hua Yu, and Klaus Zechner, 2001. Advances in Automatic Meeting Record Creation and Access. Proceedings of ICASSP-2001, Salt Lake City, UT, May 2001.

N. Boufaden, G. Lapalme, and Y. Bengio. Topic segmentation : First Stage of Dialogue-Based  Information extraction Process.  In Proceedings of the Natural Language Pacific Rim Symposium,  NLPRS-01, 2001

April 13
Guest Lecture:
Dr Hannes Vilhjalmsson

Embodiment (the role the body plays) in Dialogue

BEAT: the Behavior Expression Animation Toolkit, Cassell, J., Vilhjálmsson, H., and Bickmore, (2001), Proceedings of ACM SIGGRAPH 2001, Los Angeles, August 12-17, p.477-486

Cassell, J., T. Stocky, T. Bickmore, Y. Gao, Y. Nakano, K. Ryokai, D. Tversky, C. Vaucelle, H. Vilhjlmsson. MACK: Media lab Autonomous Conversational Kiosk. In Imagina 02. February 12-15, 2002. Monte Carlo.

April 13
(Leader: Michael Rushforth)

Tutorial Dialogue Systems
Intelligent Tutoring Systems with Conversational Dialogue. Arthur Graesser, Kurt VanLehn, Carolyn Rose, Pamela Jordan and Derek Harter. AI Magazine.

Evens, Martha W., Stefan Brandle, Ru-Charn Chang, Reva Freedman,  Michael Glass, Yoon Hee Lee, Leem Seop Shim, Chong Woo Woo,  Yuemei Zhang, Yujian Zhou, Joel A. Michael, and Allen A. Rovick  CIRCSIM-Tutor: An Intelligent Tutoring System  Using Natural Language Dialogue Twelfth Midwest AI and Cognitive Science Conference, MAICS 2001,  Oxford, OH, pp. 16-23.

 Teaching Tactics and Dialog in Autotutor. Arthur C. Graesser, Natalie K. Person, Derek Harter and the Tutoring Research Group. International Journal of Artificial Intelligence in Education, 2001.

 Supporting Constructive Learning with a Feedback Planner. M. G. Core, J. D. Moore, and C. Zinn, AAAI Fall Symposium on Building Dialogue Systems for Tutorial Applications, 2000.

 Using a Model of Collaborative Dialogue to Teach Procedural Tasks. Jeff Rickel, Neal Lesh, Charles Rich, Candace L. Sidner and Abigail Gertner. Proceedings of 10th International Conference on AI in Education, 2001.
April 20
Class Project Presentations (1)
Jens Stephan
Michael Rushforth
Anish Nair & Viktor Rozgic
April 27
Class Project Presentations (2)
Nishit Rathod
Antonio Roque
Sudeep Gandhe
Abe Kazemzadeh




Selected Specialty Topics



referring in dialogue: (see above)

grounding and repair:

(see above)
Dialogue Act Taxonomies:

(see above)

Dialogue Act Recognition:
(see above)
Initiative in Dialogue
(see above)
Learning Approaches to Dialogue
(see above)
Embodied Conversational Agents
   W.L. Johnson, J.W. Rickel, and J.C. Lester. Animated Pedagogical Agents: Face-to-Face Interaction in Interactive Learning Environments. International Journal of Artificial Intelligence in Education 11:47-78, 2000.

Jeff Rickel, Stacy Marsella, Jonathan Gratch, Randall Hill, David Traum, and William Swartout,  Towards a New Generation of Virtual Humans for Interactive Experiences, in IEEE Intelligent Systems, 17(4):32--38, 2002.

Tim Bickmore, Justine Cassell (in press) "Social Dialogue with Embodied Conversational Agents" In J. van Kuppevelt, L. Dybkjaer, and N. Bernsen (eds.),  Natural, Intelligent and Effective Interaction with Multimodal Dialogue Systems. New York: Kluwer Academic

Cassell, J., T. Stocky, T. Bickmore, Y. Gao, Y. Nakano, K. Ryokai, D. Tversky, C. Vaucelle, H. Vilhjlmsson. MACK: Media lab Autonomous Conversational Kiosk. In Imagina 02. February 12-15, 2002. Monte Carlo.

André, Elisabeth, Rist, Thomas, Susanne van Mulken, Martin Klesen and Stephan Baldes: The Automated Design of Believable Dialogues for Animated Presentation Teams, in: J. Cassell, J. Sullivan, S. Prevost and E. Churchill: Embodied Conversational Agents, MIT Press, pp: 220-255, 2000..
Multi-modal Dialogue
André, Elisabeth: Natural  Language in Multimedia/Multimodal Systems, Handbook of  Computational Linguistics, , R. Mitkov, 650-669, Oxford University  Press, 2003

AdApt - a multimodal conversational dialogue system in an apartment domain. Gustafson J, Bell L, Beskow J, Boye J, Carlson R, Edlund J, Granstrom, B, House D & Wiren M. Proc. of ICSLP, 2000.

The efficiency of multimodal interaction for a map-based task. Cohen, P. R., McGee, D. R., Clow, J.  Proceedings of the Applied Natural Language Processing Conference (ANLP), 2000.
Multi-party dialogue
(see above)

Discourse Structure in Dialogue
A coding manual for the Linkoping dialogue model (1998)  
 Nils Dahlbäck, Arne Jönsson

A Two-level Approach to Coding Dialogue for Discourse Structure: Activities of the 1998 DRI Working Group on Higher-level Structures   David R. Traum, Christine H. Nakatani   in proceedings of the  ACL'99 Workshop  Towards Standards and Tools for Discourse Tagging, pp 101--108,  June 1999.

dialogue tracking in other language-processing systems (machine translation, summarization/extraction)

(see above)
Tutorial Dialogue Systems
(see above)