ESSLLI2022 Advanced Course on
Computational Models of Grounding in Dialogue

Instructor: David Traum

Abstract

Grounding is the process by which participants in a conversation establish new common ground. This process includes not just transmission of declarative utterances, but inferential and feedback processes. This process is also of critical importance to artificial dialogue systems, which have additional challenges of imperfect input recognition and limited ontologies and inferential ability. In this course we will review models and uses for common ground in pragmatics and computational agent theories, and then examine a variety of proposals of how common ground can be established. These proposals include both descriptive analyses of behavior, as well as generative proposals that can be used by computational systems engaged in dialogue to decide on next moves. We will also look at multimodal grounding, and advanced topics, including multiparty grounding, incremental grounding and degrees of grounding, as well as how grounding models have been used for studying other social phenomena.

Pre-Requisites

No formal prerequisites, but some familiarity with dialogue and pragmatics will be helpful. The content is directed primarily toward PhD students, but will be accessible to others.

Syllabus (Lecture Notes in links for each class below)

Lecture 1: Common Ground
Preliminaries: representation, agency, mental state, communication, definitions and uses for common ground. Formalizations of common ground. Achieving common ground. Clark's Grounding models (Definite reference, referring as a collaborative process, contributing to discourse).
Lecture 2: Feedback and Repair, Early Computational Grounding Models
Computational Models of speech acts. Multifunctionality of Utterances. Multi-level dialogue acts. Errors in communication. Positive and negative feedback. Computational approaches to error and repair.
Lecture 3: Grounding acts model, Multimodal grounding
The role of communication media. Grounding in multimodal situations. Within and cross-modal grounding.
Lecture 4: Multimodal & Multiparty grounding
Gaze and gesture. Participant roles. Multiparty grounding.

Bibliography

J. Allwood, Nivre, J. and Ahlsén, E. "On the Semantics and Pragmatics of Linguistic Feedback" in Journal of Semantics, 1992
Allwood, J., & Cerrato, L. A Study of Gestural Feedback Expressions. First Nordic Symposium on Multimodal Communcation-. Paggio P. Jokinen, K. Jönsson. A. (eds). Copenhagen 23-24. September 2003, pp 7-22
Apriliani, Hany, and Ahmad Bukhori Muslim. "Grounding in Online Communication." Thirteenth Conference on Applied Linguistics (CONAPLIN 2020). Atlantis Press, 2021.
M. Baker, T. Hansen, R. Joiner and D. Traum, ``The role of grounding in collaborative learning tasks'', in Pierre Dillenbourg, editor, ``Collaborative Learning: Cognitive and Computational Approaches'', Elsevier Science Publishers B. V., pages 31--63, 1999.
Barwise, Jon. "Three views of common knowledge." Proceedings of the 2nd conference on Theoretical aspects of reasoning about knowledge. 1988.
Bavelas, Janet Beavin; De Jong, Peter; Smock Jordan, Sara; and Korman, Harry (2014) "The theoretical and research basis of co-constructing meaning in dialogue," Journal of Solution Focused Practices: Vol. 1 : Iss. 2 , Article 3.
Bavelas, J. B., De Jong, P., Korman, H., & Jordan, S. S. (2012, September). Beyond back-channels: A three-step model of grounding in face-to-face dialogue. In Proceedings of Interdisciplinary Workshop on Feedback Behaviors in Dialog(pp. 5-6).
Benotti, Luciana, and Patrick Blackburn. "Grounding as a collaborative process." In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 515-531. 2021.
Bibyk, Sarah A., Leslie M. Blaha, and Christopher W. Myers. "How Packaging of Information in Conversation Is Impacted by Communication Medium and Restrictions." Frontiers in Psychology 12 (2021): 1096.
Boye J., Fredriksson M., Goetze J., Gustafson J., Koenigsmann J. (2014) Walk This Way: Spatial Grounding for City Exploration. In: Mariani J., Rosset S., Garnier-Rizet M., Devillers L. (eds) Natural Interaction with Robots, Knowbots and Smartphones. Springer, New York, NY.
Brennan, S. 1990. Seeking and Providing Evidence for Mutual Understanding. Ph.D. Dissertation, Stanford University.
Mark Buckley and Magdalena Wolska. A Grounding Approach to Modelling Tutorial Dialogue Structures. In Jonathan Ginzburg, Pat Healey and Yo Sato, editors, Proceedings of LONDIAL 2008, the 12th Workshop on the Semantics and Pragmatics of Dialogue, pages 15-22, London, UK, 2008.
Cahn, J. E., & Brennan, S. E. (1999). A psychological model of grounding and repair in dialog. Proceedings, AAAI Fall Symposium on Psychological Models of Communication in Collaborative Systems (pp. 25-33). North Falmouth, MA: American Association for Artificial Intelligence.
Carbary, K., & Tanenhaus, M. (2011). Conceptual pacts, syntactic priming, and referential form. In Proceedings of the CogSci Workshop on the Production of Referring Expressions: Bridging the Gap Between Computational, Empirical and Theoretical Approaches to Reference (PRE-CogSci 2011) (pp. 1-6).
Carletta, J., Anderson, A. H., & Garrod, S. (2002). Seeing eye to eye: an account of grounding and understanding in work groups. Cognitive Studies: Bulletin of the Japanese Cognitive Science Society, 9(1), 1-20, March. Invited paper for the special issue entitled "Towards Sciences of Linguistic Communication".
Chai, Joyce Y., Rui Fang, Changsong Liu, and Lanbo She. "Collaborative language grounding toward situated human-robot dialogue." AI Magazine 37, no. 4 (2016): 32-45.
Chandu, Khyathi Raghavi, Yonatan Bisk, and Alan W. Black. "Grounding `Grounding' in NLP." Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. 2021.
Clark, H. H. (1996). Using language. Cambridge: Cambridge University Press.
Clark, H. H., and Brennan, S. A. (1991). Grounding in communication. In L.B. Resnick, J.M. Levine, & S.D. Teasley (Eds.). Perspectives on socially shared cognition . Washington: APA Books.
Clark, H. H., & Marshall, C. R. (1981). Definite reference and mutual knowledge. In A. K. Joshi, B. Webber, & I. Sag (Eds.), Elements of discourse understanding (pp. 10-63). Cambridge: Cambridge University Press.
Clark, Herbert H., and Edward F. Schaefer. "Collaborating on contributions to conversations." Language and cognitive processes 2.1 (1987): 19-41.
Clark, H. H., & Schaefer, E. F. (1989). Contributing to discourse. Cognitive Science , 13, 259-294.
Clark, Herbert H., and Edward F. Schaefer. "Dealing with overhearers." Arenas of language use (1992): 248-297.
Clark, H. H., & Wilkes-Gibbs, D. (1986). Referring as a collaborative process. Cognition , 22, 1-39.
Cohen, P. R., & Levesque, H. J. (1990, June). Performatives in a rationally based speech act theory. In 28th Annual Meeting of the Association for Computational Linguistics (pp. 79-88).
Pierre Dillenbourg and David Traum, Sharing solutions: persistence and grounding in multi-modal collaborative problem solving. in Journal of the Learning Sciences, 15:121--151, 2006.
Di Maro, Maria. "Computational Grounding: An Overview of Common Ground Applications in Conversational Agents." IJCoL. Italian Journal of Computational Linguistics 7.7-1, 2 (2021): 133-156.
Endsley, Mica R. "Toward a theory of situation awareness in dynamic systems." Human factors 37.1 (1995): 32-64.
Fusaroli, Riccardo, Joanna Raczaszek-Leonardi, and Kristian Tylén. "Dialog as interpersonal synergy." New Ideas in Psychology 32 (2014): 147-157.
Haber, J., Baumgärtner, T., Takmaz, E., Gelderloos, L., Bruni, E., & Fernández, R. (2019, July). The PhotoBook Dataset: Building Common Ground through Visually-Grounded Dialogue. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 1895-1910).
Harnad, Stevan. "The symbol grounding problem." Physica D: Nonlinear Phenomena 42.1-3 (1990): 335-346.
Eugenia Hee, Ron Artstein, Su Lei, Cristian Cepeda and David Traum, "Assessing Differences in Multimodal Grounding with Embodied and Disembodied Agents'', in proceedings of 5th European and 8th Nordic Symposium on Multimodal Communication, 2017.
Ho, Wan Ching. (2020) "Hello Siri, Why Don’t You Understand?–A Study on Grounding of Human and Agent Interlocutors in Dialogue." WeSSLLI student session.
Homaeian, Leila, James R. Wallace, and Stacey D. Scott. "Joint Action Storyboards: A Framework for Visualizing Communication Grounding Costs." Proceedings of the ACM on Human-Computer Interaction 5.CSCW1 (2021): 1-27.
Hough, Julian, and David Schlangen. "Investigating fluidity for human-robot interaction with real-time, real-world grounding strategies." Proceedings of the 17th Annual SIGdial Meeting on Discourse and Dialogue. 2016.
Y. Katagiri and A. Shimojima. Display acts in grounding negotiations. In Proceedings of Gotalog 2000, the 4th Workshop on the Semantics and Pragmatics of Dialogue, pages 195–198, 2000.
Seiya Kawano, Koichiro Yoshino , David Traum, Satoshi Nakamura (2021) “Dialogue Structure Parsing on Multi-Floor Dialogue Based on Multi-Task Learning” presented at Robotdial Workshop, January 2021.
Kontogiorgos, Dimosthenis, Andre Pereira, and Joakim Gustafson. "Grounding behaviours with conversational interfaces: effects of embodiment and failures." Journal on Multimodal User Interfaces 15.2 (2021): 239-254.
Larsson, Staffan, Christine Howes, and Anastasia Bondarenko. (2020) "Could you spell that again please? Towards a Formal Model of Grounding in Directory Enquiries." First AISB Symposium on Conversational AI (SoCAI)
Le Bail, C., Baker, M. J., Détienne, F., Bernard, F. X., Chartofylaka, L., & Forissier, T. (2021). Grounding and knowledge elaboration across cultural and geographical contexts: An exploratory case study. Learning, Culture and Social Interaction, 28, 100477.
David K. Lewis. Convention: A Philosophical Study. Harvard University Press, 1969.
David Lewis(1979). Scorekeeping in a language game. In Semantics from different points of view (pp. 172-187). Springer, Berlin, Heidelberg.
Marge, Matthew, and Alexander Rudnicky. "Miscommunication recovery in physically situated dialogue." Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue. 2015.
Colin Matheson, Massimo Poesio, and David Traum, Modelling Grounding and Discourse Obligations Using Update Rules, in Proceedings of the 1st Annual Meeting of the North American Association for Computational Linguistics (NAACL2000), May 2000.
Mathieu, J. E., Heffner, T. S., Goodwin, G. F., Salas, E., & Cannon-Bowers, J. (2000). The influence of shared mental models on team process and performance. Journal of Applied Psychology, 85(2), 273-283.
Susan W. McRoy and Graeme Hirst. The repair of speech act misunderstandings by abductive inference. Computational Linguistics, 21(4):5–478, 1995.
I Mushin, L Stirling, J Fletcher, R Wales (2003) Discourse structure, grounding, and prosody in task-oriented dialogue Discourse Processes 35 (1), 1-31
Nakano, Y., Reinstein, G., Stocky, T., Cassell, J. (2003) "Towards a Model of Face-to-Face Grounding" Proceedings of the Annual Meeting of the Association for Computational Linguistics. July 7-12, Sapporo, Japan.
Nonose, Kohei, Taro Kanno, and Kazuo Furuta. "An evaluation method of team situation awareness based on mutual belief." Cognition, Technology & Work 12.1 (2010): 31-40.
David G. Novick, Iván Gris: Grounding and Turn-Taking in Multimodal Multiparty Conversation. HCI (4) 2013: 97-106
Novick, D., and Sutton, S. (1994). An empirical model of acknowledgment for spoken-language systems, Proceedings of ACL-94, Las Cruces, NM, June, 1994, 96-101.
Novick, D., and Ward, K., (1993). Mutual beliefs of multiple conversants: A computational model of collaboration in air traffic control, Proc. of AAAI'93, Washington, DC, July, 1993, 196-201.
Novick, D., Walton, L., and Ward, K. (1996). Contribution graphs in multiparty conversations, Proceedings of the International Symposium on Spoken Dialogue (ISSD-96), Philadelphia, PA, October, 1996, 53-56.
T. Paek and E. Horvitz. “Conversation as Action Under Uncertainty” Proceedings of the 16th Conference on Uncertainty in Artificial Intelligence (UAI-2000), Stanford, CA, June 2000.
Pickering, M. J., & Garrod, S. (2004). Toward a mechanistic psychology of dialogue. Behavioral and brain sciences, 27(2), 169-190.
Pollock, J. L. (1987). Defeasible reasoning. Cognitive science, 11(4), 481-518.
Antonio Roque and David Traum, Degrees of Grounding Based on Evidence of Understanding In proceedings of The 9th SIGdial Workshop on Discourse and Dialogue (SIGdial 2008), June, 2008.
Antonio Roque and David Traum, Improving a Virtual Human Using a Model of Degrees of Grounding, in proceedings of International Joint Conerence on Artificial Intelligence IJCAI-09, Pasadena, CA.
Rothwell, C. D., Shalin, V. L., and Romigh, G. D. (2017). "Quantitative models of human-human conversational grounding processes," in Gunzelmann G., Howes A., Tebrink T., Davelaar E. J. editors, Proceedings of the 39th annual conference of the cognitive science society. (Austin, TX: Cognitive Science Society).
Rothwell, Clayton D., Valerie L. Shalin, and Griffin D. Romigh. "Comparison of Common Ground Models for Human--Computer Dialogue: Evidence for Audience Design." ACM Transactions on Computer-Human Interaction (TOCHI) 28.2 (2021): 1-35.
Stephen R. Schiffer. Meaning. Oxford University Press, 1972.
Schober, Michael F., and Herbert H. Clark. "Understanding by addressees and overhearers." Cognitive psychology 21.2 (1989): 211-232.
Scheutz M, DeLoach SA, Adams JA. A Framework for Developing and Using Shared Mental Models in Human-Agent Teams. Journal of Cognitive Engineering and Decision Making. 2017;11(3):203-224.
Shridhar, Mohit, Dixant Mittal, and David Hsu. "INGRESS: Interactive visual grounding of referring expressions." The International Journal of Robotics Research 39.2-3 (2020): 217-232.
Stalnaker, Robert. 1973. Presuppositions. Journal of Philosophical Logic 2:447-457.
Robert Stalnaker. 2002. "Common Ground." Linguistics and Philosophy 25 (5/6): 701–21.
Stirling, L., Mushin, I., Fletcher, J., & Wales, R. (2000). The nature of common ground units: an empirical analysis using map task dialogues. In Gotalog 2000, 4th Workshop on the Semantics and Pragmatics of Dialogue (pp. 159-165).
Atsue Takeoka and Atsushi Shimojima, "Grounding-Styles of Aged Dyads: an Exploratory Study." Third SIGdial Workshop on Discourse and Dialogue: Proceedings of the Workshop. Association for Computational Linguistics, http://www.aclweb.org, 188-195, 2002. David Traum, A Computational Theory of Grounding in Natural Language Conversation, TR 545 and Ph.D. Thesis, Computer Science Dept., U. Rochester, December 1994.
David Traum, A Computational Theory of Grounding in Natural Language Conversation, TR 545 and Ph.D. Thesis, Computer Science Dept., U. Rochester, December 1994.
David R. Traum Computational Models of Grounding in Collaborative Systems, in working notes of AAAI Fall Symposium on Psychological Models of Communication, p. 124-131, November, 1999.
David Traum and Peter Heeman. Utterance Units and Grounding in Spoken Dialogue. Proceedings ICSLP, October, 1996.
David Traum, Cassidy Henry, Stephanie Lukin, Ron Artstein, Felix Gervits, Kimberly Pollard, Claire Bonial, Su Lei, Clare Voss, Matthew Marge, Cory Hayes and Susan Hill ``Dialogue Structure Annotation for Multi-Floor Interaction'' in Proceedings of the LREC 2018 conference.
David R. Traum and James F. Allen, A Speech Acts Approach to Grounding in Conversation, In Proceedings 2nd International Conference on Spoken Language Processing (ICSLP-92), pages 137-40, October 1992.
David R. Traum and Christine H. Nakatani, A Two-level Approach to Coding Dialogue for Discourse Structure: Activities of the 1998 Working Group on Higher-level Structures, in proceedings of the ACL'99 Workshop Towards Standards and Tools for Discourse Tagging, pp 101--108, June 1999.
David Traum and Jeff Rickel, Embodied Agents for Multi-party Dialogue in Immersive Virtual World in proceedings of the first International Joint Conference on Autonomous Agents and Multi-agent Systems (AAMAS 2002), pp. 766-773, July 2002.
Udagawa, T., & Aizawa, A. (2019). A Natural Language Corpus of Common Grounding under Continuous and Partially-Observable Context. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01), 7120-7127
Udagawa, Takuma, and Akiko Aizawa. "An annotated corpus of reference resolution for interpreting common grounding." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 05. 2020.
Udagawa, Takuma, and Akiko Aizawa. "Maintaining Common Ground in Dynamic Environments." Transactions of the Association for Computational Linguistics 9 (2021): 995-1011.
Umata, Ichiro, Koki Ijuin, Tsuneo Kato, and Seiichi Yamamoto. "Floor Apportionment Function of Speaker's Gaze in Grounding Acts." In Adjunct of the 2019 International Conference on Multimodal Interaction, pp. 1-7. 2019.
Visser, T., Traum, D., & DudeVault, D. (2014). A model for incremental grounding in spoken dialogue systems. Journal on Multimodal User Interfaces, 8(1), 61-73.
Von Fintel, Kai. "What is presupposition accommodation, again?." Philosophical perspectives 22 (2008): 137-170.
Yaghoubzadeh R., Pitsch K., Kopp S. (2015) Adaptive Grounding and Dialogue Management for Autonomous Conversational Assistants for Elderly Users. In: Brinkman WP., Broekens J., Heylen D. (eds) Intelligent Virtual Agents. IVA 2015. Lecture Notes in Computer Science, vol 9238. Springer,
Naomi Yamashita, Rieko Inaba, Hideaki Kuzuoka, Toru Ishida: Difficulties in establishing common ground in multiparty groups using machine translation. CHI 2009: 679-688

ESSLLI2022 Advanced Course on Computational Models of Grounding in Dialogue