Evaluation Baseline for Open-Domain Chatbots Based on Publicly Available Models and Apps

Evaluation of conversational chatbots is an open research problem within the NLP community. Previous studies tested various automatic metrics for a proxy of human evaluation of chatbot’s naturalness. While several popular automatic metrics correlated poorly with human judgment (Liu et al., 2016), perplexity demonstrated promising results (Adiwardana et al., 2020). However, the notion of naturalness in the aforementioned study did not include a set of essential human-like conversation attributes, e.g., entertainment or empathy, as suggested by the PEACE model (Svikhnushina and Pu, 2021). The aim of this project is to create a benchmark dataset of conversations with sufficient coverage of the PEACE constructs that could be further used for evaluation and comparison of different conversational models as well as testing of novel evaluation metrics.
The student is expected to:

• Survey existing popular open-domain chatbots whose conversational responses could serve as a reasonable baseline
• Create a benchmark dataset of conversations in a similar way as described in (Adiwardana et al., 2020)
• Obtain human judgments for different conversational aspects for the curated data via crowdsourcing