Because today’s chatbots are very specialized, Google is working on the development of an almost universal conversational agent. The goal: to make him able to discuss anything and everything. In this exercise, Meena gets closer to humans.
Researchers from the Google Brain team , specializing in deep learning , have just presented Meena , a chatbot capable of holding conversations in a much more natural way than other conversational agents. They call it a conversational neural model that was trained end-to-end with 2.6 billion parameters.
Meena is based on the Evolved Transformer seq2seq architecture , and consists of a single encoding block which allows it to understand the other party’s sentences and the subject of the discussion, and 13 decoding blocks to formulate its response. The researchers used 341 GB of text from social media conversations to train the chatbot. Their goal was to reduce perplexity, that is, uncertainty in predicting the next word in conversation.
A score close to human-to-human exchanges
One of the problems in evaluating Meena’s performance was the lack of reliable metrics. The researchers therefore created a new measure called Sensibleness and Specificity Average (SSA). To do this, they collected a hundred conversations for each agent, made up of 1,600 to 2,400 rounds (or responses). Volunteers then had to write down each response, if it seemed reasonable and credible in the context, and if it was context specific and not a boilerplate response like “yes” or “me too”. The SSA is the average of the scores on these two parameters.
With this measure, Meena largely beat the other chatbots , with a score of 79% against 56% for the best. It even comes close to humans, 86% rated. The researchers also discovered that there is an inverse correlation between perplexity and SSA, which means that perplexity could be used to automatically assess the performance of conversational agents, thereby accelerating their development.