The Chatbot Pilot Data

During the second half of March (except for University of Trento, which has recently wrapped up their data collection), the first round of WeNet Pilots unfolded in four locations across the globe: National University of Mongolia, University of Aalborg (Denmark), the Universidad Católica “Nuestra Señora de la Asunción” (Paraguay) and the London School of Economics (UK).

Scenario and focus

The pilots ran on a common scenario ‘Ask for Help’ – student participants were invited to pose and answer questions to each other via a chatbot application, supported by Telegram.

Denmark	UK	Italy	Mongolia	Paraguay	Total
34	47	53	39	22	195

While the tone, content, and ‘interaction style’ varied across pilot locations, common themes could be identified.

Participants’ questions could be classed into question types or categories; a) information, b) getting to know the community, c) initiating a connection, d) opinions and experiences, e) suggestions, f) personal and sensitive, and, finally, g) academic. 

Topics of interests (or domains) covered many areas including; food, exercise, careers, work and internships, foreign languages, cultural activities, current affairs, relationships and academic issues. These themes were more or less prevalent depending on the pilot site.

At the end of data collection, an evaluation study was conducted to assess the chatbot experiment. The evaluation gathered both qualitative (focus groups) and quantitative (survey) insights.

What results did the programme evaluation elicit?

The exit survey points to a positive overall experience across all pilot sites. On average, students found chatbot 1.0 easy to use, with an overall mean score of 4.06 (out of a 5-point scale where 1 = strongly disagree and 5 = strongly agree). Users also felt comfortable using chatbot (over 85%), were interested in the experience (88%), and thought this channel was useful to provide help to others (86%).

Focus groups pointed to potential for improvement; in particular, a widespread desire for increased interactivity (the option to follow up a conversation and to see others’ answers would be particularly welcome), improved efficiency in targeting those selected to respond to questions, and clearer definition of mechanisms and purposes behind the incentive scheme.