Bot With Interactions: Improving GitHub Pull-Request Feedback Through Two-Way Communication.

Projects in our software-engineering course require students to submit GitHub pull requests to an open-source software project containing more than 30,000 lines of source code. Once submitted, code is checked by a static code analyzer, as well as a bot named Danger Bot. The Danger Bot is able to detect more than 40 programmable system-specific guideline violations (which are different from static analysis rules). Although use of the Danger Bot was associated with a decrease of 40% in guideline violations, it also emitted some false positives. Neither staff nor students could change the feedback given by the Danger Bot on pull-request pages, because there was only one way communication, from bot to humans. In this paper, we discuss how we bypass the limitations of the Danger Bot by introducing the Danger Bot 2.0 (hereinafter, "the bot") with two way communication between humans and the bot. In this way, if students or teaching staff find some false positives produced by the bot, they can tell the bot about this. After teaching staff cancel a particular message as a false positive, the bot will not report that message again until it is re-enabled by the staff. We conducted a pilot study for the bot with two-way communication. Results showed that the bot with two-way communication was associated with a significant 70% decrease of unresolved guideline violations and the elimination of 100% false positives in them. The majority of two-way communications happened between the teaching staff and the bot, especially when teaching staff cancelled all the false positive violations after inspection. There was only one communication between students and the bot.