Moderating Democratic Discourse with LLMs

Many social media platforms promote political polarization by creating online echo chambers where people are only exposed to information confirming their beliefs. Newer systems such as Polis and Kialo aim to foster constructive conversations and teach critical reasoning skills. However, these platforms rely heavily on human moderators to manage discussions effectively. This paper examines the effectiveness of large language models (LLMs) as moderators on Polis, an open-source, real-time system designed for democratic discourse. We evaluate the F1 score of various prompting techniques at classifying five Polis datasets labeled by human moderators. Our findings indicate that LLMs are robust to different prompting strategies and produce minimal false positives. While LLMs come with certain risks, we argue that they can be valuable tools to support human moderators, enabling broader participation in democratic discourse.