Comparing Few-Shot Learning with GPT-3 to Traditional Machine Learning Approaches for Classifying Teacher Simulation Responses

Teacher educators use digital clinical simulations (DCS) to provide improvisation opportunities within low-stakes classroom environments. In this study, we experimented with GPT-3 and few-shot learning to examine if it could be used with open-text DCS responses. We found that GPT-3 performed substantially worse than traditional machine learning (ML) models even on the same-sized training sets. However, the performance of GPT-3 decreased only marginally compared to traditional ML models with a training set of 20 examples (-0.06). Traditional ML models generally performed well and in some cases had similar performance to the human baseline. Future research will examine whether changes to labeling procedures or fine-tuning with existing data can improve the performance of GPT-3 with DCSs.