Towards automatic annotation of collaborative problem-solving skills in technology-enhanced environments

Background Collaborative problem solving (CPS) is important for success in the 21st century, especially for teamwork and communication in technology-enhanced environments. Measurement of CPS skills has emerged as an essential aspect in educational assessment. Modern research in CPS relies on theory-driven measurements that are usually carried out as manual annotations over recorded logs of collaborative activities. However, manual annotation has limited scalability and is not conductive towards CPS assessments at scale. Objective We explore possibilities for automated annotation of actions in collaborative-teams, especially chat messages. We evaluate two approaches that employ machine learning for automated classification of CPS events. Method Data were collected from engineering, physics and electronics students' participation in a simulation-based task on electronics concepts, in which participants communicated via text-chat messages. All task activities were logged and time stamped. Data have been manually classified for the CPS skills, using an ontology that includes both social and cognitive dimensions. In this article, we describe computational linguistic methods for automatically classifying the CPS skills from logged data, with a view towards automating CPS assessments. Results We applied two machine learning methods to our data. A Naive Bayes classifier has been previously used in CPS research, but it is only moderately successful on our data. We also present a k-nearest-neighbours (kNN) classifier that uses distributional semantic models for measuring text similarity. This classifier shows strong agreement between automated and human annotations. The study also demonstrates that automatic spelling correction and slang normalization of chat texts are useful for accurate automated annotation. Implications Our results suggest that a kNN classifier can be very effective for accurate annotation of CPS events. It achieves reasonably strong results even when trained on only half of the available data. This shows a promise towards reduction of manual data annotation for CPS measurement.