A Paradigm Shift from “Human Writing” to “Machine Generation” in Personality Test Development: an Application of State-of-the-Art Natural Language Processing

Natural language processing (NLP) techniques have become increasingly popular in areas of psychological assessment. Recently, researchers have sought to use NLP techniques for automatic item generation (AIG) in the personality domain. Nevertheless, NLP-based approaches to personality AIG are new and many questions are still unanswered. Our research builds upon previous illustrations of AIG in personality in several ways. First, we applied a prompt-based generative pre-trained transformer 3 (GPT-3) to generate personality items. This approach provides several practical advantages for researchers and practitioners compared to previous AIG approaches. Second, we thoroughly compared various psychometric properties between machine- and human-authored personality items. Lastly, we examined the measurement invariance of machine-authored personality items between gender groups to ensure fair organizational decision-making. Results revealed that the machine-authored personality items provided good psychometric properties and little measurement biases between genders. Practical considerations, contributions, and future research directions of the AIG technique for non-cognitive tests were discussed.