How Do Programmers Evaluate AI-Generated Code?

Background] Applications of generative artificial intelligence are being proposed at a rapid pace to support various software engineering tasks. Although versatile, the performance of the tools depends on multiple factors that are not always visible to the users, and they tend to camouflage their failure points (i.e., to “hallucinate”). In programming, the existing literature suggests that they shift the effort from writing to reading, comprehending, evaluating, and repairing generated code. The tools can also enable outsourcing these efforts, even when it might be unwise. [Aims] The broad aim of this research is to investigate the interaction between programmers and code generation tools to understand how the tools support the needs of software practitioners. The specific approach of this research is to examine how, when, for what, to what extent, and to what effect programmers read, comprehend, evaluate, repair, and trust AI-generated code, with the goal of theory building for predicting how these will change over time. [Method] Research is expected to consist mainly of qualitative and quantitative analyses of programmers' behaviour and trust in realistic contexts. Research methods include practitioner surveys and interviews, a scoping review, and observational field studies. [Results] The expected results include descriptions of the ways in which programmers evaluate AI-generated code in different scenarios, and a theory on how programmers' evaluation-related behaviour and trust change as they become more skilled users of the technology and/or as the technology matures, with the broader goal of supporting future research on the nature of generative AI assistance.