
Automatic annotation of temporal speech features can scale second language (L2) fluency research. However, existing approaches are validated mainly on English and vary in accuracy across measures. We present a cross-lingual pipeline that computes utterance fluency measures from audio and operationalizes the Vercellotti and Hall (2024) clause framework. We validate it on 40 L2 English and 40 L2 Japanese storytelling monologs. Evaluation follows a staged design: clause-boundary agreement, pause location classification, and concurrent validity with manual reference measures. Reference clause annotations combined independent double coding, large language model-assisted segmentation, coder review, and an audio check. In both languages, the clause boundaries placed by the pipeline agreed well with the manual reference. The pipeline’s classification of pauses as mid-clause or end-clause agreed closely with the reference in English and well in Japanese. All nine automatic measures correlated strongly with the manual measures in both languages. In a matched English comparison, the pipeline measured articulation rate and pause ratio with much smaller errors than existing acoustic and transcription-based tools. Among the clause-based measures, mid-clause pause ratio gained the most over the transcription-based tool. Speech rate, mean length of run, and overall pause ratio were the most robust measures. End-clause pause ratio was the weakest. These findings are limited to monologic storytelling. Repair fluency measures were not evaluated. Overall, the results support selective automation of clause-based fluency measurement, with human review concentrated on clause-end placement and related pause measures.
Methodological conversations around the use of AI have centered on authors, reviewers, editors, educators, and students, leaving one group comparatively unaddressed: research participants. Especially in the context of open-ended online survey designs, this gap presents methodological challenges for which our field currently lacks guidance. Drawing on my personal experience with a grant-funded online survey study, I document how suspicion of AI use led me to reduce the analytic sample from 259 to 29 participants, excluding 180 participants on the basis of that suspicion and 50 for missing or non-English data, and not meeting eligibility criteria. I report the decisions I made in the face of uncertainty, including the use of a multi-stage screening procedure, data flagging, and an expert rater panel, as well as their consequences. Rather than presenting my case as a model for handling participant AI use, I use it to highlight three methodological challenges: uncertainty about authorship attribution; how participant AI use may affect data quality, validity, and authenticity; and the consequences of researcher-driven adjudication and exclusion decisions for study validity. I conclude by sharing suggestions for how current research practice can meet these challenges and identifying the empirical research needed to develop field-wide, evidence-based guidelines for best practices surrounding participant AI use in online survey research and beyond.