Rapid advances in large language models (LLMs) have expanded their use across software engineering workflows, yet most prompt engineering remains ad hoc-yielding inconsistent quality, poor transferability, and inefficient iteration. This paper introduces and evaluates Preference-Driven Refinement (PDR), a structured prompt engineering method that iteratively updates prompts by incorporating user-selected preferred and non-preferred elements from model outputs. Building on in context learning, PDR uses synthetic example generation and explicit preference incorporation to overcome common challenges, such as instruction blindness, inadequate context capture, and alignment drift. We extend PDR with PDR+Critic, a multi-LLM refinement loop where a separate Critic model evaluates generated outputs, identifies strengths and weaknesses, and recommends improvements. To assess these structured refinement methods, we conduct a controlled, simulation-based study across five software engineering tasks and eight personas using GPT-5, with decoding parameters held constant to isolate methodological effects. To address evaluation circularity concerns, we employ multi-model cross-validation using independent LLM families (GPT-4o and Gemini-2.5-Pro) as evaluators Our results reveal both methodological contributions and challenges. PDR significantly reduces refinement iterations compared to ad hoc prompting (p = 0.020, FDR-corrected), demonstrating improved convergence efficiency through fewer refinement cycles. However, PDR exhibits longer total execution time due to structured processing overhead (Ad Hoc: 67.5s, PDR: 179.6s, PDR+Critic: 209.1s on average). PDR+Critic shows mixed effectiveness: while adding explicit evaluation loops, it often increases runtime and iteration count without consistent quality improvements, and both evaluators identified cases where critic feedback degraded output quality. Critically, multi-model evaluation revealed low inter-rater agreement (r = 0.044, p = 0.794) between independent evaluators, indicating that different LLM families apply fundamentally different quality standards. This finding weakens claims about quality improvements but contributes important evidence about current limitations in LLM-as-judge methodologies. Objective metrics (iteration counts, timing) remain valid and demonstrate PDR's efficiency trade-offs: fewer iterations but longer execution time per iteration. By formalizing PDR and PDR+Critic, evaluating their performance with rigorous cross-model validation, and transparently reporting both successes and limitations, this paper advances systematic prompt engineering research. Our findings reveal key trade-offs between structure and efficiency, document how multi-LLM Generator-Critic loops interact with software engineering tasks, and motivate future work on standardized, reliable evaluation frameworks for LLM-augmented software development.

Rapid gains in Large Language Model (LLM)-based tools are transforming software engineering, from auto-completing function stubs to drafting architectural RFCs. However, current use often depends on ad hoc prompting, resulting in brittle code snippets, inconsistent style guides, and unpredictable test coverage. To enable scalable and repeatable automation, systematic prompt engineering is essential for generating high-quality software artifacts (such as unit tests, refactor patches, and Application Programming Interface (API) documentation) from the same underlying model. To address this need, we propose the Preference-Driven Refinement (PDR) method for prompt engineering, designed to support automated software engineering workflows. PDR introduces an iterative loop where developers specify preferences (e.g., naming conventions, performance constraints, or security rules) after each generation. These preferences—typically captured by editing prompt phrasing or including curated examples—are encoded into subsequent prompts, enabling the model to produce outputs that adhere to project-specific standards and practices. This refinement loop creates a more automated, policy-aware interface between developers and generative models, supporting on-boarding, code review, and other software lifecycle tasks. We present empirical evaluations demonstrating how PDR leverages in-context learning and synthetic example generation to systematically improve prompt quality. Our results show that PDR reduces trial-and-error iterations and yields higher-quality outputs, though with modest increases in refinement time. These findings highlight how structured prompt refinement can help automate manual tasks in software engineering, thereby enhancing consistency, efficiency, and developer experience in AI-assisted development environments.

The assessment of programming assignments in computer science (CS) education traditionally relies on manual grading, which strives to provide comprehensive feedback on correctness, style, efficiency, and other software quality attributes. As class sizes increase, however, it is hard to provide detailed feedback consistently, especially when multiple assessors are required to handle a larger number of assignment submissions. Large Language Models (LLMs), such as ChatGPT, Claude, and Gemini, offer a promising alternative to help automate this assessment process in a consistent, scalable, and fair manner. This paper explores the efficacy of ChatGPT-4 and other popular LLMs in automating programming assignment assessment. We conduct a series of studies within multiple Java-based CS courses at Vanderbilt University, comparing LLM-generated assessments to those produced by human graders. The analysis focuses on key me-trics, such as accuracy, precision, recall, efficiency, and consistency, to identify programming mistakes based on predefined rubrics. Our findings demonstrate that LLMs improve grading objectivity and efficiency with appropriate prompt engineering and feature selection, serving as a valuable complementary tool to human graders in undergraduate and graduate CS education.

Prompt engineering for structured data is an evolving challenge as large language models (LLMs) grow in sophistication. Earlier studies, including prior work by the authors, tested only a limited set of prompts on a single model such as GPT-4o. This paper broadens the scope by evaluating six styles—JSON, YAML, CSV, function-calling APIs, simple prefixes, and a hybrid CSV/prefix—across three leading LLMs: ChatGPT-4o, Claude, and Gemini. Using controlled datasets, we benchmark accuracy, token cost, and generation time to deliver the first systematic cross-model comparison of prompt strategies for structured outputs. Our approach employs structured validation and custom Python utilities to ensure reproducibility, with results visualized through Technique vs. Accuracy, Token Cost, and Time graphs. Our analysis reveals clear trade-offs: simpler formats often reduce cost and runtime with little accuracy loss, while more expressive formats offer flexibility for complex data. These findings underscore how prompt design can be tuned to balance efficiency and versatility in real-world applications. Our results show prompt choice directly shapes both quality and efficiency. Claude consistently achieves the highest accuracy, ChatGPT-4o excels in speed and token economy, and Gemini provides a balanced middle ground. By extending beyond single-model evaluations, this study offers practical guidance for selecting prompts based on model capabilities and application demands, advancing prompt engineering with a comprehensive, multi-model framework for optimizing structured data generation.

Prompt engineering is becoming a critical skill for software developers by facilitating enhanced interactions with conversational large language models (LLMs), such as ChatGPT, Claude, and Gemini. This emerging discipline focuses on crafting prompts, which are instructions that guide LLMs in generating precise outputs, automating tasks, and ensuring adherence to specific qualitative and quantitative standards. Prompts are also a form of natural language programming that tailor the dialogue between users and LLMs, optimizing input, output, and interaction dynamics for many computational tasks, such as developing software, analyzing documents, and/or addressing cyber vulnerabilities. This paper introduces a comprehensive catalog of prompt engineering techniques---structured as a collection of patterns---aimed at addressing common challenges encountered when integrating LLMs into the software development lifecycle. These prompt patterns serve as an effective means for knowledge transfer, similar to software patterns. In particular, they provide reusable solutions to common problems faced in particular contexts, such as output generation and interaction when conversing with LLMs in the domain of software-reliant systems. This paper provides three contributions to research on---and the practice of---prompt engineering for applying LLMs to aid users performing computational tasks. First, it establishes a framework for documenting and deploying prompt patterns across various domains, focusing on enhancing LLM utility in software development endeavors. Second, it curates a catalog of prompt patterns, validated through successful application in refining LLM interactions and outputs. Third, it explores the synergistic potential of creating more complex prompts by combining multiple prompt patterns.

Generative artificial intelligence (AI) is redefining how we solve problems by loosening the coupling between traditional coding and computational thinking, which is often defined as telling computers what to do without specifying how to do it. Rather than coding Internet applications explicitly using traditional programming languages, practitioners can now leverage large language models and agentic AI systems that can act autonomously to accomplish complex tasks through natural language intent. This column examines how generative AI tools and frameworks—from automated agents like OpenAI’s Deep Research to agent orchestration platforms like Google’s Opal—are unbinding computational thinking from code syntax and making “no-code” problem-solving a reality at scale. We highlight use cases and consider opportunities like democratized access and accelerated development.

Although the field of computer vision has grown significantly due to the advent of convolutional neural networks (CNNs), electronic analysis of historical documents has experienced scant research and development attention. Recently, however, computer vision has matured to the point where it can be applied to outperform existing, specialized tools for document analysis. This paper demonstrates empirically how state-of-the-art results can be produced by implementing, training, and evaluating generic computer vision models on historical document segmentation tasks. We show the generality of our approach to document analysis and explain how innovation in this domain can arise from combining generic building blocks for computer vision.

The rapid advent of Large Language Models (LLMs), such as ChatGPT and Claude, is revolutionizing various fields, from education and healthcare to the engineering of reliable software systems. These LLMs operate through "prompts," which are natural language inputs that users employ to query and leverage the models' capabilities. Given the novelty of LLMs, the understanding of how to effectively use prompts remains largely anecdotal, based on isolated use cases. This fragmented approach limits the reliability and utility of LLMs, especially when they are applied in mission-critical software environments. To harness the full potential of LLMs in such crucial contexts, therefore, we need a systematic, disciplined approach to "prompt engineering" that guides interactions with and evaluations of these LLMs.

At the core of every modern airliner is a software-reliant fly-by-wire system that translates pilot inputs into electronic signals to control aircraft movements. Given the safety-critical nature of these systems they include architectural constructs and mechanisms to tolerate failures related to hardware (e.g., processor or sensor failures) and software (e.g., potential bug in the code). The goal is to reach the required levels of availability and integrity validated through a certification process that includes specific verification methods to discharge specific claims. Unfortunately, the different verification procedures and associated architectural constructs are typically developed independently and make independent assumptions that can contradict each other, thereby preventing the desired behavior or invalidating the assumptions and results of a given verification procedure. To help address these problems this paper presents how a new symbolic argumentation approach can be used to analyze a real flight incident (the flight CI202 incident in 2020) by automating the verification procedures and their assumptions. Our approach describes verification plans that start at the level of certification connected to automated verification analysis on architectural models. These plans are decomposed into analysis contracts that specify what claims they verify (e.g., availability of a fly-by-wire function> 99.99 % ), what analysis is used to verify the model (e.g., probabilistic Fault-Tree Analysis) and what assumptions it relies on (e.g., a function is replicated over processors that fail independently of each other). These plans are integrated into a symbolic argumentation implemented as a constraint satisfaction problem that is solved with a Satisfiability Modulo Theory (SMT) solver. The CI202 flight incident analysis is presented using an argumentation hierarchy on architectural models and the analysis of potential design issues that could explain a triple computer failure. We demonstrate how our approach can reason about early design decisions by pointing to unfulfilled assumptions, contradictions, and potential workarounds that have the potential to prevent these types of incidents.

Courses in computer science (CS) often assess student programming assignments manually, with the intent of providing in-depth feedback to each student regarding correctness, style, efficiency, and other quality attributes. As class sizes increase, however, it is hard to provide detailed feedback consistently, especially when multiple assessors are required to handle a larger number of assignment submissions. Large language models (LLMs), such as ChatGPT, offer a promising alternative to help automate this process in a consistent, scalable, and minimally-biased manner. This paper explores ChatGPT-4's scalablility and accuracy in assessing programming assignments based on predefined rubrics in the context of a case study we conducted in an upper-level undergraduate and graduate CS course at Vanderbilt University. In this case study, we employed a method that compared assessments generated by ChatGPT-4 against human graders to measure the accuracy, precision, and recall associated with identifying programming mistakes. Our results show that when ChatGPT-4 is used properly (e.g., with appropriate prompt engineering and feature selection) it can improve objectivity and grading efficiency, thereby acting as a complementary tool to human graders for advanced computer science graduate and undergraduate students.

The rise of large language models (LLMs) is revolutionizing information retrieval, question answering, summarization, and code generation tasks. However, in addition to confidently presenting factually inaccurate information at times (known as “hallucinations”), LLMs are also inherently limited by the number of input and output tokens that can be processed at once, making them potentially less effective on tasks that require processing a large set or continuous stream of information. A common approach to reducing the size of data is through lossless or lossy compression. Yet, in some cases it may not be strictly necessary to perfectly recover every detail from the original data, as long as a requisite level of semantic precision or intent is conveyed. This paper presents three contributions to research on LLMs. First, we present the results from experiments exploring the viability of “approximate compression” using LLMs, focusing specifically on GPT-3.5 and GPT-4 via ChatGPT interfaces. Second, we investigate and quantify the capability of LLMs to compress text. Third, we present two novel metrics-Exact Reconstructive Effectiveness (ERE) and Semantic Reconstruction Effectiveness (SRE)-that quantify the level of preserved intent between text compressed and decompressed by the LLMs we studied. Our initial results indicate that GPT-4 can effectively compress and reconstruct text while preserving the semantic essence of the original text, providing a path to leverage more tokens than current limits allow.

Translating natural language into Bash Commands is an emerging research field that has gained attention in recent years. Most efforts have focused on producing more accurate translation models. To the best of our knowledge, only two datasets are available, with one based on the other. Both datasets involve scraping through known data sources (through platforms like stack overflow, crowdsourcing, etc.) and hiring experts to validate and correct either the English text or Bash Commands. This paper provides two contributions to research on synthesizing Bash Commands from scratch. First, we describe a state-of-the-art translation model used to generate Bash Commands from the corresponding English text. Second, we introduce a new NL2CMD dataset that is automatically generated, involves minimal human intervention, and is over six times larger than prior datasets. Since the generation pipeline does not rely on existing Bash Commands, the distribution and types of commands can be custom adjusted. We evaluate the performance of ChatGPT on this task and discuss the potential of using it as a data generator. Our empirical results show how the scale and diversity of our dataset can offer unique opportunities for semantic parsing researchers.

This paper presents prompt design techniques for software engineering, in the form of patterns, to solve common problems when using large language models (LLMs), such as ChatGPT to automate common software engineering activities, such as ensuring code is decoupled from third-party libraries and simulating a web application API before it is implemented. This paper provides two contributions to research on using LLMs for software engineering. First, it provides a catalog of patterns for software engineering that classifies patterns according to the types of problems they solve. Second, it explores several prompt patterns that have been applied to improve requirements elicitation, rapid prototyping, code quality, refactoring, and system design.

BACKGROUND:It remains hard to directly apply deep learning-based methods to assist diagnosing essential tremor of voice (ETV) and abductor and adductor spasmodic dysphonia (ABSD and ADSD). One of the main challenges is that, as a class of rare laryngeal movement disorders (LMDs), there are limited available databases to be investigated. Another worthy explored research question is which above sub-disorder benefits most from diagnosis based on sustained phonations. The question is from the fact that sustained phonations can help detect pathological voice from healthy voice. METHOD:A transfer learning strategy is developed for LMD diagnosis with limited data, which consists of three fundamental parts. (1) An extra vocally healthy database from the International Dialects of English Archive (IDEA) is employed to pre-train a convolutional autoencoder. (2) The transferred proportion of the pre-trained encoder is explored. And its impact on LMD diagnosis is also evaluated, yielding a two-stage transfer model. (3) A third stage is designed following the initial two stages to embed information of pathological sustained phonation into the model. This stage verifies the different effects of applying sustained phonation on diagnosing the three sub-disorders, and helps boost the final diagnostic performance. RESULTS:The analysis in this study is based on clinician-labeled LMD data obtained from the Vanderbilt University Medical Center (VUMC). We find that diagnosing ETV shows sensitivity to sustained phonation within the current database. Meanwhile, the results show that the proposed multi-stage transfer learning strategy can produce (1) accuracy of 65.3% on classifying normal and other three sub-disorders all at once, (2) accuracy of 85.3% in differentiating normal, ABSD, and ETV, and (3) accuracy of 77.7% for normal, ADSD and ETV. These findings demonstrate the effectiveness of the proposed approach.

Prompt engineering is an increasingly important skill set needed to converse effectively with large language models (LLMs), such as ChatGPT. Prompts are instructions given to an LLM to enforce rules, automate processes, and ensure specific qualities (and quantities) of generated output. Prompts are also a form of programming that can customize the outputs and interactions with an LLM. This paper describes a catalog of prompt engineering techniques presented in pattern form that have been applied to solve common problems when conversing with LLMs. Prompt patterns are a knowledge transfer method analogous to software patterns since they provide reusable solutions to common problems faced in a particular context, i.e., output generation and interaction when working with LLMs. This paper provides the following contributions to research on prompt engineering that apply LLMs to automate software development tasks. First, it provides a framework for documenting patterns for structuring prompts to solve a range of problems so that they can be adapted to different domains. Second, it presents a catalog of patterns that have been applied successfully to improve the outputs of LLM conversations. Third, it explains how prompts can be built from multiple patterns and illustrates prompt patterns that benefit from combination with other prompt patterns.

Blockchain and distributed ledger technologies (DLT) are emerging decentralized infrastructures touted by researchers to improve existing systems that have been limited by centralized governance and proprietary control. These technologies have shown continued success in sustaining the operational models of modern cryptocurrencies and decentralized finance applications (DeFi). These applications has incentivized growing discussions in their potential applications and adoption in other sectors such as healthcare, which has a high demand for data liquidity and interoperability. Despite the increasing research efforts in adopting blockchain and DLT in healthcare with conceptual designs and prototypes, a major research gap exists in literature: there is a lack of design recommendations that discuss concrete architectural styles and domain-specific considerations that are necessary for implementing health data exchange systems based on these technologies. This paper aims to address this gap in research by introducing a collection of design patterns for constructing blockchain and DLT-based healthcare systems that support secure and scalable data sharing. Our approach adapts traditional software patterns and proposes novel patterns that take into account both the technical requirements specific to healthcare systems and the implications of these requirements on naive blockchain-based solutions.

Question-answer is a paradigm that seeks to provide automated responses to queries posed in natural language utilizing a body of textual content as the source of the answers. A key research challenge is how the changes in question formulation affect the stability of current question-answer transformer models. This paper conducts a preliminary analysis of the stability of question-answer transformer models in the medical domain when the same question is asked in different orders or with other semantically identical variations. The results from our experiments demonstrate that the arrangement of words influences the outcome and consistency of answers from transformer models.

Research in the past several years has boosted the performance of automatic speaker verification systems and countermeasure systems to deliver low Equal Error Rates (EERs) on each system. However, research on joint optimization of both systems is still limited. The Spoofing-Aware Speaker Verification (SASV) 2022 challenge was proposed to encourage the development of integrated SASV systems with new metrics to evaluate joint model performance. This paper proposes an ensemble-free end-to-end solution, known as Spoof-Aggregated-SASV (SA-SASV) to build a SASV system with multi-task classifiers, which are optimized by multiple losses and has more flexible requirements in training set. The proposed system is trained on the ASVSpoof 2019 LA dataset, a spoof verification dataset with small number of bonafide speakers. Results of SASV-EER indicate that the model performance can be further improved by training in complete automatic speaker verification and countermeasure datasets.