AI-based code generators have transformed offensive security by translating natural language descriptions into executable exploits. However, the semantic variability and implicit assumptions in NL descriptions limit their robustness and usability in this domain. This study evaluates nine state-of-the-art DL models, including fine-tuned models and instruction-tuned LLMs, under varying contextual information conditions to assess their ability to handle ambiguity, leverage useful context, and filter irrelevant information. Using a manually-curated dataset of real-world shellcodes and rigorous evaluations, we find that fine-tuned encoder-decoder models excel with related context, decoder-only indirectly benefit from unrelated context to better comprehend the task at hand, while instruction-tuned LLMs struggle to utilize context effectively, regardless of the prompting setting. These results underline the importance of optimized contextual strategies and task-specific fine-tuning for advancing AI-driven exploit generation for high-stakes applications in software security.

This paper proposes an empirical methodology to study software aging in GPU-based LLM serving systems. Traditional aging studies focus on CPU-centric software with relatively regular workloads; LLM serving is different, spanning a Python host and a CUDA device, handling requests whose cost varies by orders of magnitude, and relying on rapidly evolving software stacks. We run a 216-hour campaign across six co-located deployments under identical stress conditions, monitor host, device, and client metrics in parallel, and apply a statistical pipeline that accounts for autocorrelation and multiple testing. Our results reveal statistically significant memory aging in all deployments, with leak rates strongly dependent on the serving runtime and deployment configuration. Beyond these findings, we provide a reproducible framework that opens a research direction at the intersection of the software aging and rejuvenation and LLM serving communities.

Since manually writing software exploits for offensive security is time-consuming and requires expert knowledge, AI-base code generators are an attractive solution to enhance security analysts’ productivity by automatically crafting exploits for security testing. However, the variability in the natural language and technical skills used to describe offensive code poses unique challenges to their robustness and applicability. In this work, we present a method to add perturbations to the code descriptions to create new inputs in natural language (NL) from well-intentioned developers that diverge from the original ones due to the use of new words or because they miss part of them. The goal is to analyze how and to what extent perturbations affect the performance of AI code generators in the context of offensive code. First, we show that perturbed descriptions preserve the semantics of the original, non-perturbed ones. Then, we use the method to assess the robustness of three state-of-the-art code generators against the newly perturbed inputs, showing that the performance of these AI-based solutions is highly affected by perturbations in the NL descriptions. To enhance their robustness, we use the method to perform data augmentation, i.e., to increase the variability and diversity of the NL descriptions in the training data, proving its effectiveness against both perturbed and non-perturbed code descriptions.

Female, Black, Latinx, Native American, low-income, and rural students remain underrepresented among computer science undergraduate degree recipients. Along with student, family, and secondary school characteristics, college organizational climate, curricula, and instructional practices shape undergraduates' experiences that foster persistence until graduation. Our quasi-experimental project, Improving the Persistence and Success of Students from Underrepresented Populations in Computer Science (I-PASS), is designed to augment students' persistence until they earn their computer science degree. Drawing on prior research, including Tinto's model of effective institutional actions for retention, I-PASS Scholars-all low-income, female and/or members of underserved demographics groups- receive a four-year scholarship; mentoring, tutoring, advising; and opportunities to integrate into the academic and social life of the campus. Students' written reflections and attitude surveys suggest I-PASS's components foster their retention by, among other mechanisms, enhancing their computer science identity development and sense of belonging in the major.

This practical experience report explores Neural Machine Translation (NMT) models’ capability to generate offensive security code from natural language (NL) descriptions, highlighting the significance of contextual understanding and its impact on model performance. Our study employs a dataset comprising real shellcodes to evaluate the models across various scenarios, including missing information, necessary context, and unnecessary context. The experiments are designed to assess the models’ resilience against incomplete descriptions, their proficiency in leveraging context for enhanced accuracy, and their ability to discern irrelevant information. The findings reveal that the introduction of contextual data significantly improves performance. However, the benefits of additional context diminish beyond a certain point, indicating an optimal level of contextual information for model training. Moreover, the models demonstrate an ability to filter out unnecessary context, maintaining high levels of accuracy in the generation of offensive security code. This study paves the way for future research on optimizing context use in AI-driven code generation, particularly for applications requiring a high degree of technical precision such as the generation of offensive code.

Recent advances of artificial intelligence (AI) code generators are opening new opportunities in software security research, including misuse by malicious actors. We review use cases for AI code generators for security and introduce an evaluation benchmark.

AI-based code generators are an emerging solution for automatically writing programs starting from descriptions in natural language, by using deep neural networks (Neural Machine Translation, NMT). In particular, code generators have been used for ethical hacking and offensive security testing by generating proof-of-concept attacks. Unfortunately, the evaluation of code generators still faces several issues. The current practice uses output similarity metrics, i.e., automatic metrics that compute the textual similarity of generated code with ground-truth references. However, it is not clear what metric to use, and which metric is most suitable for specific contexts. This work analyzes a large set of output similarity metrics on offensive code generators. We apply the metrics on two state-of-the-art NMT models using two datasets containing offensive assembly and Python code with their descriptions in the English language. We compare the estimates from the automatic metrics with human evaluation and provide practical insights into their strengths and limitations.

Writing software exploits is an important practice for offensive security analysts to investigate and prevent attacks. In particular, shellcodes are especially time-consuming and a technical challenge, as they are written in assembly language. In this work, we address the task of automatically generating shellcodes, starting purely from descriptions in natural language, by proposing an approach based on Neural Machine Translation (NMT). We then present an empirical study using a novel dataset (Shellcode_IA32), which consists of 3,200 assembly code snippets of real Linux/x86 shellcodes from public databases, annotated using natural language. Moreover, we propose novel metrics to evaluate the accuracy of NMT at generating shellcodes. The empirical analysis shows that NMT can generate assembly code snippets from the natural language with high accuracy and that in many cases can generate entire shellcodes with no errors.

Neural Machine Translation (NMT) has reached a level of maturity to be recognized as the premier method for the translation between different languages and aroused interest in different research areas, including software engineering. A key step to validate the robustness of the NMT models consists in evaluating the performance of the models on adversarial inputs, i.e., inputs obtained from the original ones by adding small amounts of perturbation. However, when dealing with the specific task of the code generation (i.e., the generation of code starting from a description in natural language), it has not yet been defined an approach to validate the robustness of the NMT models. In this work, we address the problem by identifying a set of perturbations and metrics tailored for the robustness assessment of such models. We present a preliminary experimental evaluation, showing what type of perturbations affect the model the most and deriving useful insights for future directions.

Emphasis on system safety and reliability assessment allowed us to place societal trust on mass-transportation and medical systems, energy, space, financial and military infrastructures, and many other critical domains. As the technology intrudes into all aspects of our lives, existing verification and validation approaches are not sufficient anymore. In addition to meeting safety, reliability, and security standards, we expect systems to protect our privacy, uphold equity, integrity and human rights, perform without biases, and meet other stated and unstated expectations. Further, some of the logic embedded in software may be the product of machine learning, making it unintuitive for humans to understand the rules it encodes, thereby difficult to assess. In this talk, we will discuss software accountability principles. Efforts to regulate software and services are increasing, with the goal of making organizations and individuals more accountable for its consequences. But what constitutes accountable software is not understood well. We will overview the outcomes of traditional software safety assessment processes and compare them with emerging verification needs. Examples from adaptive flight controls and airport passenger management will offer preliminary illustrations of the dimensions of software accountability assurance. Our talk will conclude by outlining opportunities for future research.

Writing exploits for security assessment is a challenging task. The writer needs to master programming and obfuscation techniques to develop a successful exploit. To make the task easier, we propose an approach (EVIL) to automatically generate exploits in assembly/Python language from descriptions in natural language. The approach leverages Neural Machine Translation (NMT) techniques and a dataset that we developed for this work. We present an extensive experimental study to evaluate the feasibility of EVIL, using both automatic and manual analysis, and both at generating individual statements and entire exploits. The generated code achieved high accuracy in terms of syntactic and semantic correctness.

We take the first step to address the task of automatically generating shellcodes, i.e., small pieces of code used as a payload in the exploitation of a software vulnerability, starting from natural language comments. We assemble and release a novel dataset (Shellcode_IA32), consisting of challenging but common assembly instructions with their natural language descriptions. We experiment with standard methods in neural machine translation (NMT) to establish baseline performance levels on this task.

The paper is part of the Difference Makers Track of FIE 2020.Many institutions of higher learning are in the process of defining and implementing Data Science programs. The emerging field is revolutionizing scientific discovery and many industries. Broad availability of data sets and emerging analytical techniques for their processing are changing our economies and societies. While there is a broad agreement about underlying principles in this new discipline, defining data science as a pedagogically independent discipline has proven to be a challenge.In this paper, we present the university-wide effort that led to the creation of a new undergraduate interdisciplinary Data Science program. In particular, we point to the critical role of inclusive pedagogical design and broadening participation criteria in program development. Current and projected job market data clearly indicate that data science programs need to expand traditional STEM workforce and attract students with varying backgrounds and degrees or preparation. Such a focus has led to a broad agreement across our campus regarding the new program and its curriculum.

This Research-to-Practice Work-in-Progress Paper presents a midway report on a change initiative underway in the College of Computing and Informatics (CCI) at UNC Charlotte. Comprised of approximately 100 faculty members and nearing 2,000 undergraduate majors, CCI was awarded a 5-year, $2 million grant from the National Science Foundation in 2015 to revolutionize computer science education at the collegiate level. This initiative, which seeks to simultaneously achieve both pedagogical and organizational change, is built upon a foundation of educational innovation through engaged teaching practices. To sustain educational innovation in CCI beyond the funded scope of the project, pedagogical change is thus strategically embedded in CCI's organizational structure through a 3-stage model of faculty adoption, redesigned student course evaluations, and realigned values in the reappointment, promotion, and tenure process. By recognizing that sustained organizational change takes root when there is concurrent buy-in from organizational members-both bottom-up and top-down-this reform initiative seeks to embed two-pronged change not only via educational innovation (pedagogical change), but also in the day-to-day practices, policies, and physical environment of the College itself (organizational change).

Across the globe, we are witnessing a reassertion of hegemonic systems of domination around gender, race, socioeconomic status, ability, sexuality, religion, and immigrant status. In the United States in particular, the fragile foundation underlying civil rights and social progress in eradicating segregation, prejudice, and discrimination in education and employment is under threat. While many organizations and individuals are pushing back against these threats, the enactment of new public policy and legislation that undermines civil rights (coupled with the unraveling of existing civil rights policy and legislation) signals a shift in priority and effort away from dismantling historical structures of inequity. This shifting larger social, political, and economic context is problematic for the succession of efforts to broaden participation in computer science education (CSEd), as it reasserts old, dominant paradigms that maintain the status quo. Educational reform is a herculean task under favorable conditions, let alone an overarching environment bucking against gains made in dismantling inequity over the last several decades. Due to these obstacles, we posit that systemic organizational change within CSEd institutions and programs themselves is necessary to truly effect sustainable, transformational change in broadening participation efforts, especially under hostile conditions in the larger environment. This poster will profile one such change effort in progress within a college of computing and informatics located at a large, urban research university in the southeastern US. Comprised of three departments, approximately 100 faculty members, and approximately 1,900 undergraduate students, this CSEd change effort focuses on broadening and sustaining participation through a foundation of engaged pedagogy. Over the past two years, we built a new structure of faculty training and development that is designed to (re)focus CSEd faculty to engaged pedagogical practice, varied teaching techniques, and classroom tools that foster student learning in historically underrepresented and underserved groups that are not typically well-served by traditional teaching beliefs and practices. With approximately 25% of the ~100 CSEd faculty participating to-date (and growth projected to 35% by the end of 2018), we established our base for long-term structural change in our college. Building upon that base as a critical first step, we next retool our teaching evaluations to further enstructure inclusive teaching values within the recurring, everyday processes of the organization. By strategically focusing on an element of the reappointment, promotion, and tenure (RPT) process, we send a crucial signal to organizational members—faculty, students, staff, administrators—that inclusive teaching and learning matters beyond "lip service" and beyond environmental pressures to the contrary.

Biometric identification is a critically important technology in traveler, immigration and refugee management. The technology itself and the processes related to human identification and identity management are a prime target for identity theft, tampering, spoofing, and impersonation. In the past year, our team developed a systematic methodology for identification of biometric technology vulnerabilities and identity management process limitations. We defined several attack vectors and tried to establish objective measures of risk exposure. But to establish risk exposure, one needs to understand the social process that may lead to biometric attacks. Therefore, we developed a methodology to monitor publicly available information sources that may reveal the extent of threats, availability and sophistication of attack tools and how-to recipes for biometric attacks at US Ports of Entry. This report presents the search techniques we developed and overviews current results.

Touch dynamics is a behavioral biometric that authenticates users by analyzing the characteristics of the touch gestures executed on mobile devices. Current research in this field has mostly focused on identifying the best algorithms and attributes to improve authentication performance. However, such systems must also be resilient against environmental variables. In this paper, we demonstrate that the user's posture, device size and configuration have a significant impact on the performance of touch-based authentication systems. Our results indicate that authentication accuracy increases with the device size. Furthermore, we conclude that using a device's 3-D orientation is necessary to attain better authentication performance. Our findings indicate that the features used in state-of-the-art touch-based authentication systems are insufficient to provide constant, reliable performance when any of the studied environmental variables change. With this paper, we release a new data set. Unlike the currently publicly available touch-based authentication datasets, our collection protocols control for all the studied variables. Our research study demonstrates threats to validity that noisy environmental conditions introduce to these currently available public datasets. This work is an extension of a previous publication. Presented user authentication approaches are unique and may have immediate benefits to the development of better touch-based authentication systems. (C) 2018 Published by Elsevier Inc.

Several reports have highlighted that spoofing biometric traits represents a serious threat for active identity management systems. Applications of biometrics are critically important technologies for traveler, immigration and refugee management systems deployed or considered for deployment by the Department of Homeland Security. Due to motivations and consequences of identity misrepresentations at US borders, threats specific to these categories differ from generic biometric applications. Thus, there is a strong need to reduce the risk of spoof-based fraud mitigation strategies are needed. Biometric system attack vector analysis is growing but still not as fast as the general level of security threats. Therefore, in this paper we analyze feasibility of biometric presentation attacks behind identity misrepresentation and discuss a practical methodology for classifying biometric identity attack vectors based on their risks. The classification will lead towards practical vulnerability assessment methods and countermeasures, technical as well as managerial. Our findings aim to enable identification of biometric presentation attack risks and severities, leading towards a welldefined defense strategy.