The Kukui Cup project investigates the use of “meaningful play” to facilitate energy awareness, conservation and behavioral change. Each Kukui Cup Challenge combines real world and online environments in an attempt to combine information technology, game mechanics, educational pedagogy, and incentives in a synergistic and engaging fashion. We challenge players to: (1) acquire more sophistication about energy concepts and (2) experiment with new behaviors ranging from micro (such as turning off the lights or installing a CFL) to macro (such as taking energy-related courses, joining environmental groups, and political/social advocacy.) To inform the design of the inaugural 2011 Kukui Cup, we relied heavily on prior collegiate energy competitions, of which there have been over 150 in the past few years. Published accounts of these competitions indicate that they achieve dramatic reductions in energy usage (a median of 22%) and cost savings of tens of thousands of dollars. In our case, the data collected from the 2011 Kukui Cup was generally in agreement, with observed energy reductions of up to 16% when using data collection and analysis techniques typical to these competitions. However, our analysis process caused us to look more closely at the methods employed to produce outcome data for energy competitions, with unexpected results. We now believe that energy competitions make significant unwarranted assumptions about the data they collect and the way they analyze them, which calls into question both the accuracy of published results from this literature and their effectiveness as serious games. We believe a closer examination of these issues by the community can help improve the design not only of future energy challenges, but other similar forms of serious games for sustainability. In this paper, we describe the Kukui Cup, the design myths it uncovered, and the fixes we propose to improve future forms of meaningful play with respect to energy in particular and serious games in general.

Assessment of serious game frameworks is emerging as an important area of research. This paper describes an assessment mechanism called the Serious Game Stakeholder Experience Assessment Method (SGSEAM). SGSEAM is designed to provide detailed insights into the strengths and shortcomings of serious game frameworks through a stakeholder perspective based approach. In this paper, we report on the use of SGSEAM to assess Makahiki, an open source serious game framework for sustainability. Our results provide useful insights into both Makahiki as a serious game framework and SGSEAM as an assessment method.

For more than 15 years, researchers at the Collaborative Software Development Laboratory at the University of Hawaii at Manoa have looked for analytics that help developers understand and improve development processes and products. This article reviews that research and discusses the trade-off between studying easily obtained analytics and studying richer analytics with higher overhead.

To achieve the full benefits of the Smart Grid, end users must become active participants in the energy ecosystem. This paper presents the Kukui Cup challenge, a serious game designed around the topic of energy conservation which incorporates a variety of energy feedback visualizations, a multifaceted serious game with online educational activities, and real-world activities such as workshops and excursions. We describe our experiences in developing energy feedback visualizations in the Kukui Cup based on in-lab evaluations and field studies in college residence halls. We learned that energy feedback systems should address these factors: they should be actionable, that domain knowledge must go hand in hand with energy feedback systems, and that this feedback must be “sticky” to lead to changes in behaviors and attitudes. Keywords-Serious games; energy feedback; energy; energy literacy; smart grid.

The Kukui Cup is an advanced dorm energy competition whose goal is to investigate the relationships among energy literacy, sustained energy conservation, and information technology support of behavior change. Two general purpose open source systems have been implemented: WattDepot and Makahiki. WattDepot provides enterprise-level collection, storage, analysis, and visualization of energy data. Makahiki is a web application framework that supports dorm energy competitions of varying degrees of complexity, including a personalized homepage where participants can complete tasks designed to increase energy literacy that can be verified by competition administrators. The technology and approach will be evaluated in a dorm energy competition to take place in the Spring of 2011, with hundreds of University freshmen. The energy use of each pair of dormitory floors will be metered in near-realtime, and the energy literacy of participants will be assessed before and after the competition.

The authors currently engaged in two projects to improve human-computer interaction (HCI) designs that can help conserve resources. The projects explore motivation and persuasion strategies relevant to ubiquitous computing systems that bring realtime consumption data into the homes and hands of residents in Brisbane, Australia. The first project seeks to increase understanding among university staff of the tangible and negative effects that excessive printing has on the workplace and local environment. The second project seeks to shift attitudes toward domestic energy conservation through software and hardware that monitor real-time, in situ electricity consumption in homes across Queensland. The insights drawn from these projects will help develop resource consumption user archetypes, providing a framework linking people to differing interface design requirements.

Test-driven development (TDD) is a style of development named for its most visible characteristic: the design and implementation of test cases prior to the implementation of the code required to make them pass. Many claims have been made for TDD: that it can improve implementation as well as design quality, that it can improve productivity, that it results in 100% coverage, and so forth. However, research to validate these claims has yielded mixed and sometimes contradictory results. We believe that at least part of the reason for these results stems from differing interpretations of the TDD development style, along with an inability to determine whether programmers actually follow whatever definition of TDD is in use.Zorro is a system designed to automatically determine whether a developer is complying with an operational definition of Test-Driven Development (TDD) practices. Automated recognition of TDD can benefit the software development community in a variety of ways, from inquiry into the "true nature" of TDD, to pedagogical aids to support the practice of test-driven development, to support for more rigorous empirical studies on the effectiveness of TDD in both laboratory and real world settings.This paper describes the Zorro system, its operational definition of TDD, the analyses made possible by Zorro, two empirical evaluations of the system, and an attempted case study. Our research shows that it is possible to define an operational definition of TDD that is amenable to automated recognition, and illustrates the architectural and design issues that must be addressed in order to do so. Zorro has implications not only for the practice of TDD, but also for software engineering "micro-process" definition and recognition through its parent framework, Software Development Stream Analysis.

WattDepot is an open source, Internet-based, service-oriented framework for collection, storage, analysis, and visualization of energy data. WattDepot differs from other energy management solutions in one or more of the following ways: it is not tied to any specific metering technology; it provides high-level support for meter aggregation and data interpolation; it supports carbon intensity analysis; it is architecturally decoupled from the underlying storage technology; it supports both hosted and local energy services; it can provide near-real time data collection and feedback; and the software is open source and freely available. In this paper, we introduce the framework, provide examples of its use, and discuss its application to research and understanding of the Smart Grid.

Hackystat is an open source framework for automated collection and analysis of software engineering process and product data. Hackystat has been in development since 2001, and has gone through eight major architectural revisions during that time. In 2007, we performed the latest architectural revision, whose primary goal was to reimplement Hackystat as a service-oriented architecture (SOA). This version has now been in public release for a year, and this paper reports on our experiences: the motivations that led us to reimplement the system as a SOA, the costs and benefits of that conversion, and our lessons learned.

For empirical software engineering to reach its fullest potential, we must develop effective, experiential approaches to learning about it in a classroom setting. In this paper, we report on a case study involving a new approach to classroom-based empirical software engineering called the ldquoSoftware ICUrdquo. In this approach, students learn about nine empirical project ldquovital signsrdquo and use the Hackystat framework to put their projects into a virtual ldquointensive care unitrdquo where these vital signs can be assessed and monitored. We used both questionnaire and log data to gain insight into the strengths and weaknesses of this approach. Our evaluation provides both quantitative and qualitative evidence concerning the overhead of the system; the relative utility of different vital signs; the frequency of use; and the perceived appropriateness outside of the classroom setting. In addition to benefits, we found evidence of measurement dysfunction induced directly by the presence of the Software ICU. We compare these results to case studies we performed in 2003 and 2006 using the Hackystat framework but not the Software ICU. We use these findings to orient future research on empirical software engineering both inside and outside of the classroom.

2 Related Work 5 2.1 Hackystat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.2 Software Project Telemetry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.3 Software Development Stream Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.4 Pattern Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.5 Evidence-based software engineering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.6 Results from prior NSF research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

If empirical software engineering is to grow as a valid scientific endeavor, the ability to acquire, use, share, and compare data collected from a variety of sources must be encouraged. This is necessary to validate the formal models being developed within computer science. However, within the empirical software engineering community this has not been easily accomplished. This paper analyses experiences from a number of projects, and defines the issues, which include the following: (1) How should data, testbeds, and artifacts be shared? (2) What limits should be placed on who can use them and how? How does one limit potential misuse? (3) What is the appropriate way to credit the organization and individual that spent the effort collecting the data, developing the testbed, and building the artifact? (4) Once shared, who owns the evolved asset? As a solution to these issues, the paper proposes a framework for an empirical software engineering artifact agreement. Such an agreement is intended to address the needs for both creator and user of such artifacts and should foster a market in making available and using such artifacts. If this framework for sharing software engineering artifacts is commonly accepted, it should encourage artifact owners to make the artifacts accessible to others (gaining credit is more likely and misuse is less likely). It may be easier for other researchers to request artifacts since there will be a well-defined protocol for how to deal with relevant matters.

Improving the software engineering development process requires collection of data, but collection of data interferes with how developers work. At present, most of the software engineering tools, data collection, and analysis techniques available use manual data collection, despite known problems with reliability, correctness, and timeliness of the data. To overcome such limitations and reduce interference with the development process, software engineering researchers must develop tools and data analysis techniques that collect data without human interactions. Such tools produce very detailed and extensive data, but lack the filtering and classification that humans perform on manually collected data. This unfiltered data requires the development of new analysis techniques and new prediction models to use it effectively. This workshop focuses on defining the research challenges created by in process software measurement and analysis of the software development process using tools that do not affect or modify the process but extract data automatically from it.

"UltraLargeScale Systems: The Software Challenge of the Future" identifies "Engineering Management at Large Scales" as an important focus of research. Engineer ing management for software typically involves measure ment and monitoring of products and processes in order to maintain acceptable levels of important project characteristics including cost, quality, usability, performance, reliability, and so forth. Our research on software engineering measurement over the past ten years has exhibited a trend towards increasing automation and autonomy in the collection and analysis of process and product measures. In this position paper, we extrapolate from our work so far to con sider what new forms of automation and autonomy might be required for software engineering management of ULS systems.

Zorro is a system designed to automatically determine whether a developer is complying with an operational definition of Test-Driven Development (TDD) practices. Automated recognition of TDD can benefit the software development community in a variety of ways, from inquiry into the "true nature" of TDD, to pedagogical aids to support the practice of test-driven development, to support for more rigorous empirical studies on the effectiveness of TDD in both laboratory and real world settings. This paper introduces the Zorro system, its operational definition of TDD, the analyses made possible by Zorro, and our ongoing efforts to validate the system.

The challenges in fault prediction today are to get a prediction as early as possible, at as low a cost as possible, needing as little data as possible and preferably in such a language that your average developer can understand where it came from. This paper presents a fault sampling method where a summary of a few, easily available metrics is used together with the results of a few sampled classes to generalize the fault content to an entire system. The method is tested on a large software system written in Java, that currently consists of around 2000 classes and 300,000 lines of code. The evaluation shows that the fault generalization method is good at predicting fault-prone clusters and that it is possible to generalize the values of a few representative classes.