The high performance computing (HPC) community is increasingly aware that traditional low-level, execution-time measures for assessing high-end computers, such as flops/second, are not adequate for understanding the actual productivity of such systems. In response, researchers and practitioners are exploring new measures and assessment procedures that take a more wholistic approach to high performance productivity. In this paper, we present an approach to understanding and assessing development-time aspects of HPC productivity. It involves the use of Hackystat for automatic, nonintrusive collection and analysis of six measures: Active Time, Most Active File, Command Line Invocations, Parallel and Serial Lines of Code, Milestone Test Success, and Performance. We illustrate the use and interpretation of these measures through a case study of smallscale HPC software development. Our results show that these measures provide useful insight into developmenttime productivity issues, and suggest promising additions to and enhancements of the existing measures.

0 7 4 0 7 4 5 9 / 0 5 / $ 2 0 . 0 0 © 2 0 0 5 I E E E can neither predict nor control what you cannot measure.”3 Despite metrics’ theoretical potential, effectively applying them appears to be far from mainstream in practice. For example, a recent case study of more than 600 software professionals revealed that only 27 percent viewed metrics as “very” or “extremely” important to their software project decision-making process.4 The study also revealed that most respondents attempted to use metrics only for cost and schedule estimation. Practitioners face various barriers in applying metrics (see the sidebar “Explaining the Gap between Software Metrics Theory and Practice”). It’s no wonder that many practitioners find it daunting to apply best practices to their own situation. Indeed, the agile community generally argues against model-based metrics applications, promoting softer metrics for decision making.5 Fortunately, creating predictive models based on historical project data isn’t the only possible way to apply software metrics to project management. Our team at the Collaborative Software Development Laboratory has developed a new telemetry-based approach.

Conventional wisdom in the software engineering research community says that metrics can make project management more effective. Software metrics range from internal product attributes such as size, complexity, and modularity to external process attributes such as effort, productivity, and reliability. Software project telemetry facilitates local, in-process decision making.