The Challenges of Labeling Vulnerability-Contributing Commits

Software projects developed using version control are enhanced incrementally through commits, some of which inevitably introduce security vulnerabilities. The features of these vulnerability-contributing commits (VCCs) could be used to train a VCC detector or to inform software development best-practices. Previous work has attempted to label VCCs in open-source software projects for this purpose. We present a manual approach to VCC labeling using the fix commits listed in Common Vulnerabilities and Exposures (CVEs). We show that a published automated method of VCC labeling disagrees with our manual method on 42% of VCCs. We argue that the automated method, while effective in scaling VCC labeling, is therefore not sufficiently accurate. Finally, we discuss the benefits and drawbacks of trying to predict vulnerable software components rather than VCCs.