Thou Shalt Discuss Security: Quantifying the Impacts of Instructions to RFC Authors

The importance of secure development of new technologies is unquestioned, yet the best methods to achieve this goal are far from certain. A key issue is that while significant effort is given to evaluating the outcomes of development (e.g., security of a given project), it is far more difficult to determine what organizational practices result in secure projects. In this paper, we quantitatively examine efforts to improve the consideration of security in Requests for Comments (RFCs)--- the design documents for the Internet and many related systems --- through the mandates and guidelines issued to RFC authors. We begin by identifying six metrics that quantify the quantity and quality of security informative content. We then apply these metrics longitudinally over 8,437 documents and 49 years of development to determine whether guidance to RFC authors changed these security metrics in later documents. We find that even a simply worded --- but effectively enforced --- mandate to explicitly consider security created a significant effect in increased discussion and topic coverage of security content both in and outside of a mandated security considerations section. We find that later guidelines with more detailed advice on security also improve both volume and quality of security informative content in RFCs. Our work demonstrates that even modest amounts of guidance can correlate to significant improvements in security focus in RFCs, indicating a promising approach for other network standards bodies.