arithmetic for calculating risk may appeal to some, suggesting that risk is an exact science, yet most if not all the existing models have not been validated. It is also reasonable to suspect that the more complex risk calculations get, the less they can actually be linked to empirical reality. The stock trader Nassim Taleb, author of the popular book The Black Swan, recently commented that in the last century, financial decision makers favored the philosophy: “If there is a formula in it, don’t use it” when thinking about inherent risk in specific portfolio strategies and finance products. This trend of the last century underwent a profound reversal when the banking industry and the financiers of Wall Street embraced an extremely complex risk formula. There is little argument that blind acceptance of the output of a risk calculation ultimately contributed to the global financial crisis of 2008. The validity of a measurement method is determined by how accurately it measures what it claims to measure. applied to cyber risk, the question is, does the risk-based approach really measure the likelihood of experiencing a cyber attack? This is certainly an important question for any government regulator and even more so 9 eric byres, David Leversage, and Nate Kube, “Security incidents and trends in SCaDa and process industries,” The Industrial Ethernet Book vol. 39, issue 2 (May 2007): pp. 12-20. http://www.mtl-inst.com/images/uploads/datasheets/Iebook_May_07_ SCaDa_Security_Trends.pdf. 10 Vilhelm Verendel, “Quantified security is a weak hypothesis: a critical survey of results and assumptions,” in Proceedings of the 2009 workshop on New security paradigms workshop, Oxford, uK, September 8-11, 2009, http://portal.acm.org/citation. cfm?id=1719030.1719036, pp. 37-50. 11 Nassim Nicholas Taleb, The Black Swan: The impact of the highly improbable (New york: random House, 2007). 12 Felix Salmon, “recipe for Disaster: The Formula That Killed Wall Street,” Wired vol 17, issue 3, (February 23, 2012), http://www. dpwireless.net/backPage/TheSecretFormulaThatKilledWallStreet_200903.pdf.

Security, reliability and interoperability are indispensable in today's distributed heterogeneous information infrastructure. For government and military applications, it is crucial to conduct effective and efficient testing of security properties for newly developed systems, which are to be integrated into existing information system. Yet little progress has been made in the technology advancement of rigorous and automated security testing. In this contribution we present virtual cyber security testing capability (VCSTC) - a DoD funded project-for developing an automated testing capability that can assess the operational functions and security impact of a target system without physically integrating it into an intended network infrastructure. VCSTC first synthesizes a model to emulate the real network infrastructure; then it automatically generates and executes test cases with guaranteed coverage of the features and security properties under test. This report presents the architecture of VCSTC, its key techniques and experimental results on real systems.