Risk Determination versus Risk Perception: A New Model of Reality for Human-Machine Autonomy

We review the progress in developing a science of interdependence applied to the determinations and perceptions of risk for autonomous human-machine systems based on a case study of the Department of Defense's (DoD) faulty determination of risk in a drone strike in Afghanistan; the DoD's assessment was rushed, suppressing alternative risk perceptions. We begin by contrasting the lack of success found in a case study from the commercial sphere (Facebook's use of machine intelligence to find and categorize "hate speech"). Then, after the DoD case study, we draw a comparison with the Department of Energy's (DOE) mismanagement of its military nuclear wastes that created health risks to the public, DOE employees, and the environment. The DOE recovered by defending its risk determinations and challenging risk perceptions in public. We apply this process to autonomous human-machine systems. The result from this review is a major discovery about the costly suppression of risk perceptions to best determine actual risks, whether for the military, business, or politics. For autonomous systems, we conclude that the determinations of actual risks need to be limited in scope as much as feasible; and that a process of free and open debate needs to be adopted that challenges the risk perceptions arising in situations facing uncertainty as the best, and possibly the only, path forward to a solution.