Characterizing The Limits Of Autonomous Systems

This note answers two central question in the intersection of decision making and causal inference - when human input is needed and, if so, how it should be incorporated into an AI system. We introduce the counterfactual agent who proactively considers human input in its decision-making process. We prove that a counterfactual agent dominates the standard autonomous agent who does not consider any human input (i.e., the experimental agent) in terms of performance. These results suggest a trade-off between autonomy and optimality-while the full autonomy is often preferred, using human input could potentially improve the efficiency of the system. We further characterize under what conditions experimental and counterfactual agents can reach the same level of performance, which elicits the settings where full autonomy can be achieved.