Ad hoc Teamwork and Moral Feedback as a Framework for Safe Agent Behavior.

As technology develops, it is only a matter of time before agents will be capable of long-term autonomy, i.e., will need to choose their actions by themselves for a long period of time. Thus, in many cases agents will not be able to be coordinated in advance with all other agents with which they may interact. Instead, agents will need to cooperate in order to accomplish unanticipated joint goals without pre-coordination. As a result, the ad hoc teamwork problem, in which teammates must work together to obtain a common goal without any prior agreement regarding how to do so, has emerged as a recent area of study in the AI literature. However, to date, no attention has been dedicated to the social aspect of the agentsu0027 behavior, which is required to ensure that their actionsu0027 influences on other agents conform with social norms. In this research, we introduce the STAR framework used to teach agents to act in accordance with human social norms with respect to their teammates. Using a hybrid team (agents and people), if taking an action considered to be socially unacceptable, the agents will receive negative feedback from the human teammate(s). We view STAR as an initial step towards achieving the goal of teaching agents to act more consistently with respect to human morality.