Explainable AI for earth observation: A review including societal and regulatory perspectives

•Emerging regulations and guidelines require AI algorithms to be explainable.•Explainable AI is new to the field of Earth Observation, and will have great impact.•Random Forests are considered interpretable by EO, but not by social sciences.•Explainable AI methods for non-expert users are required, but lacking.•Generally, Explainable AI methods are presented but their efficacy isn’t tested.