Remembering is more than an activation of a memory trace. As retrieval cues are often not uniquely related to one specific memory, cognitive control should come into play to guide selective memory retrieval by focusing on relevant while ignoring irrelevant information. Here, we investigated, by means of EEG and fMRI, how the memory system deals with retrieval interference arising when retrieval cues are associated with two material types (faces and spatial positions), but only one is task-relevant. The topography of slow EEG potentials and the fMRI BOLD signal in posterior storage areas indicated that in such situations not only the relevant but also the irrelevant material becomes activated. This results in retrieval interference that triggers control processes mediated by the medial and lateral PFC, which are presumably involved in biasing target representations by boosting the task-relevant material. Moreover, memory-based conflict was found to be dissociable from response conflict that arises when the relevant and irrelevant materials imply different responses. The two types of conflict show different activations in the medial frontal cortex, supporting the claim of domain-specific prefrontal control systems.