Not All Secrets Are Equal: Type-Aware Unlearning for Language Model Secret Removal

Large language models (LLMs) can reproduce memorized sensitive data, including SSH keys, API tokens, and personally identifiable information (PII), when prompted with the right context. Removing these secrets is difficult: gradient-based unlearning often damages model utility, while preference-based methods can leave secrets recoverable, especially at larger scales. We show that a central reason is type heterogeneity: different secret types exhibit different gradient-saliency patterns and require different correction strengths. We introduce Per-Secret-Type Unlearning (PSTU), a training-free task-arithmetic method that replaces a single global scaling factor with per-type saliency and layer-group corrections. For 7B+ models, PSTU-Trim denoises the task vector before subtraction. Across our evaluated Pythia (1.4B to 6.9B) and Llama-3.1-8B configurations, PSTU removes all 175 measured memorized secrets with 1.3 ±1