DistillMIKE: Editing Distillation of Massive In-Context Knowledge Editing in Large Language Models

Anonymous

Abstract: Among the recently emerged knowledge editing methods, in-context knowledge editing (IKE) $\cite{IKE}$ has shown respectable abilities on knowledge editing in terms of generalization and specificity. Noting the promising advantages but unexplored issues of IKE, we propose $\textbf{DistillMIKE}$ as a novel extension of IKE, i.e., editing $\textbf{distill}$ation of ``$\textbf{M}$assive'' $\textbf{I}$n-context $\textbf{K}$nowledge $\textbf{E}$diting in large language models (LLMs), mainly consisting of two expansions; 1) $\textit{Massive in-context knowledge editing (MIKE)}$, which extends IKE to a massive editing task, aiming to inject not a single edit but a set of massive edits to LLMs; To preserve specificity, our key novel extension is a ``selective'' retrieval augmentation, where the retrieval-augmented IKE is only applied to ``in-scope'' examples, whereas the unedited model without IKE is employed for ``out-of-scope'' ones. 2) $\textit{Editing distillation }$of MIKE using low-rank adaptation (LoRA), which distills editing abilities of MIKE to parameters of LLMs in a manner of eliminating the need of lengthy in-context demonstrations, thus removing the computational overhead encountered at the inference time. Experimental results on the zsRE and CounterFact datasets demonstrate that MIKE shows the state-of-the-art perfomrances and DistilMIKE show comparable performances with MIKE. Our code is available at $\url{https://github.com/xxxx/xxxx}$.

Paper Type: long

Research Area: Interpretability and Analysis of Models for NLP

Contribution Types: Model analysis & interpretability

Languages Studied: English