MULFE: A Multi-Level Benchmark for Free Text Model Editing

Anonymous

Abstract: Adjusting the outdated behaviors of large langugae models (LLMs) after deployment remains a significant challenge. It motivates the model editing research, which is however mainly explored in a restricted task form with triple-based edit requests. Some recent works have initiated a transition to a more practical and unified editing task that takes free-form text as edit requests. However, there is gaps in nuanced benchmark designs and re-evaluation of existing methods. To bridge the gaps, we introduce a multi-level benchmark for free text model editing (\textsc{Mulfe}). The benchmark categorizes probe queries into three levels of generalization, ranging from basic literal memory to deeper understanding and reasoning. Based on the benchmark, we conduct extensive experiments across various base models, edit sizes, and editing methods, including adaptations of mainstream locate-and-edit and hypernetwork methods. The results highlight the inconsistent behaviors of edited models on different generalization levels. Higher level of generalization is still difficult. Based on the findings, we propose \textsc{Side}, a simple yet effective method based on in-context distillation to enhance the generalization performance. The benchmark and baseline methods will be publicly available for facilitating further study.

Paper Type: long

Research Area: Resources and Evaluation

Contribution Types: Reproduction study, Data resources

Languages Studied: English