Leveraging Factored Action Spaces for Off-Policy Evaluation

Aaman Peter Rebello, Shengpu Tang, Jenna Wiens, Sonali Parbhoo

Keywords: Off-policy evaluation; behaviour; evaluation; factored actions; variance; bias

TL;DR: We propose and study a new family of decomposed IS estimators that leverage the inherent factorisation structure of actions. We theoretically prove that our proposed estimator is unbiased and has low variance.

Abstract: In high-stakes decision-making domains such as healthcare, off-policy evaluation (OPE) can help practitioners understand the performance of a new policy before deploying it. However, existing OPE estimators often exhibit high bias and high variance in problems involving large, combinatorial action spaces. We investigate how to mitigate this problem using factored action spaces i.e. expressing each action as a combination of independent sub-actions from smaller action spaces. We propose and study a new family of "decomposed" importance sampling (IS) estimators based on factored action spaces. Given certain assumptions on the underlying problem structure, we prove that the decomposed IS estimators have less variance than their original non-decomposed versions, while preserving the property of zero bias. This results in lower mean squared error. Through simulations, we empirically verify our theoretical results, probing the validity of various assumptions. Provided with a technique that can derive the action space factorisation for a given problem, our work shows that OPE can be improved "for free" by utilising this inherent problem structure.

Submission Number: 122