Exploring validation metrics for offline model-based optimisation with diffusion models
arxiv(2022)
摘要
In model-based optimisation (MBO) we are interested in using machine learning
to design candidates that maximise some measure of reward with respect to a
black box function called the (ground truth) oracle, which is expensive to
compute since it involves executing a real world process. In offline MBO we
wish to do so without assuming access to such an oracle during training or
validation, with makes evaluation non-straightforward. While an approximation
to the ground oracle can be trained and used in place of it during model
validation to measure the mean reward over generated candidates, the evaluation
is approximate and vulnerable to adversarial examples. Measuring the mean
reward of generated candidates over this approximation is one such `validation
metric', whereas we are interested in a more fundamental question which is
finding which validation metrics correlate the most with the ground truth. This
involves proposing validation metrics and quantifying them over many datasets
for which the ground truth is known, for instance simulated environments. This
is encapsulated under our proposed evaluation framework which is also designed
to measure extrapolation, which is the ultimate goal behind leveraging
generative models for MBO. While our evaluation framework is model agnostic we
specifically evaluate denoising diffusion models due to their state-of-the-art
performance, as well as derive interesting insights such as ranking the most
effective validation metrics as well as discussing important hyperparameters.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要