Metric Mirages in Cell Embeddings

Although biological studies increasingly rely on embeddings of single cell profiles, the quality of these embeddings can be challenging to assess. Such evaluations are especially important for avoiding mis- leading biological interpretations, assessing the accuracy of integration methods, and establishing the zero-shot capabilities of foundational models. Here, we posit that current evaluation metrics can be highly misleading. We show this by training a three-layer perceptron, Islander, which outperforms all 11 leading embedding methods on a diverse set of cell atlases, but in fact distorts biological structures, limiting its utility for biological discovery. We then present a metric, scGraph, to flag such distortions. Our work should help learn more robust and reliable cell embeddings.