2025/03/05 – Sonya Hanson

The Inaugural Flatiron Institute Cryo-EM Conformational Heterogeneity Challenge

Despite the rise of single particle cryo-EM as a premier method for resolving macromolecular structures at atomic resolution, methods to address molecular heterogeneity in vitrified samples have yet to reach maturity. With an increasing number of new methods to analyze the multitude of heterogeneous states captured in single particle images, a systemic approach to benchmarks and metrics in this field is needed. With this motivation, we issued a challenge to the community to analyze two curated cryo-EM image stacks of the thyroglobulin molecule with conformational heterogeneity: the first was a typical experimental dataset, and the second was synthetically generated, allowing control over the distribution of molecular structures in the particle images. This synthetic dataset also enabled direct comparison between participants’ submissions and the ground truth molecular structures and distributions. Participants were asked to submit 80 molecular volumes representing the heterogeneous ensemble in the dataset and estimate their respective populations in the particle stack provided. Participation in the challenge was strong, with submissions from nearly all developers of heterogeneity methods, resulting in 40 submissions across both datasets. Submissions qualitatively exceeded expectations, with the molecular motions identified by methods resembling both each other and the ground truth distribution. However, quantitatively assessing these similarities was a challenge in and of itself.  In the process of assessing the submissions to this challenge, we developed several validation metrics, most of which require reference to the underlying ground truth volumes, but one which does not. These approaches allowed us to assess the similarity and accuracy in map quality, molecular motions, and distribution estimates of different submissions. These metrics and the efforts of all participants will help chart a path forward for the improvements of heterogeneity methods for cryo-EM and future challenges to test these new methods as they continue to be developed by the community.