Driven by advances in AI, several groups in astronomy are developing large Foundation Models for Astrophysics, large general purpose ML models for performing many tasks. Our group is involved these efforts, in particular connecting these models to natural language models (LLMs). Little work has been done, however, in evaluating the performance of these models. The aim of this problem is to develop a set of benchmarks across a range of astrophysical applications (gravitational lending, galaxy morphology, photometric redshift determination, stellar parameter determination) to test the performance of current and future models.

The SUDS scholar would work on gathering relevant benchmark data sets from the astronomical literature, starting from some that we have already used and then expand to others, write code to run these through existing astronomical foundation models (such as AION-1), and create summary statistics and visualizations of the foundation models’ performance on these benchmarks. Finally, the SUDS Scholar will create an easily accessible resource for others to run the benchmarks on their own models (e.g., sharing it on huggingface).