The ability to quickly and accurately identify covariate shift at test time is a critical and often overlooked component of safe machine learning systems deployed in high-risk domains. While methods exist for detecting when predictions should not be made on out-of-distribution test examples, identifying distributional level differences between training and test time can help determine when a model should be removed from the deployment setting and retrained. This project will evaluate the Detectron model https://github.com/rgklab/detectron on a wide variety of datasets from the WILDS benchmark (https://wilds.stanford.edu/) and the SUBPOPBench (https://github.com/YyzHarry/SubpopBench) benchmark dataset. This will enable ML researchers to identify promising next steps to build guardrails to protect against distribution shift. The student will be responsible for coding, designing and running experiments using Pytorch on a large scale GPU cluster to study, compare and contrast different methods to detect when a machine learning model might fail on publicly available datasets. This project will introduce the student to slurm, pytorch and empirical research in machine learning.