Multimodal Zero Shot

My current project at the Singh Lab is a multimodal model which attempts to use contrastive learning in an attempt to learn an association between medical biopsy images and their clinical diagnoses. Our goal for this project is to be able to create a model which can associate different data modalities (namely biopsy image, clinical text, and genomic), and peform zero shot predictions.

Zero shot prediction is a task where models make predictions on datapoints from classifications unseen during training. The motivation for our zero shot predictive capability comes from how the model shares a singular latent space across all modalities. Theoretically, a well-trained constrastive learning model would be able to encode any new data into a relevant part of the shared latent space based on the information stored in that data, and then would be able to decode a nearby encoding into a different modality.

Recently, ImageBind demonstrated the capability for Deep Learning models to learn a shared latent space across multiple data modalities, even in cases where most data points do not contain all modalities.

Through this project, I have gained a lot of hands-on experience on working with contrastive learning models, from preprocessing massive TCGA datasets with multiple modalities, to coding, training, and tuning complex models that take advantage of pre-existing work like pre-trained CLIP encoders.