This project proposes to address reliability issues of current large vision-language models (VLMs), when utilised in generative contexts, towards enabling their safe deployment in healthcare applications, such as the summarization of MRI and X-Ray images, retrieval systems for diseases, conditions, or treatments, and modelling censored data.The accurate estimation of well-calibrated uncertainty levels is of paramount importance to deliver dependable and actionable predictions. Clinicians may base their trust or scepticism regarding the recommendations of such automated systems on the associated prediction uncertainty values. However, the existing methods for uncertainty estimation are ill-suited for VLMs. Research in this direction has recently gained momentum and shows promising results on publicly available datasets. Nevertheless, the proposed methods often incur significant retraining costs or are tailored to specific use cases, limiting the generalizability of VLMs. Additionally, these studies do not encompass complex healthcare applications into their evaluations.
We hypothesise that the field of multimodal uncertainty estimation can draw valuable insights from the extensive body of literature concerning probabilistic unimodal models. A comprehensive examination of how vision-language alignment techniques impact the probabilistic characteristics of these unimodal models will provide insight that can be leveraged to facilitate the generation of cross-modal uncertainty estimates. This line of investigation will uncover efficient alignment techniques, which will significantly enhance the reliability of vision and language encoders when integrated into a unified VLM (Vision-Language Model) framework. The investigation ultimately aims to expedite the integration of VLM systems into clinical workflows thus enabling more efficient patient care.
