Image Analysis and Report Generation for Medical Co-Pilots

Medical Co-Pilots are a combination of conversational AI technologies with medical data analysis technologies that are finding a wide range of use cases in nearly every aspect of healthcare, from radiological report generation to interactive control and query of surgical robotic systems to medical student training.

NVIDIA’s Project MONAI has a long history of proven successes for medical image analysis AI, from classifying lung x-ray images for COVID to winning medical image challenges for abdominal aortic segmentation in CT images to cell labeling in pathology images to tracking instruments in robot-assisted laparoscopic videos.

In MONAI M3, NVIDIA has pioneered the use of trained medical image analysis models as experts that contribute annotations or classifications to the conversational context of large vision and language models such as LLAMA3. M3 is available as part of the open-source MONAI platform. Additionally, NVIDIA has used M3 to enhance the VILA LLM and release VILA-M3 as a pre-trained foundational model for brain tumor MRI image interpretation. VILA-M3 is setting new standards for accuracy and ease of fine-tuning among medical co-pilots.

Combining VILA-M3 with NVIDIA’s edge and cloud accelerated computing platforms such as Holoscan and NVAIE, researchers and application developers can:

• Evaluate the improved accuracy provided by integrating medical imaging AI experts as resources for additional context in LVM conversations
• Refine or extend the capabilities of VILA-M3 by refining the foundational VILA or by adding new MONAI-trained medical imaging AI models as additional experts covering other modalities or diseases.
• Explore alternative LLM+expert paradigms for the continual advancement of LVMs as medical imaging co-pilots.