Vision-Language Models on the Edge


Date:  Wednesday, May 21

Start Time: 2:40 pm

End Time: 3:10 pm

In this presentation, we provide an overview of vision-language models (VLMs) and their deployment on edge devices using Hugging Face’s recently released SmolVLM as an example. We will examine the training process of VLMs, including data preparation, alignment techniques and optimization methods necessary for embedding visual understanding capabilities within resource-constrained environments. We will explain practical evaluation approaches, emphasizing how to benchmark these models beyond accuracy metrics to ensure real-world viability. And to illustrate how these concepts play out in practice, we’ll share data from recent work implementing SmolVLM in an edge device.

Track

Session Speakers

  • Dr. Cyril Zakka completed his medical training at the American University of Beirut Medical Center, where he founded and served as the Director of the Artificial Intelligence in Medicine program. As a postdoctoral fellow in the Department of Cardiothoracic Surgery at Stanford University, his work has focused on building, training and evaluating multimodal large language models for clinical medicine, as well as foundation models for surgery and cardiac imaging. His work has been featured in highly regarded journals and conferences such as NEJM AI, ML4H and AATS. Prior to his medical training, Dr. Zakka was heavily involved in software development, and he continues his work building open-source tools and applications. He currently heads the Medical AI Research team at Hugging Face, responsible for medical AI research across the United States and Europe.


Share

Vision-Language Models on the Edge