In this presentation, we will provide an overview of workflows for deploying compressed deep learning models, starting with PyTorch and creating native C++ application code running in real-time on embedded hardware platforms. We illustrate these workflows on smartphones with real-world examples targeting ARM-based CPU, GPUs, and NPUs as well as embedded chips and modules like the NXP i.MX8+ and NVIDIA Jetson Nano. We examine TorchScript, architecture-side optimizations, quantization and common pitfalls. Additionally, we show how the PyTorch deployment workflow can be extended to conversion to ONNX and quantization of ONNX models using an ONNX Runtime. On the application side, we demonstrate how deployed models can be integrated efficiently into a C++ library that runs natively on mobile and embedded devices and highlights known limitations.
Deploying PyTorch Models for Real-time Inference On the Edge
Session Speakers
-
Moritz August
CDO, Nomitri GmbH
In his role as CDO at Nomitri and as a leading deep learning scientist at Mindpeak, a company building AI tools for digital pathology, Moritz August has acquired a diverse set of skills: Optimizing state-of-the-art DL models for embedded deployment, designing DL algorithms for vision tasks, and architecting and coding of deep learning software stacks in low-level PyTorch. He has also established processes for the collection and management of training data, including annotation concepts, related tools, training and hiring of annotation teams. Moritz earned his PhD at TUM in machine learning techniques for numerical quantum mechanics. He has 8+ years of experience in research and development, where he implemented DL software stacks in Theano, PyTorch and Tensorflow with focus on modularity, maintainability and reusability. He is also experienced in evaluating advances in deep learning research for their applicability to problems or products, as well as in rapid prototyping of machine learning solutions.
