Introducing MLPerf for Community-driven Benchmarking of Embedded Vision Systems

Rigorous and systematic benchmarking is key to developing efficient and reliable computer systems. While a solid methodology is necessary, it is not sufficient without representative workloads. In this talk, we describe the progress of developing MLPerf (mlperf.org), a new suite of machine learning workloads contributed to by a wide community across industry and academia. In particular, we focus on vision workloads being included in the edge inference division, targeting mobile and embedded systems. We describe key metrics for initially selected vision tasks (image classification, object detection), models and datasets; describe reference implementations and workflows, with sample data; and provide guidelines for submitters and evaluators. We also present insights on how decision makers may interpret MLPerf results to accelerate innovation in efficient systems for machine vision.