About the Talk

Physical AI—systems that perceive and act in the physical world—is advancing rapidly, and expectations are running ahead of results. The prevailing bet is that end-to-end models trained on massive datasets will do for robots and autonomous systems what large language models did for text. Ken Goldberg argues that this bet underestimates a structural obstacle, and that closing the gap will require a different approach to both data and system architecture.

The obstacle is data. Where language models had an internet containing roughly 100,000 years’ worth of human reading material, there is no comparable archive of physical interaction. Video carries limited and often misleading information about geometry, weight and material properties, and collecting data through human teleoperation does not scale. Goldberg will present new work from UC Berkeley’s AUTOLab on using vision-language models and agents to process images and video into detailed 3D physics models, iterating until simulation matches observed reality, generating training data at scale without additional robots or human demonstration.

He then turns to architecture, contrasting end-to-end learned systems with what he calls “Good Old-Fashioned Engineering”: modular designs in which learned components are integrated with proven algorithmic methods such as state estimation, outlier rejection and motion planning. Drawing on deployed systems, including package-sorting robots that have handled over 100 million packages, he will examine why modular designs are often more efficient and more debuggable under real-world constraints on power, latency and reliability.

Integrating these two cultures remains an open problem. Goldberg will close by asking whether AI coding agents, which generate inspectable and testable code rather than opaque policies, offer a path forward, and what the next generation of physical AI systems will demand of machine perception.