Welcome to Frontiers - a series where we bring top researchers, engineers, designers, and leaders working at the cutting edge of various fields to go deep on their work with the Manifold Community.
For this talk, our speaker was Pranav Guruprasad. Pranav is a Founding Research Engineer at Fig, and the Research and Engineering lead of the MultiNet research project at Manifold. At Fig, he works on advancing Multimodal Action Models for AI systems capable of executing complex, long-horizon digital tasks. Through MultiNet, he is developing comprehensive benchmarks to evaluate generalist Multimodal Action systems across diverse modalities, tasks, and domains, pushing the frontier of measurable and achievable progress in Multimodal AI.
Recording available here:
Abstract
Frontier AI models exhibit a critical performance paradox: near-ceiling scores on canonical benchmarks (HumanEval, MMLU) coupled with systematic failures on production deployments. Current benchmarks predominantly assess narrow, specialized competencies - competition mathematics, doctoral-level reasoning, software engineering proficiency - while neglecting the foundational capabilities required for robust multimodal action models.
In this talk, we’ll explore these gaps and introduce MultiNet v1.0, a new benchmark that addresses these issues by systematically profiling state-of-the-art architectures across three model classes (VLMs, VLAs, generalist models) on a unified suite spanning: common-sense reasoning, object detection, visual question answering, tool use, multi-agent coordination in discrete action spaces, and robotic manipulation and locomotion. Early results reveal stark capability disparities across modalities and task domains, exposing architectural prerequisites, training methodologies, and essential data distributions for true multimodal generalization. These findings, and the Multinet Benchmark more broadly, establish a blueprint for the next-generation of benchmarks designed for the functional intelligence era - where models must demonstrate not just knowledge, but reliable, generalizable action.
Frontiers is our invited speaker series. We invite researchers, engineers, and industry leaders working on the most interesting problems in foundational and applied AI. Check out our Events page for more info on this and other upcoming events! Want to get involved? Join our discord community here!
