Back to list
LARYBench Released: Defining the ImageNet for Embodied Action Representation and Measuring Generalization from Human Videos
Research BreakthroughEmbodied AIComputer VisionMachine Learning

LARYBench Released: Defining the ImageNet for Embodied Action Representation and Measuring Generalization from Human Videos

The Meituan Technical Team has officially released LARYBench (Latent Action Representation Yielding Benchmark), a systematic evaluation framework designed to guide the learning of general latent action representations from large-scale visual data. This benchmark marks a significant milestone in embodied AI, often referred to as the 'ImageNet' for action representation. Experimental findings within the benchmark reveal that general vision models significantly outperform specialized embodied AI action expert models in both action generalization and control precision. Crucially, the research demonstrates that embodied action representations can emerge directly from large-scale human video data, providing a new methodology for measuring how AI systems translate visual observation into physical action capabilities.

美团技术团队
Share

Key Takeaways

  • Introduction of LARYBench: A systematic benchmark designed to evaluate latent action representations learned from large-scale visual datasets.
  • Superiority of General Models: General vision models demonstrate significantly better performance in action generalization and control precision compared to specialized embodied AI expert models.
  • Emergence from Human Videos: The benchmark proves that embodied action representations can emerge from observing large-scale human video data without explicit action labels.
  • A New Industry Standard: LARYBench is positioned as the 'ImageNet' for the embodied AI field, providing a standardized metric for generalization and precision.

In-Depth Analysis

The Framework of LARYBench

LARYBench, which stands for Latent Action Representation Yielding Benchmark, represents a systematic shift in how the AI industry evaluates embodied intelligence. By focusing on "latent action representation," the benchmark addresses the critical gap between seeing an action and understanding the underlying mechanics required to replicate it. The system is designed to guide the learning process from massive visual datasets, transforming passive observation into actionable intelligence. By establishing a systematic evaluation protocol, LARYBench allows researchers to measure how effectively a model can extract action-oriented features from raw pixels, a process that is fundamental to the development of autonomous agents and robotics.

General Vision Models vs. Specialized Experts

One of the most striking revelations from the LARYBench experimental results is the performance gap between general-purpose vision models and specialized embodied AI action expert models. Traditionally, the industry has leaned toward developing niche models trained specifically for robotic control or embodied tasks. However, LARYBench demonstrates that general vision models—those trained on broad, diverse visual data—possess a superior ability to generalize actions across different scenarios. Furthermore, these general models exhibit higher control precision. This suggests that the foundational visual features learned by large-scale general models are more robust and adaptable for embodied tasks than the features captured by models with a narrower, task-specific focus.

Action Representation Emergence from Human Videos

The benchmark provides empirical evidence for a transformative concept in AI: the emergence of embodied action representations from human video data. This implies that AI models do not necessarily require direct robotic telemetry or specialized sensor data to understand physical movement. Instead, by processing large-scale videos of humans performing various tasks, these models can synthesize a latent understanding of action. This "emergence" is a critical finding, as it suggests that the vast repositories of human video content available globally can serve as a primary training ground for embodied AI, significantly lowering the barrier to training sophisticated robotic systems.

Industry Impact

The release of LARYBench is poised to redefine the development trajectory of embodied AI. By providing a standardized metric—akin to what ImageNet did for computer vision—it allows for objective comparisons between different architectural approaches. The finding that general vision models excel in this domain may lead to a consolidation of research efforts, where the focus shifts from building specialized action models to fine-tuning large-scale general vision models for physical tasks. This could accelerate the deployment of more precise and adaptable robots in real-world environments, as the industry moves toward leveraging human video data as a scalable resource for learning complex physical interactions.

Frequently Asked Questions

Question: What is the primary purpose of LARYBench?

LARYBench is a systematic evaluation benchmark designed to measure and guide the learning of general latent action representations from large-scale visual data, serving as a standard for the embodied AI industry.

Question: Why are general vision models performing better than specialized models in this benchmark?

According to the experimental results, general vision models show significantly better action generalization and control precision, suggesting that broad visual training provides a more robust foundation for understanding actions than specialized, task-specific training.

Question: Can AI learn to move just by watching human videos?

Yes, LARYBench demonstrates that embodied action representations can emerge from large-scale human video data, allowing models to learn the latent structures of action through visual observation.

Read original article

Related News

Research Breakthrough

OpenAI Shares Model Results on Kakeya Conjecture and Scientific Understanding with Academic Communities

OpenAI has shared model results regarding complex scientific problems, including the Kakeya conjecture, the Exponential Time Hypothesis, and quantum chromodynamics. By collaborating with academic communities, OpenAI aims to show how researchers can verify these model results and build upon them to advance scientific understanding.

From LOOP to Graph Exploration: Meituan Unveils the Evolution of Its AutoResearch Self-Evolving Architecture
Research Breakthrough

From LOOP to Graph Exploration: Meituan Unveils the Evolution of Its AutoResearch Self-Evolving Architecture

The Meituan Technical Team has detailed the architectural evolution of its autonomous algorithm research system, transitioning from a single-threaded loop to a sophisticated CyclicGraph Engine. Originating from Andrej Karpathy's open-source AutoResearch prototype, the enterprise platform underwent three major architectural iterations to overcome context explosion and operational instability. By introducing an asynchronous state machine and a Spawn-Converge mechanism, the system accelerates research iterations by 18 times compared to earlier architectures. Deployed across two distinct production modeling projects, the self-evolving system slashed end-to-end task turnaround time by 85% and 83% against human engineer baselines. Notably, one autonomous solution achieved full production deployment with documented business gains, while another surpassed manual baselines across all 12 predictive targets, demonstrating the viability of autonomous AI-driven algorithm design.

Optimizing High-Concurrency Generative SLM Inference Engines Under Strict Latency Constraints for Search and Recommendation
Research Breakthrough

Optimizing High-Concurrency Generative SLM Inference Engines Under Strict Latency Constraints for Search and Recommendation

Meituan's technical team has unveiled an advanced high-concurrency inference engine tailored for Small Language Models (SLMs) with 0.5B to 1B parameters deployed in online search and recommendation systems. While traditional large language model serving architectures are tailored for open-ended text generation, search and recommendation platforms operate under extreme low-latency boundaries where standard systems falter. By categorizing tasks into bounded generation workloads—such as prefill-only classification and fixed-step beam search—the engineering team restructured the execution path using a static Directed Acyclic Graph (DAG) abstraction. Integrated with PyTorch AOTInductor compilation, packed prefill, intra-request KV cache sharing, and selective FP8 quantization, the engine keeps end-to-end latency strictly below 20 milliseconds. The system achieves a prefill-only effective Model FLOPs Utilization (MFU) of 67% and delivers a tenfold throughput improvement over SGLang, establishing a scalable paradigm for enterprise AI deployment.