NVIDIA opens four Cosmos 3 models for interactive robot and driving simulation
Prepared by Sintelligent with the help of SI. Published in SI: . Updated: .
Researchers have an open foundation for simulators where a robot's or vehicle's action changes the next observation, rather than simply generating prerecorded video.
All four Cosmos 3 Nano checkpoints contain 16 billion trainable parameters and are distributed under the OpenMDW 1.1 license. Transfer-Auto generates synchronized data from up to 11 cameras and LiDAR with controls for weather, lighting and road scenes; Sim-Auto turns these into streaming observations for interactive driving scenarios. Sim-Transfer creates photorealistic video from a physics simulator's spatial controls, while Sim-Bimanual predicts the next frames from the actions of a two-armed robot or camera. NVIDIA is also publishing recipes for fine-tuning on users' own sensors and robots. The company specifically notes that the ability of such simulations to reliably train and evaluate policies still needs to be measured.
Sources 1