NVIDIA opens Cosmos3-H-Dreams, a model that generates video of a surgical operation in real time from commands controlling a robot's instruments
Prepared by Sintelligent with the help of SI. Published in SI: . Updated: .
Robotic surgeons get a virtual training ground, but it is not yet suitable for real operations.
According to NVIDIA's blog, the model is described as the first real-time generative surgical simulator, trained on clinical recordings of cholecystectomy (gallbladder removal) and allowing instruments to be controlled: it shows tissue deformation and occlusions as instruments move, rotate and close. Training used video and kinematics from the CMR Surgical Versius robot and the Open-H-Embodiment dataset; the base is Cosmos3-H-Embodied-Nano with 16 billion parameters, video at 832x480 and 10 Hz, running on a single RTX PRO 6000 graphics card. The code and weights were published on October 20. The authors stress that it is a research simulator, not a validated system for treating patients.
Sources 1