All news

KAIST: diffusion language model reads lips with 19.4% error rate on LRS3

Prepared by Sintelligent with the help of SI. Published in SI: . Updated: .

Generation order turned out to matter more than scaling the model: filling in masks by diffusion delivered a new best result in lip reading, which means speech recognition will work where there is no sound or it is unintelligible.

Professor No Yong Man's team at KAIST announced on November 15 a visual speech recognition system that, instead of generating words sequentially, first decodes the positions with the clearest articulation and uses them as context for ambiguous spots. It runs on a diffusion large language model that repeatedly fills in masked positions; training is two-stage — first aligning lips with text, then modeling sentence length. On the LRS3 dataset (433 hours of video with reference transcripts) it achieved a word error rate of 19.4% — according to the university, the best result among methods trained only on LRS3 with transcripts. The paper "Diffusion Large Language Models for Visual Speech Recognition" received an Outstanding Paper Award at EMNLP 2026; the first author is Yeo Jeong-hun.