Nvidia shows a method that cuts the success of "hidden command" attacks on SI assistants from 38.1% to 0.1% on one test
Prepared by Sintelligent with the help of SI. Published in SI: . Updated: .
Researchers found a way that almost completely stops one kind of deception of SI assistants through hidden commands in text, but it works only with open models.
According to Nvidia's description, the NXDirection method subtracts the "instruction direction" from the model's internal activations on tokens of untrusted external content, so that the assistant reads an email, page or document as data rather than commands. On nine open models, the share of successful attacks on the BIPIA test fell from 38.1% to 0.1% (usefulness on ordinary tasks went from 78.4% to 76.4%, per AI Weekly), and on the tool-using AgentDojo test from 28.4% to 3.7%; for some models on individual tasks the residual share of successful attacks reached 13.1%. The method requires access to the model's hidden states, so it suits only open or self-hosted deployments, not closed APIs. Nvidia calls it "a promising defense, not a guarantee"; the results are the authors' data.
Sources 2