Google open-sources Agile Deliberation for humans and SI to tune subjective classifiers together
Prepared by Sintelligent with the help of SI. Published in SI: . Updated: .
Instead of guessing the perfect prompt, a person can show the model disputed examples step by step and communicate their own criteria more precisely.
The method is designed for categories without clear predefined boundaries, such as "healthy food" or misleading advertising. During concept scoping, a person breaks the category into subcategories and exclusions; the model then selects borderline examples, and user feedback automatically refines the classifier's rules. Google conducted 18 live 1.5-hour sessions and compared the approach with conventional zero-shot classifiers, an autonomous agent, and manual tuning. According to the company, Agile Deliberation improved labeling quality and consistency and reduced participants' fatigue and anxiety; the code and algorithms have been released as open source.
Sources 1