All news

Sber opens Pollux Judge, a bilingual 4-billion-parameter judge model

Prepared by Sintelligent with the help of SI. Published in SI: . Updated: .

Evaluating dialogues and agents against custom criteria could become cheaper and more accessible for local deployment without a gigantic judge model.

Pollux evaluates answers in Russian and English using criteria, scales, and formats defined by the user. It handles not only a prompt-answer pair but also dialogues, RAG answers with retrieved context, and chains of tool calls. The model was trained with GRPO on bilingual data and can be deployed locally or through an OpenAI-compatible API. The accompanying benchmark contains 35 task groups and 2,115 expert-written prompts. According to Sber, on an independently expert-labelled set, Pollux's Spearman correlation with specialists was 0.744, higher than the tested general-purpose models with up to 1.1 trillion parameters. The code, demo, criteria, and labels are open; DeepEval and LightEval are supported.

Sources 1