Together Compiler uses agents to continuously optimize inference for live traffic
Prepared by Sintelligent with the help of SI. Published in SI: . Updated: .
Inference can continuously adapt to new models, chips, and real workloads instead of relying on one-time manual tuning.
According to Together AI's measurements, the TKCC system increases output speed for open models by up to 2.6 times in less than a day and reaches 1,111 tokens per second on MiniMax M3.1-Flash and 1,205 on DeepSeek v4.1-Flash in the low-latency InferenceX test. Persistent engines allow the tested forward pass to be replaced without restarting the entire server, while a group of agents proposes optimizations in parallel; the company estimates a 252-fold increase in experiment throughput. TKCC accounts for the model, hardware, and changing traffic shape and already operates in Together's production inference. Limited beta testing is available for optimizing customers' own dedicated endpoints.
Sources 2