AgentSpec addresses speed degradation in speculative decoding for LLM agents during large batch inference. The authors identify high token rejection rates and under-utilized dynamic budgets as primary causes. The proposed algorithm optimizes these factors to improve inference efficiency without sacrificing generation quality.
- Speculative decoding suffers speed drops in large batches due to token rejection.
- Dynamic token budgets are often under-utilized in current agent inference setups.
- AgentSpec targets these specific bottlenecks to restore inference efficiency.
- Optimization maintains generation quality while handling higher batch sizes.