Latency Optimization

Online Learning for Cost-Efficient LLM Routing

This article describes how an LLM gateway optimizes model routing using an online learning approach. It specifically employs Thompson sampling over lognormal latency posteriors to dynamically select the fastest and most reliable LLM route for each incoming request. This strategy is designed to achieve cost-efficient operation by ensuring optimal performance and reliability in real-time.