Smart LLM Routing Tested: How Does OpenRouter Perform on Cost and Reliability?

Written by

in

In 2,250 request tests of OpenRouter’s smart LLM routing, Auto had inference costs approximately 59% and 81% lower than GPT-4.1 and Claude, respectively. However, Auto’s strict format pass rate was only 25%, with 107 cases of truncation. When extracting the final answer was permitted, 256 of 320 functional evaluations passed. This suggests that routing can reduce costs, but may not directly meet production requirements for format compliance and completeness.

Source:MediumGenerative AI · Read Original

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *