In 2,250 request tests of OpenRouter’s smart LLM routing, Auto had inference costs approximately 59% and 81% lower than GPT-4.1 and Claude, respectively. However, Auto’s strict format pass rate was only 25%, with 107 cases of truncation. When extracting the final answer was permitted, 256 of 320 functional evaluations passed. This suggests that routing can reduce costs, but may not directly meet production requirements for format compliance and completeness.
Source:MediumGenerative AI · Read Original
Leave a Reply