I ran a benchmark for my personal project, and Claude Opus and GPT Sol are terrible for my use case.

I ran a benchmark for my personal project, and Claude Opus and GPT Sol are terrible for my use case.

Have you run your benchmark today? This is a routine reality for any team maintaining an LLM-powered feature. A new model drops every day, and if you're running a cloud model, it can get silently updated without notice.

In my case, I process books, currently running two pipelines: the first is fairly trivial, almost a two-step classification, and the other is more complex, with five-step data extraction.

Interestingly, a model from May (gemini-3.1-flash-lite) is the best fit for my specific case. Also worth highlighting: glm-5.3-flash (Ox Alpha), released recently. Meanwhile, more powerful models like gpt-sol and opus 5 perform noticeably worse.

Here's another reminder: stop blindly trusting large-scale published benchmarks. Those public benchmarks will rarely solve a problem specific to your company. It's on you to know what your business actually needs, run your own benchmark, and discover that reality looks quite different.

As always, I'm talking about LLMs embedded in a system here, not day-to-day productivity. I use opus 5 and gpt-sol every day and wouldn't trade them for anything else — but the tasks you do day-to-day and the tasks a system needs to solve are very different things.

How's your gold dataset looking today? If you use an LLM inside your system and don't have one, that's simply crazy.