[{"data":1,"prerenderedAt":46},["ShallowReactive",2],{"mdc-eio57h-key":3},{"data":4,"body":5},{},{"type":6,"children":7},"root",[8,16,21,26,31,36,41],{"type":9,"tag":10,"props":11,"children":12},"element","p",{},[13],{"type":14,"value":15},"text","I ran a benchmark for my personal project, and Claude Opus and GPT Sol are terrible for my use case.",{"type":9,"tag":10,"props":17,"children":18},{},[19],{"type":14,"value":20},"Have you run your benchmark today? This is a routine reality for any team maintaining an LLM-powered feature. A new model drops every day, and if you're running a cloud model, it can get silently updated without notice.",{"type":9,"tag":10,"props":22,"children":23},{},[24],{"type":14,"value":25},"In my case, I process books, currently running two pipelines: the first is fairly trivial, almost a two-step classification, and the other is more complex, with five-step data extraction.",{"type":9,"tag":10,"props":27,"children":28},{},[29],{"type":14,"value":30},"Interestingly, a model from May (gemini-3.1-flash-lite) is the best fit for my specific case. Also worth highlighting: glm-5.3-flash (Ox Alpha), released recently. Meanwhile, more powerful models like gpt-sol and opus 5 perform noticeably worse.",{"type":9,"tag":10,"props":32,"children":33},{},[34],{"type":14,"value":35},"Here's another reminder: stop blindly trusting large-scale published benchmarks. Those public benchmarks will rarely solve a problem specific to your company. It's on you to know what your business actually needs, run your own benchmark, and discover that reality looks quite different.",{"type":9,"tag":10,"props":37,"children":38},{},[39],{"type":14,"value":40},"As always, I'm talking about LLMs embedded in a system here, not day-to-day productivity. I use opus 5 and gpt-sol every day and wouldn't trade them for anything else — but the tasks you do day-to-day and the tasks a system needs to solve are very different things.",{"type":9,"tag":10,"props":42,"children":43},{},[44],{"type":14,"value":45},"How's your gold dataset looking today? If you use an LLM inside your system and don't have one, that's simply crazy.",1788360944717]