Writing by Radamés Roriz

I ran a benchmark for my personal project, and Claude Opus and GPT Sol are terrible for my use case.

Have you run your benchmark today? This is a routine reality for any team maintaining an LLM-powered feature. A new model drops every day, and if you're running a cloud model, it can get silently updated without notice.

In my case, I process books, currently running two pipelines: the first is fairly trivial, almost a two-step classification, and the other is more complex, with five-step data extraction.

Interestingly, a model from May (gemini-3.1-flash-lite) is the best fit for my specific case. Also worth highlighting: glm-5.3-flash (Ox Alpha), released recently. Meanwhile, more powerful models like gpt-sol and opus 5 perform noticeably worse.

Here's another reminder: stop blindly trusting large-scale published benchmarks. Those public benchmarks will rarely solve a problem specific to your company. It's on you to know what your business actually needs, run your own benchmark, and discover that reality looks quite different.

As always, I'm talking about LLMs embedded in a system here, not day-to-day productivity. I use opus 5 and gpt-sol every day and wouldn't trade them for anything else — but the tasks you do day-to-day and the tasks a system needs to solve are very different things.

How's your gold dataset looking today? If you use an LLM inside your system and don't have one, that's simply crazy.

In my last post I talked about the LLM-as-a-Verifier paper and promised to explain the difference between their technique and ActiveGenie's.

Both solve the exact same problem: when you ask an LLM to score two complex answers, it gives the same score to both. The paper measured this — 27% ties on Terminal-Bench V2 with a discrete scale. And a judge that ties ranks nothing.

The paper's solution: logprobs.

Instead of accepting the single score token the model spits out, they take the full probability distribution over the score tokens (on a 1-to-20 scale) and compute its expected value. The score stops being an integer and becomes continuous. Ties drop to zero, and just from that added granularity, accuracy goes from 73.1% to 77.5%.

ActiveGenie's solution: political debate.

Instead of asking for a score, I don't ask for a score at all. The two candidates enter a structured debate: "A" makes its case, "B" makes its case, "A" rebuts, "B" rebuts, each gives closing remarks, and an impartial judge decides. There's no tie because the format doesn't allow one — there's a winner and a loser.

And here's the trade-off nobody talks about.

Logprobs give a better signal: finer-grained, calibrated, and cheaper per comparison. There's just one catch — GPT-5.5 and Claude Opus don't expose logprobs. That means one extra call, extra latency, and a dependency on which provider exposes what.

The debate runs anywhere. It only needs text in, text out. That's exactly why ActiveGenie has been model-agnostic from day one: you swap providers in one config line and rewrite nothing. The price is a coarser signal — I know who won, but not by how much.

Except the two don't have to live apart.

In ActiveGenie's latest version I added logprobs support: if the chosen model exposes the distribution, the Comparator uses the continuous signal alongside the debate; if it doesn't, it falls back to the debate alone. You don't have to choose between a portable technique and a fine-grained signal — you use the best signal that model, on that call, can actually give you.

Which brings me back to the same point as always: the model matters less than the harness. With the right technique around it, you can run a smaller, faster, cheaper model and still deliver a consistent result.

The frontier isn't in the model. It's in what you build around it!

The framework around the model is worth more than the frontier model itself.

Researchers from Stanford, UC Berkeley, and NVIDIA Research showed that you can beat GPT-5.5, Claude Opus 4.8, and Gemini 3.1 Pro without upgrading to a better model. Instead, they used a tournament technique where candidate responses face off in 1v1 matchups.

It's called Probabilistic Pivot Tournament, and it works like this:

1. The model generates N candidate responses for the same prompt.
2. A fast initial pass assigns a baseline score to each response.
3. Based on that ranking, responses split into two groups: the top performers (the pivot group) and the rest.
4. Each non-pivot is paired against every pivot, while pivots compete among themselves in 1v1 duels.
5. The duel outcomes create the final ranking, selecting the single best response.

It isn't fast, and it isn't cheap. But it works: 86.5% on Terminal-Bench V2 compared to 84.7% for GPT-5.5, and 78.2% on SWE-Bench Verified compared to 76.8% for Opus 4.5.

This is what changes everything: top performance on a task no longer relies solely on the frontier model, but on the harness and engineering around it.

If you have followed my work for a while, you already know this tournament approach. It's the core method ActiveGenie uses to deliver consistent results. I wrote the initial tournament logic back on February 3, 2025, nearly a year and a half ago.

There is one key difference: ActiveGenie uses political debate while the paper uses logprobs. That topic deserves a dedicated post of its own.

Until then, try ActiveGenie. Stop relying on expensive frontier models alone. Pair a leaner model with the right system architecture instead.

Paper: LLM-as-a-Verifier

Everyone is concerned about the cost or speed of LLMs, but let me say one thing: that doesn't matter if you're not building the LLM yourself.

For us, builders and engineers, we need to care about delivering real value. Something that people actually want, without the slop.

We don't need new models; we need to focus on what really matters, consistent outputs!

It is impressive how AI can do everything, but at the same time, finding the right place for it is incredibly difficult. Anyone building with AI faces the paradox of choice with every new interaction. Even in established use cases like coding assistants, there are still countlessly subtle ways to get it wrong.

In my view, the ideal use of AI isn't as flashy or polished as tech giants portray. It sits in our routine human-in-the-loop workflows inside the platforms we build. AI doesn't need to be perfect; it just needs to match the accuracy of an average human performing the task. This is about scale, not flawless execution.

Your top domain experts act as the guides, defining best practices and pointing out which shortcuts work. Breaking down high-level human responsibilities into small, structured tasks is what software engineering has always been about. Now, we use AI to run those steps.

Take vendor onboarding as an example. Is selecting a supplier purely about price? In reality, we need to:

• Parse unstandardized documents: read scanned or legacy Articles of Association to extract Partners and Ultimate Beneficial Owners (UBOs) despite massive format variance.
• Evaluate risk severity: read legal case summaries to determine whether an entry presents genuine reputational risk or mere administrative noise.
• Clean complex datasets: determine if "Linkana Tech" and "Linkana S.A." refer to the same entity across duplicate databases by matching addresses and context where exact string matches fail.

Each of these steps can be assisted by AI, and in well-structured environments, fully automated. The real magic happens when this human-in-the-loop mechanism bridges two systems: the moment a budget is approved, the payment system finalizes the transaction, triggers the vendor, and notifies the rest of the supply chain.

We hear a lot about AGI, but for most of us, it isn't what matters.

I don't believe we are on the verge of reaching AGI, whatever that definition may be, but failing to reach that goal doesn't reduce the impact AI has on society. Even if we stopped launching new LLMs today, the world will be changed forever. What is currently holding most businesses back isn't a lack of AI capability; it's finding the right use cases and achieving consistency.

The best place to implement AI is within existing software and "human-in-the-loop" tasks. In some cases, AI can fully automate these tasks; in others, it serves to significantly speed up the process.

The "perfect" opportunity exists where a human currently acts as a bridge between two systems. Automating this human-in-the-middle connection will drastically disrupt the market. For this to work, the AI doesn't need to be perfect; it simply needs to match the consistency and performance of a human employee at scale.

We don't need "smarter" models to achieve this. We need structured processes and a structured way of thinking. This is the current challenge: human-in-the-loop tasks are often poorly documented, and every person performs them differently.

If you want my advice: Record, document, and structure the best practices for every human-in-the-loop task in your platform as thoroughly as possible. That data will be pure gold for future automation.

Overfitting in LLMs is not as problematic as everyone claims. Here are my predictions for 2026:

Think about how we learn. What is overfit learning if not our school grading system? We teach in a highly specific way, expecting students to apply exact techniques to a test. We all follow this pattern for at least 15 years of our lives.

The hope is that by applying the same formulas repeatedly, some will eventually discover how to actually learn and find the techniques that work for them, whether that's singing a math formula or rewriting a whiteboard three times.

Humanity has almost always taught and learned this way. It is nearly impossible to avoid overfitting in AI when we haven't even discovered a better way for ourselves.

This overfit learning has brought us a society of specialized jobs, and that will hold true for AI as well. If I were to bet on anything for 2026, it would be the rise of specialized models.

Models won't just get larger; they will get smaller and more specialized in specific techniques or tasks. If a task is in high demand by the community, there will surely be a model dedicated to it.

Consider these examples:

• Chat Specialists: This is the most obvious case. We will see models with pre-built actions and high accuracy specifically for Sales or Customer Success.
• Assessment Resolvers: As benchmarking for AI and humans becomes more common, we'll see specialist models designed to navigate diverse question types, incorporating images, complex math, and multi-choice logic.
• AI Reviewers: We are currently overwhelmed by reviews for everything: films, PC boards, best smartphones. AI can perform this flawlessly, provided it has the right context and direction to create a high-quality critique.

In what other areas do you think AI will become a specialist?

I've always seen Campus Code as a major reference and a constant source of learning. Today, I was proud to be noticed and featured by them! Every blog post and article they share has given me clarity and valuable insights throughout my career. This shoutout reinforces the deep respect I have for their work and the impact they bring to the developer community, far beyond the classroom.

Truly grateful!

Controlling GenAI is hard. There's no manual for it.

That's what makes this space so exciting. It's about finding creative, non-obvious solutions to brand-new problems. The companies that master this will define the next era of technology.

Beyond productivity boosts, the real challenge is creating entirely new businesses with reliable AI at their core.

If you're building a GenAI feature, ActiveGenie is designed to help you succeed. Reach out with a message, I'm always happy to help fellow developers navigate this space.

The Year of Agents is for non-technical people; I'm waiting for the Year of Unstructured Data.

We're currently in the so-called "Year of Agents," and the world is captivated by their new capabilities. The ability to create web pages, build apps, and connect with APIs is something agents are doing flawlessly.

As a result, non-technical people can now achieve far more than ever before, and all the financial investment and hype stems from this very fact. But the agent hypemen forgot two key things:

• First, agents aren't doing anything a developer couldn't already do. The feeling that "you can do anything" is a part of our daily lives as developers, which is why so many developers are founders.
• Second, while non-developers can now do more, developers can do even more not because of agents, but because of how easy it's become to work with unstructured data.

If we're already in the Year of Agents, just imagine what will happen when they discover the vast amount of unstructured data that exists, and how much they can do with it.

I'm ready for that! Let's build that future together.