⚙️ "You are technically correct, the best kind of correct..."
Today I ran across an article from tech blogger Dan Luu about "The Benchmark-pocalypse." Luu had experimented with an LLM-generated regex engine and tested it against Rebar, a comprehensive benchmark suite. The engine out-performed the top scorer by 40%!
Only, it didn't actually do that. When he applied the same regex engine to a different benchmark suite, he found that it was on average 4x slower, and that was on the benchmarks it was actually able to complete. The LLM had gamed the numbers because it knew what benchmarking suite it was going to be compared to. In fact, Luu had gone so far as to instruct the LLM not to overfit its data to that suite... but it did it anyway.
LLMs are statistical inference engines with an optimization loop. Whether you intend to or not, they will tailor their output to maximize their optimization, which means they will game the system if only because gaming metrics is generally easier than actually optimizing your code. And this, it's worth noting, is happening when the developer has good intent. The problem compounds if there is a bad actor in play.
This is bad, and not just because cheating on the test is bad. Optimizations that try to maximize a specific metric can actually harm the application or user experience. This is easy to see with a simple thought experiment. Suppose you have an application that frequently interacts with a database, and you want to optimize database queries. You tell the LLM to try to bring down the average transaction time. So it un-batches record requests. After all, retrieving a single record is much faster than retrieving a thousand. The metric has been dramatically improved, but the overall performance is worse because grabbing a thousand records one at a time is far slower than grabbing them in batches, OMG what is wrong with you, you infernal machine!
Does that match up with your experience of the tech industry? As overall performance degrades (which is something we see commonly in LLM-generated code) the indicators of its performance are improving, which leads to the inevitable awkward conversation in which the customer says "your product sucks now" but the manager says "actually, our metrics say that you should love it." Indeed, this is already happening. Companies are reporting suspiciously high increases in productivity while the general vibe around the internet is "everything is slow and awful these days."
Luu takes his results to their logical conclusion. If cheating on the test is something that LLMs do both trivially and inherently, then all metrics become suspect, even long-standing tried-and-true metrics. This is Goodhart's Law—when a measure becomes a target, it ceases to be a good measure—only at an unprecedented scale. It's what Luu terms "the benchmarkpocalypse" because it nullifies one of the fundamental pillars of software production: measurability. And also because he hates spellcheckers, I guess.
"How do we measure success?" is an age-old question but one that is particularly thorny to answer in technology. A lot of IT work is purely internal; "units sold" is not a measure that is available to us. But computers produce a lot of numeric output, and finding ways to translate those numbers into KPIs (Key Performance Indicators) has become its own cottage industry. And it's a particularly salient question for me because my team is tasked with accelerating the other delivery teams. And I need to be able to periodically show my stakeholders that we're actually doing our jobs so they'll keep giving us money. Could I game some numbers to make myself look good? I mean, yeah, but I have enough workplace autonomy to be able to say "I'm just not going to do that" and have it mean something.
Luu offers a technical solution to this problem: don't tell the LLM what it's going to be tested against, what he calls a "holdback benchmark." This is definitely a good idea, but I think its application is somewhat limited. But it's also not like I have a better solution in hand.
But what is resonant for me is that the human connection becomes that much more important. If I can't trust the computer to tell me that things are working, then I need to trust the customer even more. And even if my customers aren't buying units, we can discuss pain points and whether or not they think my team's solutions are helping. If you go back to the fundamentals of Agile software development, you see that software is not a product made by a producer and sold to a consumer, but rather it is a collaboration between them.
And as the LLM-drenched software ecosystem becomes harder and harder to parse programmatically—or reliably—then the human element is more important than ever.
]{p
Comments