Can you know if coding agents are worth the cost?
How to think about managing the economics of software production
A recent theme in our conversations with VPs of Engineering has been a concern about (and perhaps a mild skepticism about) the cost of tokens being sunk into coding agents. A VP of a team of ~100 engineers said to us, “We’re spending about five figures a month on coding agents. I think it’s making the team more productive overall. I’m not sure, but I think.”
As concerns about coding agent overuse, budget overruns, and the ultimate impact on productivity mount, we’ve found ourselves wondering about the economics of coding agents. Coding agents can generate code faster than engineers working alone, but faster production is not the same as better economics. The fact that you can generate more code doesn’t mean that you should. To be clear, this isn’t a question of writing code by hand – all code will be generated by agents – but what tasks you pick, what models you use, and how you manage your agents.
To illustrate the point, let’s take a somewhat obvious example. A group of four can drive from SF to LA in 6 hours and pay, perhaps $100 total for gasoline. That same group of people can fly from SF to LA in 1 hour and pay $200 each for airfare. Both are valid choices and depend on the urgency of the trip, what they’ll do once they arrive, and their price sensitivity.
Engineering leaders are grappling with the fact that the economics of producing software, managing engineering time, and ultimately building products have changed dramatically. If you get your parameters wrong, you’ll be paying a lot of money to build the wrong thing very fast.
The economics of software production
If we rewind to the halcyon days of 2022, the way most engineering teams budgeted was by having a set headcount and allocating those cycles to whatever the highest priority projects were. Assuming the team delivered as expected, you could roughly say that you were spending $N a quarter to produce the software you shipped that quarter.
Today, you can spend something like $2-3N to create the same software in, perhaps, a week. While it’s hard to quantify, this is definitely a more efficient process – a cost increase of 2-3x and time improvement of about 10x is some pretty easy math for any project that you can look at.
For a real world example, consider the recent port of Bun from Zig to Rust. The blog post detailing the post says that the port took about 11 days and cost $165k at API pricing for work that would have taken a “small team of engineers a full year.”
This introduces a totally foreign dimension to software budgeting. To be clear, this is a huge net improvement. A small team, say engineers, working for a year will easily cost you $600k. The fact that you can do this task for ~25% the cost doesn’t mean that you should do it. If the task was done with the same number of tokens on Sonnet but required 2x as much human input, that would cost ~$50k of tokens combined with an extra 10 days of salary. The reality probably isn’t that simple, but it’s illustrative of the new dimension of cost in producing software.
Until recently, we haven’t really had to ask this question about software production, but coding agents have completely changed the game. Engineers and engineering teams now have to grapple with questions about which model they should use, what level of speed and accuracy they need, traded off against a willingness to use a cheaper model that requires more guidance.
Ultimately, the economics of software production are going to increasingly have to be tied to the value of what’s being produced, not just the time it takes to produce it. Which brings us to the humans using the models…
The economics of engineering time
Measuring engineering productivity has historically been extremely hard. As an industry, we long ago ruled out silly metrics like lines of code, commits made, or tasks closed. In their place, we’ve mostly just agreed that the good engineers are the ones that everyone else on the team agrees are the good ones. DORA metrics can point at the function of an engineering team but don’t tell us much about individual engineering performance.
Recently, engineering leaders have started measuring engineers on token usage. This is unequivocally a bad idea, but it highlighted a key concern around how you think about whether engineers are using their time well and delivering on the right things. As is now becoming common wisdom, what you build matters dramatically more in a world where you could theoretically build anything.
In practice, the 2x2 between good vs. bad productivity and low vs. high token use has 4 valid quadrants. In other words, token use does not correlate positively or negatively with engineering productivity. Just because an engineer burns through a lot of tokens, that doesn’t tell you very much about what’s getting done – they might be mismanaging context windows, regularly patching issues created by poor prompting, or undoing and redoing a lot of work because of poor prompting. Just recently, an engineer on our team accidentally burned through ~$300 tokens on Claude very quickly by mistakenly pinning Fable. That engineer is generally very good but obviously didn’t suddenly get 10x better than everyone on that day. Simultaneously, we have plenty of examples of engineers who are very judicious about token use because they plan carefully, give precise instructions, and invoke the agent when they’re confident in which direction they want to head.
There’s no easy answer here. Measuring engineering productivity has genuinely gotten dramatically harder in the past 6 months because we’ve introduced a whole new dimension that simply didn’t exist before. You shouldn’t crown an engineer because they burned through a bunch of tokens, and you shouldn’t fire them either.
Engineering decisions & product decisions
The technology is new, and these aren’t easy decisions to make. In the stylized example above, we suggested that doing the same task with Sonnet instead of Fable would have been possible with 2x as much human input – that might be true, or Sonnet might have very quickly gone off the rails and generated totally useless results. Perhaps $165k + 10 days of time is the cheapest possible implementation with the current state of the art. It’s impossible to know otherwise until you actually try. Unfortunately, trying is expensive.
While token costs dropped rapidly in 2023-24, the rate of change has decreased over the last year – and the rate of token consumption has skyrocketed. Managing this tradeoff has gotten harder as we’re now all balancing model choice with effort level and task scope. Given that it’s new, every engineer is going to find this challenging, but this will be a skill that engineering teams develop over time with practice.
In the meantime, you need to keep a close eye on your token use.