Customers want more AI, not better AI

We’ve said many times that quality is our North Star at RunLLM — we want to make sure that we’re delivering the highest possible quality answers to the questions that our end users are asking. We’ve also said many times that evaluating AI tools is a bit of a pain right now because no one actually know what better is or how to measure it. Most commonly, customers rely on vibes-based evaluations — trying out a few questions and seeing if the answers generally meet their expectations or not.

The consequence of vibes-based evals is that when customers are comparing two AI-powered solutions that provide similar functionality, telling the difference might be difficult. There are many reasons why: The solutions might not be that different, the cases where they are different might be difficult to identify, or the customer might not be willing to invest the time into properly comparing them. Whatever the case may be, the end result is that you, as a product builder, have to find a way to convince the customer to use your product.

In this context, relying on better AI is difficult. It’s not that quality doesn’t matter — we believe strongly that AI quality matters — but you’re going to have a tough time convincing prospects that it does. This is especially difficult when relying on vibes-based evals because the handful of questions that a prospect chooses to evaluate on may skew their evaluation against you, even if your solution is better on average.

While there’s plenty of greenfield (customers that don’t yet have a solution) in the market today, this challenge becomes even more difficult when you’re selling into a customer that has an existing solution. Whether real or imagined, the time & emotional investment that a customer has in the existing solution is going to be difficult to overcome if you don’t have a more compelling argument than, “We’re better on average.”

All this is to say that better AI is going to be increasingly difficult to rely on as the market matures. We say this with heavy hearts, but we’re pretty convinced it’s true. And if better AI isn’t going to work, that means we’re going to instead have to rely on selling more AI. So what does that mean?

---

We’ll start by telling you what it doesn’t mean: Building shoddy features and claiming that you’re doing AI for the sake of it. We know none of you would ever do that, of course! But we’ve seen plenty of it, and this blog post is by no means a call for more AI-washing of features. Claiming that every feature you’ve built is now AI is going to annoy your customers (and us).

However, thoughtfully built new features that aren’t AI-washed will give you a huge advantage. There are a few reasons why.

First, most AI features are still new to most customers. In our little AI bubble, we’re all looking at the latest products and stealing cool ideas from adjacent products, but the rest of the world isn’t as plugged in as we are. If you can be the first person to show your customers a feature that they haven’t seen before, they’re going to be pretty excited about it — especially if it’s something that your competition hasn’t yet implemented.

Second, the reality is that customer expectations for AI are tempered by all the snake oil that’s out on the market today. Again, this isn’t an excuse to go build shoddy features or ship the first crazy idea that you have. But if you ship something that’s aligned with customer needs, even if you haven’t yet nailed the details, it’s likely that that’s going to get you credits with them. If done well, this also means that you can start putting together a more compelling feature matrix when customers are evaluating multiple solutions.

Finally, as we’ve discussed elsewhere, this is going to be valuable for you from a product development perspective as well. AI product best practices haven’t been fully established yet, and the more shots on goal you get for getting customer feedback on new ideas, the better your instincts will get — even if some of your attempts are misses.

---

All that said, what do we actually mean by “new features?” For us, that means focusing on the problems that you can solve with AI that are genuinely novel — the work that AI enables you to do that wasn’t possible before. It’s easy to focus on what the technology is capable of doing today, but to be blunt, we have more than enough chatbots in the world already. If you focus instead on serving a full job function with your AI product, you’ll find that there’s a lot more you can do with the technology than just chat — analyzing data, surfacing insights, generating new content, and so on.

It’s also worth saying that this isn’t going to be true forever. This is an observation about the market as it is today, not a statement about the future of the AI market. As the market matures and buyers have more established buying processes, it’s likely that this argument will be flipped on its head. In other words, in 2-3 years time, differentiation from having more features might well be competed away, and quality might once again the thing that we need to be spending all of our time on — especially if we get better & more nuanced evaluations! But for now, having a breadth of good features is more valuable.

We say all this for the sake of argument, but this is going to be a difficult needle to thread. Again, it’s still a net negative to spam silly features, and you’re going to want to think through things carefully before shipping them. But of course, AI markets are crowded, so time of is of the essence. In other words, you’re in a bit of a catch-22: You want to ship cool features as fast as possible, but you don’t want to lose credibility by shipping nonsense. You’ll have to figure out the right balance for yourself: At RunLLM, we generally tend to be conservative and wait to get strong signal before shipping, but it’s something that we think we’ll need to loosen the constraints on.

Wherever you fall on this spectrum, the call to action is to ship more new stuff — it’ll pay dividends very quickly.