So you want to buy your first AI product

A totally unbiased guide to evaluating AI applications

As you've probably heard a million times by now, 2025 is the year the agent or the year of AI adoption or the year of continued vague hype about AI. Whichever phrase you prefer, there's a lot of folks who are going to be buying their first AI-enabled product this year. We've worked with a lot of teams buying AI products for the first time at RunLLM, and we've seen a lot of successes and failures. As a completely selfless act, we thought it would be helpful to write a guide to buying your first AI product. 😉

We've said this many times already, but it's worth starting with a disclaimer: It's early, and every AI product is going to be at a different level of maturity. We encourage you to approach your buying process with an open mind, but you also shouldn't be shy about deciding that a product isn't mature enough for you to use yet. For example, see our recent analysis of our experience with Devin.

With that said, let's jump into what you should be thinking about as you buy your first AI product.

Source: DALL-E 3.

Know what you're looking for. It sounds silly, but this is the most common failure case we see with prospects for RunLLM. Unlike the market for well-established products like databases or CRMs, there are very few guides and features matrices that you can follow to figure out how to buy an AI product. That puts a burden on the team building the product to help you understand what's possible and what's not, but it's just as important for you to have a clear sense of what your priorities are. In the absence of clear priorities, it's very difficult to know whether the product meets your needs or not. That means that you’re either going to buy a product that isn’t up to snuff or that you’ll have unrealistic expectations that cause you to miss out on real value. Either way, you're simply going to waste both your time and the vendor's time.

Understand the economics of AI products. SaaS products have historically been priced based on access — usually, seat-based pricing. Good AI products should be priced based on the work that they're doing, not on access. (Pricing on access is at a minimum a yellow flag!) You may be initially surprised at the cost of some AI products (e.g., many AI SDRs have entry-level pricing at $3k/month), but you should be thinking about this in the context of the time it unlocks for your team. If a good AI SDRs saves a half of the average SDR's time (which, to be fair, is a high bar!), then the cost probably worth it.

A common objection we hear is that there’s already someone on staff filling that role. Just because you're paying a person to do something currently, that doesn't mean it's the best use of their time — they’re likely capable of stepping up their responsibilities if you unlock their time. If an AI product can do it just as well, then that person is freed up to do more important things.

Don't buy the first thing you see. AI is popular, so there's unfortunately a lot of snake oil out there, and we've seen plenty of companies rush to buy a $10/month chatbot that provides mediocre answers — all for the sake of saying that they’ve bought an AI product. There’s potentially a time and place for this kind of product, but you have to be aware of the risks. The standard for an internal tool is likely lower — your team will be tolerant of mistakes. On the other hand, buying a lower quality tool for something that’s customer-facing is a silly choice. This doesn’t mean you necessarily need to run a formal process that evaluates six different tools, but you should certainly make sure that the product you're buying solves a business purpose. Otherwise, you'll be frustrated with the results and churn after a few months.

Have realistic expectations. As a slight caveat to the point above, make sure your expectations are in line with what AI can do today. We've had customers tell us that they expect RunLLM to be a full-fledged solutions architect. While we're bullish on the evolution of AI, we will be the first ones to tell you that we're nowhere near that level of maturity. We can still solve a real business problem — automating handling technical support tickets with high precision — but we're not quite ready to fill an entire job function. If you come in with that kind of expectation, you're going to be disappointed.

Know when to say 'no.' As we mentioned at the top, not every product is going to be ready for widespread use or even your specific use case. We tried Devin and found that it simply wasn’t ready for us to be paying to use it regularly. That's okay — it's an early market.

To give another brief example, we spent last fall evaluating a handful of AI SDRs, and we ended up saying no to all of them and are trying more traditional sales platforms. While the concept made sense, we found that we weren't able to get the cold outbound messaging to reflect our priorities. That doesn't mean it wouldn't have worked for another business, but we felt that for a technical product audience like RunLLM's, we needed more specificity and technical depth in the content. The emails we got removed technical depth and sounded far too generic.

Work with a team that wants to iterate. If you hadn't gotten the message by now, it's early in the AI market. There's a lot of teams building great products that might not have every integration or knob that you're looking for yet. Beyond the specifics of the product, we believe you should prioritize working with teams that show a willingness to adapt and iterate — more than anything, they should be hungry for feedback and eager to execute. This is more likely to come from smaller companies than more traditional enterprises. Startups will be able to build the integrations that you're looking for, and more importantly, they'll be receptive to feedback and able to keep up with the latest technology. All of that is critical in building a product that's going to continue to mature over the next few years.

---

Much of this boils down to generic advice about buying an product in an early market, but it's more important than ever with AI. If you pick the wrong product, you'll find yourself frustrated with low quality results and potentially bearish on AI in general. If you make that mistake, find a way to get out early. When you find the right products, you'll find that you can make your team more productive and get boring drudgery out of the way — that's the holy grail!