Why a Winning A/B Test Isn't Enough to Ship an AI Feature
I used to have a simple rule for shipping features. If an A/B test wins, you ship it. I've followed it for most of my career in consumer products and it rarely let me down.
Then came the AI era and working on building AI features and what do you know? it stopped holding up.
The example that works best to explain this is the AI photo feedback feature I shipped at Bumble. Where users upload their profile photos and get suggestions to make their photo set better for their dating profile. Building this taught me so much about the complexities of shipping AI in products. Our tests were well designed, our results were accurate, but they just didn't answer the question I needed answered, which was whether we should ship it!
It all started before the test even ran. Our research validated that people were open to AI helping them improve their profiles as long as it was not generated by the AI, so the demand looked obvious. Then after we launched, not many people used it. The research was not wrong. Saying you're open to AI and choosing to tap a button in the middle of your evening swiping session are just very different things. So what made the difference wasn't the model. It was where the feature showed up in the flow, and whether people understood what it was for. When users see AI they have questions and we need to be even more descriptive about what the feature will do and how it works than we would have otherwise to build trust and adoption.
Then we looked at the results. With a normal feature, everyone in a test arm sees the same thing. With a generative one, everyone sees something slightly different. Our top-line numbers looked fine, and then I saw a sample of feedback telling a person to smile more in a photo where they were already beaming. The average didn't show that. You only find it by reading the actual outputs, and then setting up evals so you can keep reading them once there are too many to check by hand. How much this matters naturally depends on what the feature does.
Then there's the bill. Most of us grew up on systems where scale makes things cheaper per user. Generative AI often runs the other way: every time someone uses the feature, you pay for a new generation, so the more it succeeds, the more it costs. We had a version that performed well and cost far more than we could justify. That's an uncomfortable meeting to sit in, because the dashboard is green and the answer is still no.
Naturally, we tried a cheaper route. We moved to cheaper models and mixed in some conventional machine learning. Costs went down, and so did the impact. That's when it really sank in for me that a winning result also belonged to that particular model, not to the idea itself. Now we were back to square one to find and test a version that would fit within budget and also test positively.
The last surprise had nothing to do with probability. With most features, a win in one country is a win everywhere. AI regulation doesn't work like that. It varies by market and keeps moving, and our feature stayed blocked in most countries until each one was cleared. You can only learn where you're allowed to test, so a result in one market doesn't tell you much about the next. Bringing legal and compliance in before any experimentation starts can be critical to ensure you are minimising a lot of re-work and planning scope and feasibility with the right stakeholders.
All this to say, none of this has put me off experimentation one bit. Randomised tests are still the most honest way I know to learn what a feature does to real people. I've just stopped treating a positive result as the finish line now! Before I scale an AI feature, I want to know a lot more, like whether the outputs hold up past the average, whether it pays off once every cost is counted, whether the win survives on a model we can afford, and where we're allowed to ship it.
"Did it win?" is still where I start. It's just not where I stop anymore.