Why can’t AI shopping chatbots just tell me the right price?

I’m in the market for a new pair of headphones after losing my 11th pair of Apple AirPods. Yes, I know, it’s embarrassing. I decided to try headphones this time and after a bit of research, I landed on the Sony WH-1000XM5. They are a bit more expensive than my preferred 2nd Generation AirPods, but I figured they are also much larger and harder to lose. 

Normally, I would just buy my replacement AirPods on the Apple Store App. But since I am going to get some Sony headphones this time, I decided to try using a few of the new AI shopping chatbots I’ve heard so much about to help me find a good deal on them. 

It didn’t go well. 

Quick process note: All of the following queries were made on January 31st, 2024 from Berkeley, CA. Results vary over time, but every time I’ve tried to ask for shopping links, I’ve found similar problems to those outlined below.

Microsoft Bing Copilot

First up, I tried Microsoft Bing Copilot. I use ChatGPT everyday, and am familiar with how to structure a prompt to get specific information. My goal here was to find a good price, not to read a product overview. So I asked for three shopping links, pricing info, and a summary of the shipping policies to avoid any unexpected charges. Here was my exact prompt: 

And here is what I got back from Copilot:

You’ll quickly notice a few problems here:

1. The first and cheapest option is Amazon. But the link provided doesn’t actually go to Amazon. It goes to Crutchfield.com. And the price is off by $50.

2. The link to Best Buy doesn’t go to Best Buy. It goes to Amazon. The price is off by $20. 

3. The third link is to Amazon.sa. That’s Saudi Arabia, I had to look it up. The link takes you to a CAPTCHA in Arabic. I don’t speak Arabic and wasn’t able to confirm if the pricing in Saudi Riyals was accurate or not, but it certainly isn’t relevant to someone shopping in English from California. 

That’s 0 for 3 on accurate and useful shopping links from Copilot.

Perplexity

Next up, I decided to try Perplexity. I gave it the exact same prompt as before. Note: I was using the standard experience, not Copilot.

Here were the results:

As with Microsoft Bing, the results were not great. Here are the issues.

1. The top link is to Amazon. Unlike with Bing, the link works and goes to the correct retailer. However actual pricing on Amazon is $328 - 32% higher than reported. It also failed to provide useful shipping formation. 

2. The second link is to Costco. Perplexity failed to provide pricing or the shipping information and incorrectly stated that the product is out of stock. If it were actually out of stock, why provide it as a result you might ask? I don’t know. 

3. The third link is to a retailer I’ve never heard of that requires you to create an account and sign in to see pricing information. 

At least Perplexity included a little disclaimer at the end letting you know the shopping results won’t be very useful. 

Google Bard

Having gone 0 for 6 on accurate shopping links from Bing Copilot and Perplexity, I decided to give Google Bard a try. Once again, I gave it the exact same prompt asking for three shopping links as well as pricing and shipping information. Here were the results:

After my experience with Bing Copilot and Perplexity.ai, I was honestly shocked to see three results that matched my request and all had accurate pricing. Has Google figured out how to provide this critical shopping information with their AI chatbot? I decided to check out another pair of headphones I was looking at to see. Here was my query: 

This time Google Bard did not do as well. Here were the full results:

Evaluating the results from top to bottom, we see that while the Amazon link works, the pricing is incorrect. At the time I reviewed this the pricing on Amazon was $284, not $349. 

There was a similar issue with the Best Buy link. The retail price there is $299, not $349. 

The final link is to the UK variant of Sennheiser’s official site. Google quoted the price in USD even though the link they provide is for GBP. I’m not sure if Google attempted to do a currency conversion, but if they did they got it wrong. The link they provided was for £309, which would be $393, not $349 as Bard said. 

Conclusion

At this point we are all familiar with LLM hallucinations, but the shopping experiences Microsoft and Google are promoting suffer from a more fundamental problem: bad links and inaccurate pricing. It’s honestly a bit unclear why Google and Microsoft, in particular, are finding this problem so hard to solve since at a minimum they have merchant shopping feeds they could draw from, but it seems like a major barrier to winning over users for AI shopping chatbots.