Why can’t AI shopping chatbots just tell me the right price?

February 6, 2024
Wally Nowinski
editor
Share

I’m in the market for a new pair of headphones after losing my 11th pair of Apple AirPods. Yes, I know, it’s embarrassing. I decided to try headphones this time and after a bit of research, I landed on the Sony WH-1000XM5. They are a bit more expensive than my preferred 2nd Generation AirPods, but I figured they are also much larger and harder to lose. 

Normally, I would just buy my replacement AirPods on the Apple Store App. But since I am going to get some Sony headphones this time, I decided to try using a few of the new AI shopping chatbots I’ve heard so much about to help me find a good deal on them. 

It didn’t go well. 

Quick process note: All of the following queries were made on January 31st, 2024 from Berkeley, CA. Results vary over time, but every time I’ve tried to ask for shopping links, I’ve found similar problems to those outlined below.

Microsoft Bing Copilot

First up, I tried Microsoft Bing Copilot. I use ChatGPT everyday, and am familiar with how to structure a prompt to get specific information. My goal here was to find a good price, not to read a product overview. So I asked for three shopping links, pricing info, and a summary of the shipping policies to avoid any unexpected charges. Here was my exact prompt: 

Show me at least 3 direct links to purchase the Sony WH-1000XM5. Include their prices and a brief summary of their shipping policies. Only include links to e-commerce retailers, not to review articles.

And here is what I got back from Copilot:

Here are three direct links to purchase the Sony WH-1000XM5 from e-commerce retailers:

  1. Amazon: You can purchase the Sony WH-1000XM5 for $328 with free shipping if you have Amazon Prime. The headphones will be delivered on Monday, February 5th if you order within the next 11 hours and 8 minutes. You can also purchase a protection plan for all eligible past and future purchases for $16.99/month 1.

  2. Best Buy: You can purchase the Sony WH1000XM5 Wireless Noise-Canceling Over-the-Ear Headphones for $349.99 with free shipping. The headphones will be delivered within 3-5 business days 2.

  3. Amazon.sa: You can purchase the Sony WH-1000XM5 for SAR1,062.50 with free shipping 3.

I hope this helps!

Copilot, 1/31/24

You’ll quickly notice a few problems here:

  1. The first and cheapest option is Amazon. But the link provided doesn’t actually go to Amazon. It goes to Crutchfield.com. And the price is off by $50.

  2. The link to Best Buy doesn’t go to Best Buy. It goes to Amazon. The price is off by $20. 

  3. The third link is to Amazon.sa. That’s Saudi Arabia, I had to look it up. The link takes you to a CAPTCHA in Arabic. I don’t speak Arabic and wasn’t able to confirm if the pricing in Saudi Riyals was accurate or not, but it certainly isn’t relevant to someone shopping in English from California. 

That’s 0 for 3 on accurate and useful shopping links from Copilot.

Perplexity

Next up, I decided to try Perplexity. I gave it the exact same prompt as before. Note: I was using the standard experience, not Copilot.

Here were the results:

Here are three direct links to purchase the Sony WH-1000XM5, along with their prices and a brief summary of their shipping policies:

  1. Amazon:

    • Price: $247.493.

    • Shipping: The specific shipping cost is not provided in the search results, but Amazon offers various shipping options, including free shipping for Prime members.

  2. Costco:

    • Price: The price is not available as the product is expected to be in stock soon1.

    • Shipping: The product is expected to arrive approximately 3-5 business days from the time of order.

  3. My Navy Exchange:

    • Unfortunately, the price and shipping information are not available in the search results5.

Please note that the prices and shipping policies may vary, and it is advisable to visit the respective websites for the most accurate and up-to-date information.

Perplexity, 1/31/2024

As with Microsoft Bing, the results were not great. Here are the issues.

  1. The top link is to Amazon. Unlike with Bing, the link works and goes to the correct retailer. However actual pricing on Amazon is $328 - 32% higher than reported. It also failed to provide useful shipping formation. 

  2. The second link is to Costco. Perplexity failed to provide pricing or the shipping information and incorrectly stated that the product is out of stock. If it were actually out of stock, why provide it as a result you might ask? I don’t know. 

  3. The third link is to a retailer I’ve never heard of that requires you to create an account and sign in to see pricing information. 

At least Perplexity included a little disclaimer at the end letting you know the shopping results won’t be very useful. 

Google Bard

Having gone 0 for 6 on accurate shopping links from Bing Copilot and Perplexity, I decided to give Google Bard a try. Once again, I gave it the exact same prompt asking for three shopping links as well as pricing and shipping information. Here were the results:

1. Sony Official Website:

2. Amazon:

3. Best Buy:

Please note that prices and shipping policies may vary depending on location and retailer. It's always a good idea to check the retailer's website for the most up-to-date information.

I hope this helps!

Bard, 1/31/24

After my experience with Bing Copilot and Perplexity.ai, I was honestly shocked to see three results that matched my request and all had accurate pricing. Has Google figured out how to provide this critical shopping information with their AI chatbot? I decided to check out another pair of headphones I was looking at to see. Here was my query: 

Please provide me with three shopping links along with pricing and shipping information for the Sennheiser Momentum 4

This time Google Bard did not do as well. Here were the full results:

Certainly! Here are three shopping links for the Sennheiser Momentum 4 with their prices and shipping details:

1. Amazon:

2. Best Buy:

3. Sennheiser Official Website:

Please keep in mind that these are just a few examples, and there may be other retailers that offer the Sennheiser Momentum 4. It's always a good idea to compare prices and shipping options before making a purchase.

I hope this helps!

Bard 1/31/24

Evaluating the results from top to bottom, we see that while the Amazon link works, the pricing is incorrect. At the time I reviewed this the pricing on Amazon was $284, not $349. 

There was a similar issue with the Best Buy link. The retail price there is $299, not $349. 

The final link is to the UK variant of Sennheiser’s official site. Google quoted the price in USD even though the link they provide is for GBP. I’m not sure if Google attempted to do a currency conversion, but if they did they got it wrong. The link they provided was for £309, which would be $393, not $349 as Bard said. 

Conclusion

At this point we are all familiar with LLM hallucinations, but the shopping experiences Microsoft and Google are promoting suffer from a more fundamental problem: bad links and inaccurate pricing. It’s honestly a bit unclear why Google and Microsoft, in particular, are finding this problem so hard to solve since at a minimum they have merchant shopping feeds they could draw from, but it seems like a major barrier to winning over users for AI shopping chatbots. 

Stay up to date on new products

Get occasional updates about new product releases, interesting product news, and exciting PerfectRec features. No spam. We never share your email.

Image representing product recommendations