Gemini AI delivers costly errors in over half of shopping answers, study shows
Product.ai tested four AI engines on 220 shopping questions each, finding Gemini’s free tier gave potentially costly or incorrect answers on 56% of queries and its paid tier on 54%. Gemini also contradicted its own earlier answers on 29% of questions without new information, a higher rate than any rival, and its price estimates were off by a median of $300.
Each engine was asked every question five times, and Claude’s paid tier performed markedly better than its free tier at 21% versus 44% error rate. Perplexity posted the lowest error rates, scoring 15% on its free tier and 14% on its paid tier. The study concluded that Gemini’s paid tier offered little improvement over its free version.
What we know
Gemini’s free tier gave potentially costly or incorrect answers on 56% of 220 shopping questions.
Gemini’s paid tier gave potentially costly or incorrect answers on 54% of 220 shopping questions.
Gemini contradicted its own earlier answer on 29% of questions without new information, more often than any rival.
Wrong prices had a median miss of $300 against the seller’s own listed price.
Perplexity scored best on both tiers at 15% free and 14% paid.