So, before I start I’d like to say that I believe Bing/ChatGPT integration (Prometheus model) will be important and in some way change how we search. But from the basic example results offered up by Bing, lets say I am less than impressed.
Whilst Google has been mocked for their $100 billion AI mistake, Bing’s sample offering is also very underwhelming and here’s my breakdown of why I’m rather disappointed.
1: Is Bing Chat better than basic organic SERPs? Not really.
“Plan me a workout for my arms and abs with no situps and no gym equipment. It should only take 30 minutes”
I can read Bings Chat reply, which lists PopSugar in its first citation, alternatively, I could just read – or watch! – the article/video that is pretty much an exact answer to the question listed by its side.
This kind of search screams out for video content, explaining and showing you the exercises. That’s a much better result than a text write-up. In my opinion, with the search examples provided by Bing Chat, you may as well choose the first listing in the organic results. They are perfectly acceptable, or at least as useful, as the Bing Chat result. So what has Bing really added? Not a lot.
AI chat can help with a lot of search queries but is this really one of them? It seems strange to me that Microsoft would choose an example like this to demonstrate an improvement in search when I struggle to look beyond how much better the first two organic results look in comparison.
2: Does Bing Chat give quality citations? Not always, in my opinion.
Let’s take this sample question given to us by Bing.
“Arts and crafts ideas, with instructions for a toddler using only cardboard boxes, plastic bottles, paper and string”
What joys will Bing offer up that will get my imaginary toddler in the mood for some crafting? Here is a part of its response…
“You can make a plastic bottle bird feeder with a 2-liter bottle, some chopsticks, twine, a pin and birdseed. You can find the instructions here.”
Inspiring…
Now if Microsoft are concerned that ‘40% of queries result in someone clicking back’ then I would suggest that someone clicking on their citations may end up pushing that figure up further.
Obviously, it comes down to personal opinion but the citations listed, especially in this example, feel like the sort of websites that I’d click on and then make a hasty retreat. Does quality feature in this feature or does that come later? I imagine the latter.
You can tell though that Bing Chat is trying and it’s certainly not a bad result, even if I am mocking it for its lacklustre appeal. Bing could have just linked to an article of ideas but at least, in this case, it is sending you to an article that lists out the necessary components along with a step-by-step guide. I’d love to see video content in there but AI right now is very focused on language. Whilst there are video transcripts, that I assume it could access, I suspect this is also something for the future.
3: Does Bing Chat citations lead you to deeper answers? Not always.
“I am planning a trip for our anniversary in September. What are some places we can go that are within a 3 hour flight from London Heathrow?”
Go me. I’m off on holiday and Bing Chat tells me that Berlin is the place to be for culture and nightlife. Now tell me more, lets visit the citation and get a full rundown of the Berlin holiday experience…
Well, that wasn’t what I was expecting. That’s a whole 22 words longer than Bings description and in my opinion not really worth the click. Perhaps here I am being a little harsh. Citations aren’t necessarily always going to expand on what Bing has pinched from the site, after all, it is just giving us an example of where it got the information from. But let’s go back to point one, is this better than simply clicking on the first result for your search? And are you going to be clicking the back button that Microsoft is so desperate to avoid?
The idea is obviously for you to continue your journey in this chat, by asking ‘tell me more about the nightlife in Berlin’, something we can’t do right now. But I am not overly enamoured with its sources thus far and may just find a trusted site to read instead.
ChatGPT blew me away when I was testing it, but right now I’m questioning its value in answering my search queries (or rather its own search queries that is it using to promote the feature).
4: How clever is Bing Chat in sourcing its material/citations?
This is obviously an area of interest for SEO. Bings Chat is going to answer a lot of questions, and if you want to get the traffic you’ll clearly want to be listed as a citation or within the ‘Learn More’ results. Achieving that goal will likely become a new focus for SEO, similar to when featured snippets first appeared in SERPs.
My suspicion is that Bings chat does a fair bit of work to back up its response. In the initial search results you won’t see all of the ‘learn more’ websites, at least on page one. However, if you rephrase the search to be more focused on the examples Bing Chat lists then they all begin to appear.
Take for example the Bird Feeder suggestion. Once the chat response has decided to give you this as an example, the next logical search would be something along the lines of ‘kids art craft ideas how to make a plastic bottle bird feeder’. And when you make that search you surface not only the citation source but also several other sites featured within the ‘learn more’ section such as Instructables and PBS.
Why this matters
At this point, maybe Bing Chat is earning some points and when full functionality is rolled out to users perhaps we will find ourselves using the chat feature to refine our search in this way. However, as mentioned earlier, I am not convinced that the search examples provided by Bing really need a chat interface. It feels like we simply need better quality content in the top organic positions.
And that is probably what I have taken away from all this. If you are continually clicking back from search results because you didn’t find what you needed then that’s not necessarily you making a bad search, rather search engines doing a bad job of serving you quality content. A chat bot that simply curates the same content and links to it feels less than useful.
I remain very excited to see how AI will change our search behaviour, but whilst this rollout hasn’t cost Microsoft billions in terms of bad PR it has been remarkably underwhelming in my opinion. I’m sure once full functionality becomes available it will have a lot of valuable features but I remain unconvinced that it is about to change my search habits.