ChatGPT, Bard, GPT-4, and the like are often pitched as ways to retrieve information. The problem is they'll "retrieve" whatever you ask for, whether or not it exists.
Tumblr user @indigofoxpaws sent me a few screenshots where they'd asked ChatGPT for an explanation of the nonexistent "Linoleum harvest" Tumblr meme,
It’s only a “shortcoming” if you aren’t aware of how these LLMs function and are using it for something it’s not good at (in this case information retrieval). If instead you want it to be making stuff up, what was previously an undesirable hallucination becomes desirable creativity.
This also helps illustrate the flaws in the “they’re just plagarism machines” argument. LLMs come up with stuff that definitely wasn’t in their training data.
I didn’t mean to argue against the usefulness of LLMs entirely, they absolutely have their place. I was moreso referring to how everyone and their dog are making AI assistants for tasks that need accurate data without addressing how easy it is for them to present you bad data with total confidence.