ChatGPT, Bard, GPT-4, and the like are often pitched as ways to retrieve information. The problem is they'll "retrieve" whatever you ask for, whether or not it exists.
Tumblr user @indigofoxpaws sent me a few screenshots where they'd asked ChatGPT for an explanation of the nonexistent "Linoleum harvest" Tumblr meme,
An LLM never lies because it cannot lie.
Corollary: An LLM never tells the truth because it cannot tell the truth.
Final theorem: An LLM cannot because it do not be
So you’re saying they think it be like it is but it don’t?