I have had this happen pretty often. I think even the wording used in the “prompt” can also influence whether the chatbot hallucinates a solution.

I remember recently asking a third party’s mandatory chatbot support system (if it can’t help, it then forwards you to a human) whether I was able to change the mapping of some fields in an upload to them without the need for a support agent. It walked me through the process but it didn’t work. Eventually, after three failed chatbot modifications to the instructions, I was forwarded to a human that confirmed I was not, in fact, able to do it myself. Such a fucking waste of time.