I was recently reading Ben Aaronvitch’s ‘Rivers of London’ and I was struggling to follow the plot. So I engaged in discussions with both ChatGPT and Gemini and both really struggled to not make lots of mistakes. Both AIs would regularly give inaccurate info about major plot points, and when I questioned them about those issues, they would apologize profusely and then try to continue on. I basically had to coach them through to get them to remember important points of the plot! They also struggled to not reveal spoilers past a certain chapter. It was interesting to see the AIs repeatedly fail.
Did the AI “fail,” or did you use AI in a suboptimal way?
If you had started the conversation by feeding in the actual text of the book, you would have very different results, because you would be grounding on context rather than asking the model to “figure it out somehow” based on training data.
Not sure what you were expecting, using AI is mostly about context management, and if you don’t set up the context correctly you’re asking for results similar to this.
It appeared it had access to the entire book - it linked me to a website that somehow had the text. I was able to coax better responses out of it when I corrected its mistakes about plot etc.
Unless you provided it, the context window did not have the context. “Appeared to” is also not a valid way to evaluate technology.