It seems a lot of people still don’t realize that it can’t do analysis; it just predicts things that can look a lot like analysis and sometimes it even predicts the correct conclusion. But it’s training data will include many cases where a question is posed and responses jump right to the answer without showing any of the underlying work that got there.
So the training isn’t “user a asks question, user b considers factors x, y, and z, including how y and z interact complexly with each other, as well as how a subtle complexity of the question means the usual case for these doesn’t apply in the same way, and posts this response”, which is what an actual intelligent being would need to get from the question to the answer like an expert would.
The training is “user a asks question, user b posts response”. That training is more like how to train someone to pretend to be an expert because all the information is just the question/response pair (with no reasoning that wasn’t included in the response), and the tone or language used to convey the response.
That’s why they always sound so confident, because they are predicting what a confident response would look like, but the actual content of the response is based purely on correlations between the words used in the question and whatever words followed variations of those words used in other questions or stories or wherever the training data came from.
Though even if everyone wrote out their full line of thought for every response, it still wouldn’t result in actual intelligence because it’s still just correlating words, not applying logic and analysis to them. That’s all LLMs do. Hallucinations are fundamental to how they work.
You might have some luck turning their AI against them, as I’ve found it’s not very hard to get it to talk about LLM weaknesses (though it might be from my own verbose communication style and it could backfire if the AI is stubborn, though I find anthropic’s models aren’t very stubborn).
You could even get it to talk about the general usefulness of inexperienced users getting claude to diagnose or debug issues they themselves have no idea how to fix.
Not because LLMs have anything particularly profound to say about the topic but because those users are basing their “contribution” on the credibility of LLMs, so they might realize how dumb that is if an LLM tells them.
It seems a lot of people still don’t realize that it can’t do analysis; it just predicts things that can look a lot like analysis and sometimes it even predicts the correct conclusion. But it’s training data will include many cases where a question is posed and responses jump right to the answer without showing any of the underlying work that got there.
So the training isn’t “user a asks question, user b considers factors x, y, and z, including how y and z interact complexly with each other, as well as how a subtle complexity of the question means the usual case for these doesn’t apply in the same way, and posts this response”, which is what an actual intelligent being would need to get from the question to the answer like an expert would.
The training is “user a asks question, user b posts response”. That training is more like how to train someone to pretend to be an expert because all the information is just the question/response pair (with no reasoning that wasn’t included in the response), and the tone or language used to convey the response.
That’s why they always sound so confident, because they are predicting what a confident response would look like, but the actual content of the response is based purely on correlations between the words used in the question and whatever words followed variations of those words used in other questions or stories or wherever the training data came from.
Though even if everyone wrote out their full line of thought for every response, it still wouldn’t result in actual intelligence because it’s still just correlating words, not applying logic and analysis to them. That’s all LLMs do. Hallucinations are fundamental to how they work.
Yep. The worst thing is when the customer literally copies the log we’re generating on remote desktop, and throws it into Claude in front of us.
Like either they don’t trust us, or they’re trying to train it to clone our products.
It’s at the point that I’m tempted to require access where they can’t see what we’re doing exclusively when we capture the verbose logs
You might have some luck turning their AI against them, as I’ve found it’s not very hard to get it to talk about LLM weaknesses (though it might be from my own verbose communication style and it could backfire if the AI is stubborn, though I find anthropic’s models aren’t very stubborn).
You could even get it to talk about the general usefulness of inexperienced users getting claude to diagnose or debug issues they themselves have no idea how to fix.
Not because LLMs have anything particularly profound to say about the topic but because those users are basing their “contribution” on the credibility of LLMs, so they might realize how dumb that is if an LLM tells them.