A word generator will generate anything you tell it to.

I’ve “convinced” ChatGPT that it was both sentient and conscious in the span of about 10min, despite it having explicit checks in place to avoid those kinds of statements. It doesn’t mean I was correct, just that it’s a “dumb” computer that has no choice but to ultimately follow the logic presented in syllogisms.

These things don’'t know what they’re saying; they’re just putting coherent sentences together based on whatever algorithm guides that process. It’s not intelligent in that it is doing something novel, it’s just a decent facsimile to human information processing. It has no mechanism to determine the reasonability or consequences of what it generates.