The way generative AI works is by using things called “tokens”. Usually 1 word == 1 token, but compound words would be 2 tokens, punctuation would be a token, things like “-ed” or “-ing” could be tokens, etc.
When you give an AI a prompt, it breaks your response down into tokens. It then finds what tokens were statistically most likely to appear near that content and gives them as a response.
This has been the approach for a while, but the modern breakthroughs have come from layering AIs inside of each other. So in our example, the first AI would give an output. Then a second AI would take that output and apply some different rules to it - this second AI could have a different idea of what a “token” is, for example, or it could apply a different kind of statistical rule. This could be passed to a third AI, etc.
You can “train” these AI by looking at their output and telling it if it was good or bad. The AIs will adjust their internal statistical models accordingly, giving more weight to some things and less weight to others. Over time, they will tend towards giving results that the humans overseeing the AI say are “good”. This is very similar to how the human brain learns (and it was inspired by how humans learn).
Eventually, the results of all these AI get combined and given as an output. Depending on what the AIs were trained to do, this could be a sentence/response (ChatGPT) or it could be a collection of color values that humans see as a picture (DALL-E, Midjourney, etc.).
Because there are so many layers of processing, it’s hard to say “this word came from this source.” Everything the AI did came from a collection of experiences, and generally as long as the training data was sufficiently large you can’t really pinpoint “yeah it was inspired by this.” It’s like how when you think of a dog, you think of all the dogs you’ve experienced in your lifetime and settle on one idea of “dog” that’s a composite of all those dogs.
Interestingly, you can sometimes see some artifacts of this process where the AI learned the “wrong” thing. One example: if you asked an AI what 3 + 4 is, it knows from its experiences that statistically it should say “7”. Except people started doing things like asking for what “Geffers + HippoLady” was, and the bot would reply “13”, consistently.
It seemed there were these random tokens that the bot kept interpreting as numbers. Usually they were gibberish, but sometimes you could make out individual words being treated as 1 token despite being 2 separate words.
It turned out that if you googled these words, you’d get redirected to a subreddit - specifically /r/counting. The tokens were actually the usernames of people who contributed often to /r/counting. This is one way it was determined that the bot was training on Reddit’s data - because these usernames appeared near numbers a lot, the bot assumed they were numbers and treated those tokens accordingly.
Such a detailed response, thank you for that. It walked the line well between keeping it fairly simple but still detailed to understand it.
Because of the complexity and “mystery box” nature of ai for me, it’s hard not to just allow it in my mind to just to consider it as another form of intelligence. But people dismiss it as not intelligent because they have a far better understanding of how the AI has been trained and also in how it came to the results it has. This, VS humans where you’re like “oh I know he went to college in the medical field” but you don’t have as intimate an understanding how how thoughts, ideas and responses are formed because of that obscured source information. Also of course, far more complex with the other impacting factors like chemicals in the body, sleep, mood, experiences and perceptions.
I guess this is a bit of a run on, but it still makes me wonder if it’s just a case of creating enough obscured understanding that allows for consciousness to be as accepted. Not saying that ChatGPT is like a genuine consciousness, but more that it’s the underpinnings or beginnings for something like that. But this is said as someone with absolutely no training in the medical field as well as the artificial intelligence field, so yeah.
it’s hard not to just allow it in my mind to just to consider it as another form of intelligence.
If it makes you feel better, that’s probably a biological response that everyone has, to varying degrees. I heard the phrase “textual pareidolia”, meaning that if we see text that looks human enough, we’ll automatically put a human face on it and want to treat the author like an actual human. Even though it’s just a way of creating sentences that mimics human language, and has no form of “intelligence” whatsoever. It has no idea what it’s saying and does not understand the meaning of any of the words it’s producing. But our lizard brain is still fooled because it sounds good enough. It’s like seeing a face in the clouds or Jesus on burnt toast.
Even though I know what it’s doing, and can “break” it by making it sound very not-human without too much effort, it’s still hard for me not to end a chat session with “Thank you, have a nice day!”
Interesting - kind of weird how in the visual realm there’s the uncanny valley, but I suppose that would be explained by how significant and instinctual vision has played a role in human evolution to detect faces/weird faces etc
The way generative AI works is by using things called “tokens”. Usually 1 word == 1 token, but compound words would be 2 tokens, punctuation would be a token, things like “-ed” or “-ing” could be tokens, etc.
When you give an AI a prompt, it breaks your response down into tokens. It then finds what tokens were statistically most likely to appear near that content and gives them as a response.
This has been the approach for a while, but the modern breakthroughs have come from layering AIs inside of each other. So in our example, the first AI would give an output. Then a second AI would take that output and apply some different rules to it - this second AI could have a different idea of what a “token” is, for example, or it could apply a different kind of statistical rule. This could be passed to a third AI, etc.
You can “train” these AI by looking at their output and telling it if it was good or bad. The AIs will adjust their internal statistical models accordingly, giving more weight to some things and less weight to others. Over time, they will tend towards giving results that the humans overseeing the AI say are “good”. This is very similar to how the human brain learns (and it was inspired by how humans learn).
Eventually, the results of all these AI get combined and given as an output. Depending on what the AIs were trained to do, this could be a sentence/response (ChatGPT) or it could be a collection of color values that humans see as a picture (DALL-E, Midjourney, etc.).
Because there are so many layers of processing, it’s hard to say “this word came from this source.” Everything the AI did came from a collection of experiences, and generally as long as the training data was sufficiently large you can’t really pinpoint “yeah it was inspired by this.” It’s like how when you think of a dog, you think of all the dogs you’ve experienced in your lifetime and settle on one idea of “dog” that’s a composite of all those dogs.
Interestingly, you can sometimes see some artifacts of this process where the AI learned the “wrong” thing. One example: if you asked an AI what 3 + 4 is, it knows from its experiences that statistically it should say “7”. Except people started doing things like asking for what “Geffers + HippoLady” was, and the bot would reply “13”, consistently.
It seemed there were these random tokens that the bot kept interpreting as numbers. Usually they were gibberish, but sometimes you could make out individual words being treated as 1 token despite being 2 separate words.
It turned out that if you googled these words, you’d get redirected to a subreddit - specifically /r/counting. The tokens were actually the usernames of people who contributed often to /r/counting. This is one way it was determined that the bot was training on Reddit’s data - because these usernames appeared near numbers a lot, the bot assumed they were numbers and treated those tokens accordingly.
Such a detailed response, thank you for that. It walked the line well between keeping it fairly simple but still detailed to understand it.
Because of the complexity and “mystery box” nature of ai for me, it’s hard not to just allow it in my mind to just to consider it as another form of intelligence. But people dismiss it as not intelligent because they have a far better understanding of how the AI has been trained and also in how it came to the results it has. This, VS humans where you’re like “oh I know he went to college in the medical field” but you don’t have as intimate an understanding how how thoughts, ideas and responses are formed because of that obscured source information. Also of course, far more complex with the other impacting factors like chemicals in the body, sleep, mood, experiences and perceptions.
I guess this is a bit of a run on, but it still makes me wonder if it’s just a case of creating enough obscured understanding that allows for consciousness to be as accepted. Not saying that ChatGPT is like a genuine consciousness, but more that it’s the underpinnings or beginnings for something like that. But this is said as someone with absolutely no training in the medical field as well as the artificial intelligence field, so yeah.
Thanks again for your response.
If it makes you feel better, that’s probably a biological response that everyone has, to varying degrees. I heard the phrase “textual pareidolia”, meaning that if we see text that looks human enough, we’ll automatically put a human face on it and want to treat the author like an actual human. Even though it’s just a way of creating sentences that mimics human language, and has no form of “intelligence” whatsoever. It has no idea what it’s saying and does not understand the meaning of any of the words it’s producing. But our lizard brain is still fooled because it sounds good enough. It’s like seeing a face in the clouds or Jesus on burnt toast.
Even though I know what it’s doing, and can “break” it by making it sound very not-human without too much effort, it’s still hard for me not to end a chat session with “Thank you, have a nice day!”
Interesting - kind of weird how in the visual realm there’s the uncanny valley, but I suppose that would be explained by how significant and instinctual vision has played a role in human evolution to detect faces/weird faces etc
Throw in a bit of Cold Reading and it really feels better than it is, but honestly it’s a great tool if you understand the limitations.