We fed all of human knowledge into the system and now we can just decompress bit by bit. Issue is the moment ai is fed the ai stuff it’s like that meme jpeged ten times. It’s getting so bad you can not even understand it anymore.
AI is just a thought compression algorithm and can not come up with new ideas.
It’s just not, no matter how many times people say it. When it regurgitates data, that’s an artifact of the training process. It means that information was not properly encoded into the whole
And it most certainly is creative, as the eldrich mathematical construct it is, it can produce truly new concepts. More creative than humans, even - we very, very rarely have a brand new idea. Instead, we’re masters of lateral thinking - we take patterns we learn in one place and apply them to something entirely different. Our creativity is probably significantly more useful, but LLMs are creative in a way unbounded by human logic. It’s fair to say they’re unhinged in their creativity
The first time I really understood this was when, after hallucinating hide and seek several times (at the time I was running an LLM with no context window or system prompt just to get a feel for how it worked), it came up with a game called hide the comma, where it gave you a sentence with missing punctuation.
I’m pretty confident no human has ever done that, because it’s really fucking stupid. It’s not useful at all, you can put a comma in all sorts of places and completely change the meaning of a sentence. The AI would agree to any answer you give, because they’re chaotic little chat bots wired to meet expectations
But it was a brand new idea, an unhinged one, but creative in an inhumanly creative way
Here an even better one.
Data is just binary, right? Zeroes aren’t worth anything so, you can simply delete all of them. Next, count how many ones are left and record that number. If you started with 20 MB of data, you’ll probably end up with an answer like 82 296 203. That’s the number you need to keep.
When you want to look at the picture, simply reverse the process.
It can be used as a form of lossy compression, yeah, though it’s not the best tool for the job, or the best job for the tool
Holy shit, after reading up a bit more on quantization and weights last night, I was literally going post the same thing this morning. It’s lossy compression for knowledge.
And that’s dangerous because there are perfectly valid nuggets of information that just haven’t been “amplified” enough to get through the noise.
This is a perfectly succinct way of describing it
Theres a nice YouTube series exploring this concept: “Compression is intelligence” by 3blue1brown
It’s actually more like if you kept taking everyone’s hand, drawing an outline around it on paper, all sharing the same paper… ignore the fact that the paper will get filled in after enough samples. In theory, you’ll eventually have an outline which has every human hand factored in. You may call it whatever you want; it’s basically a super robust model of hands.
Now imagine that you can interact with the drawing. You can “pull out” a hand, any hand of your choosing. In fact, it doesn’t even need to have come from the same hand because it’s all measured using the same intersecting graphite lines. The model is so vast, full of hands, that it can generalize the patterns which make hands. So you can “pull out” a short-fingered fat-palmed hand, even if all sampled hands either had fat-palms or short-fingers.
Our hand model isn’t a compression of all hands.
Its more like if you were mining for gold… you’ve got to rinse out all the dirt from the mined substrate. In the case of our hand model, “dirt” is anything which isn’t mathmatically necessary for a hand. Maybe it’s the mistakes which occurred during tracing, or a boil on someone’s thumb… all of that gets blurred out by the much grander similarities from the totality of the sampled set. It’s those similarities in particular which we want to isolate. Call it the “formula of the hand,” determined via the most rigorous way one could: collecting every single simularity from every single example, isolating those simularities, then using the result as a function.
AI models do this with our language.
Reminds me of this bit in a book I read recently, Service Model by Adrian Tchaikovsky about AI robots in a post human society. Definitely go read it.
Rather than spoil the wonderful book, I’m gonna use a different approach.
You can feed all the words in the world into the great AI machine, and theoretically you can extract from it the total sum of all human knowledge.
All literature, poetry, art, music, mathematical proofs, all secrets of the universe can be potentially extracted from AI, you just have to pull the words out in the correct order.
If you consider AI a compression algorithm, then sure, but then so are a million monkeys with a million typewriters.
i just finished that book last week and i can read your subtext about the archivists. great little book.
I have a dictionary. It’s basically the same thing.
(For the record, it seems pretty likely humans are also a sensory compression algorithm; but they do come up with new ideas. Most new ideas are other ideas well compressed.)
It has already demonstratably come up with new ideas.
Do we know that for sure, or is the LLM telling us it came up with new ideas? All LLM does is average things. It can’t think, it just works quickly enough to convince people it can. LLMs just aggregate opinions, perhaps apply some weighting system devised by the creators of the LLM, and spit out the average result. Nothing new created, just averaged with no respect for truth or reality.
My problem whenever I see this argument is how do you know this is not what human minds are doing, that we just aggregate opinions and seem convincing. The benefit of the doubt we give to other humans is that if it convincingly seems like they are thinking then we are convinced.
Human brains are a lot more complicated with magnitutes more connections, life experience, and combines those provide opportunities to create something that hasn’t existed intentionally and not just as part of a random combination of training data and chance. A human crrates to accomplish something. AI outputs something based on prompts.
“It” as in AI which is not synonymous with LLMs.
Google DeepMind’s AlphaFold solved protein folding for example and the team won Nobel’s prize for it.
You clearly have absolutely no idea how LLMs work, so I’m looking forward to your explanation of how they have been able to solve novel problems that have eluded humans for decades. Say, these:
- https://openai.com/index/navier-stokes-solution/
- https://www.anthropic.com/research/ecdsa-curve-origins
- https://deepmind.google/blog/alphadev-discovers-faster-sorting-algorithms/
And since OP wasn’t talking about LLMs but AI in general, it would be remiss to not mention AlphaFold.
In the Stokes case a number of researchers that where close to the solution suspect OpenAI was reading their homework.
LLM’s certainly have the capability of extracting novel correlations from their vast training sets. But whether that really counts as “new thinking” remains to be seen.
On a couple of those, it’s because the scientists involved had been using LLMs in their work and it scooped them because it knew where to look. IIRC the consensus in the field had been that computer time would have been required.
On one of those.
LLMs had been used to solve problems as tools long before corporations started shoving it down our throats. That’s doesn’t give it some magical self awareness or ability to innovate. It was able to do all the things you linked because of people that know how to use it.
LLMs had been used to solve problems as tools long before corporations started shoving it down our throats.
[citation needed]. LLMs didn’t exist “long before” corporations started shoving them down our throats. GPT-1 only came out in 2018 and the paper that invented the architecture was released in 2017. And they were pretty useless until like 2022 or 2023.
yeah im guessing @[email protected] meant machine learning or ai.
Yeah, I stand corrected on the exact term.
