it’s a compressed lookup table. rather than there being one response for every input, the input is used as a seed to decompress relevant parts of the dataset, with some added randomness. you can even do it with gzip itself: https://nathan.rs/posts/gzip-lm
It could be represented with a series of lookup tables, especially quantized LLMs. A series of inputs results in a specific output that gets passed to the next set of nodes. Repeat 7 billion times, and the final output from the last set of nodes is a set of token probabilities.
? A neural network is not a lookup table.
it’s a compressed lookup table. rather than there being one response for every input, the input is used as a seed to decompress relevant parts of the dataset, with some added randomness. you can even do it with gzip itself: https://nathan.rs/posts/gzip-lm
the linked paper is very good: https://arxiv.org/pdf/2309.10668
That’s like saying the neural network in your head has compressed your knowledge of language and just expands the relevant parts when needed.
yup
It could be represented with a series of lookup tables, especially quantized LLMs. A series of inputs results in a specific output that gets passed to the next set of nodes. Repeat 7 billion times, and the final output from the last set of nodes is a set of token probabilities.