That's because LLM doesn't just produce

It is ostensibly correct for large-scale language models (LLMs) to predict the probability of the next word, but fundamentally different from that of "statistical language models" before deep learning.

That's because LLM doesn't just produce statistics by counting the frequency with which words in the learning data appear. Past statistical language models simply looked up a list of words that frequently appeared between "in the library" and "book" when predicting words that would go in the blank in the sentence "I read a book yesterday." But the way LLM works is much more sophisticated than this.

It's not just statistics how LLM predicts the next plausible word. LLM is trained to identify the context of the preceding words through the attention mechanism. The attention mechanism works as if we are 'attentioning' the important parts of a sentence. It calculates how relevant each word is to all the other words, and expresses them as weights.

For example, in the example above, LLM focuses on words like "library," "book," "read," and "read," and then figures out how they relate to each other. And based on that, it calculates the probabilities of the right words to fit in the blanks ("fun," "beneficial," "interesting," etc.).

More importantly, LLM can grasp multiple overlapping contexts at the same time rather than a single context. In the example above, the context of the 'place of library', the context of 'the act of reading books', and the context of 'the style of describing personal experiences' exist at the same time. Since language is multidisciplinary and context-dependent by nature, it is necessary to grasp the context of multiple layers.

These semantic connections are mapped to locations in abstract higher-dimensional spaces. It's similar to placing words on a special map. For example, words like 'king' and 'queen', 'man' and 'woman' have a certain distance relationship in terms of gender. 'chair' and 'sofa' will be placed close in terms of 'sitting furniture' and placed with slight differences in terms of 'comfort'.

Each word has a specific coordinate value in hundreds of semantic dimensions like this. A kind of map is created. Of course, it's a map that represents meaning. It's called a map, but it's not the kind of two-dimensional map we usually see. It's a higher-dimensional map of hundreds of dimensions. In this high-dimensional space, semantic relations between words are expressed in geometric relationships such as distance and angle.

What is even more surprising is that these semantic maps are not simply created according to the rules set by the designer. LLM constructs this semantic space on its own by learning huge textual data. As a child learns a language, it is like naturally grasping the meaning of words and the relationship between them.

Maybe meaning is the connection relationship itself in this abstract, higher-dimensional space. And when we have a map of the connection relationship, I think that's what we understand. Recent neuroscience studies have shown that our brains represent concepts in a similar way. Different regions of the brain are responsible for specific dimensions of meaning, and they encode relationships in multiple dimensions.

For example, when we think of the concept of an apple, several areas of our brain are activated at the same time, including the visual cortex (shape and color), the olfactory cortex (scent), and the taste area (flavor). These multiple activation patterns are the neurological representation of the concept of an apple. The semantic representation of LLM is also surprisingly similar.

The way humans feel that they understand something is maybe not that different, because when we understand a new concept, we place it in relation to existing concepts, like, "Oh, it's kind of like this," or, "Well, it's not like this."

This perspective offers interesting insights into the relationship between artificial intelligence and human intelligence. LLM's ability to understand and generate language cannot be said to be exactly the same as that of humans, but at least its basic mechanism seems to have surprising similarities. Of course, this does not mean that LLM has consciousness or ego like humans. However, in terms of understanding and expression of meaning, the fact that artificial intelligence may operate more similar to that of humans is significant.

It also has important implications for the future of AI. Beyond simply learning more data into larger models, we may be able to create a more efficient and natural language understanding system based on our understanding of human cognitive structures.