How AI Models Understand Our Text❗💻🔠🔡
In today's world, interacting with AI models, passing prompts and generating ideas has undoubtedly become a necessity. But since ages, understanding language has always felt like a very human thing! . And now with the introduction of this large language models (LLMs) and their ability to interpret everything we pass as a prompt, definitely raises some sort of curiosity to figure out the driving mechanism behind it. So, in an attempt of trying to understand a few of these things, I explored the topic a bit and whatever I have learned, I would love to share in this blog in the simplest way possible. . . Whenever we pass a prompt to AI, immediately it does not understand anything, Let's take a simple sentence: ' The boy dropped the glass and it broke ' Step 1 : Tokenization The first thing the AI model does is break the sentence into small pieces/tokens (just like separating puzzle pieces before solving it). EXAMPLE: The | boy | dropped | the | glass | and | it | broke S...