Multi Modal (integration of multi data types) breakthrough in 12 months, page-37

They are exactly probability models. The training data is organised into "tokens". The tokens are then tagged with additional information about meaning, sentiment, source, etc. Many tags! The machine learning algorithm is trained with the tagged tokens, producing a model.

When you give ChatGPT a prompt (e.g. ask it a question), it breaks the prompt into tokens. It then checks the model to see which tokens are most likely to appear in response to the prompt tokens. These responses are ranked in order or probability. Depending on what "temperature" setting you have asked ChatGPT to use, it will pick a response token from amongst the most common tokens. At a temperature of 1, ChatGPT will choose only from amongst the very topmost probability responses. This will give you the least variability and the answers are most likely to be factual. At a temperature of 0, it will choose from well down the probability list giving you some pretty crazy responses.

The most important point here is that ChatGPT never "understands" the prompt in any meaningful sense. It is just a blob of data that that is used to identify potential responses using the model. Similarly it has no meaningful "understanding" of what it is presenting in the response... it's just a sequence of more or less likely tokens.

Over the top of this is a layer of language recognition that allows ChatGPT to identify subject matter tokens from your prompt. It will then substitute your subject tokens into the corresponding subject tokens in its response. This is how it can make up a story about "Calvo and the Three Bears". Interestingly, in the resulting story:

1. Calvo always seems to be a fairy
2. Calvo always gives in to the temptation to enter the bears' house
3. There are always three things for Calvo to sample
4. The first two things are always wrong in opposite ways with the third thing being just right
5. The bears always return while Calvo is still
6. The bears are always amused by Calvo's antics
7. Calvo and the bears become the best of friends