You type a question, and within a second a fluent, well-structured answer rolls out. It feels as if something on the other side understands your question, thinks for a moment and then replies. What actually happens is something quite different — and once you see it, you never look at a chatbot the same way again.

The core principle: predicting the next word

A language model essentially does one thing: based on the text so far, it predicts which word (technically: which fragment of text) is likely to come next. Then it does it again, and again, word by word, until the answer is complete.

Compare it to the word suggestions on your phone, but vastly more powerful. Where your phone might suggest “weekend” after “have a nice”, a large language model can weigh the entire preceding conversation: your question, the tone, the topic, earlier answers. That is why it produces not loose fragments but coherent paragraphs, in the right style, on almost any subject.

But it remains prediction. The model does not look anything up in a database of facts, it does not reason the way a human does, and it has no idea whether what it says is true. It calculates: given all these words, which word fits best statistically?

Where does that prediction come from? Training data

To predict well, the model was trained on enormous amounts of text: web pages, books, articles, code and more. During training, the model was repeatedly shown a piece of text and asked to guess the next word. When it got it wrong, its internal settings — billions of dials, called parameters — were adjusted a tiny bit. Repeat that an unimaginable number of times, and you get a system that has learned the patterns of language astonishingly well: grammar, style, factual associations that appeared often in the text, and even something resembling reasoning.

Three limitations follow directly from that training data, and you notice them in practice every day:

Fluent is not the same as correct

Here comes the most dangerous misunderstanding. We humans use language skill as a quality signal all our lives: someone who formulates clearly and confidently usually knows what they are talking about. With a language model, that rule of thumb collapses completely.

The model is trained to produce plausible language — that is literally its trade. A fabricated answer therefore comes out just as fluent, structured and assertive as a correct one. There is no difference in tone whatsoever between a fact and a fabrication. The model attaches no confidence score and does not “know” it is inventing something: it simply predicted words that fit well, exactly as it does when the answer happens to be right.

The most important sentence in this article: a language model produces language that fits your question, not information that is true. The two often coincide — on well-covered topics they usually do — but the model itself cannot tell the difference. That is your job.

What is a context window?

Another term that explains a lot: the context window. It is the amount of text the model can “see” at once and factor into its prediction — your questions, its own answers, and any documents you have uploaded. Everything inside the window counts; everything outside it simply does not exist for the model.

Three practical consequences:

  1. A language model has no memory the way we do. In a very long conversation, information from the beginning can fall outside the window or fade in weight — and the model “forgets” agreements you made earlier. Started a new chat? Then the model begins with a blank slate (some tools store a separate profile or notes, but that is an added feature, not a property of the model itself).
  2. Large documents are not always fully processed. If a document barely fits in the window, a summary may miss parts. For important documents, always check whether specific sections were actually taken into account.
  3. What you put in the window steers the answer. Clear instructions, relevant background and examples improve output dramatically — the model can only work with what it sees.

So why does it work so well anyway?

After all these caveats, the obvious question: how can word prediction produce such useful results? The answer: because language contains an enormous amount of structure. Learning to predict how people write, at this scale, implicitly means learning a great deal about how people think, argue and explain. For tasks like summarising, rewriting, structuring, translating and brainstorming — tasks where the input is already in front of you — a language model is genuinely strong. The risk sits mainly in tasks where the model has to supply facts itself.

That distinction is the practical heart of AI-literate working:

What to take away

You do not need to go deeper into the technology than this. But these four sentences deserve a place in the back of your mind every time you open a chatbot:

  1. The model predicts words; it does not consult a database of facts.
  2. It only knows what appeared (often enough) in its training data.
  3. Fluent and assertive says nothing about correct.
  4. Only what is in the context window exists for the model.

This is exactly the kind of basic knowledge the AI Act means by AI literacy — understanding what your tool does, so you can use its strengths and see the risks coming. Curious how solid your basics are? Take the free quiz. And if you want to master this — and the rest of the fundamentals — properly, have a look at our AI literacy course.