How are LLMs like ChatGPT actually trained?

Language models known as LLMs (Large Language Models), such as those powering ChatGPT, are not created in a single step. Before becoming publicly available, these models go through a rigorous learning process, structured in several phases combining massive datasets, human supervision and progressive adjustments.

Here are the main stages that make these models so capable.

1. Pre-training

This is the first, foundational phase. It consists of training the model on a giant dataset: billions of texts from the internet are collected, cleaned and converted into tokens — the language units the model can process.

🔹 Knowledge base: massive extraction of textual content.
🔹 Conversion into tokens: words are broken down to make processing easier.
🔹 Neural network: the model learns to detect complex relationships between tokens using deep architectures.
🔹 Next-word prediction: this is probabilistic training — the model learns to guess the next word in a sentence.

At this stage, the model is not yet aligned with conversational logic. It has raw knowledge of language, but no "behaviour" yet.

2. Supervised fine-tuning

A second phase humanises the model. Human annotators create sets of questions and answers, which serve as examples for the model to learn to respond like an assistant.

🔹 Example data: concrete scenarios are fed into the system.
🔹 Targeted adjustment: the model refines its answers from the human examples.
🔹 Initial alignment: it starts adopting conversational behaviour, with useful and coherent logic.

This is a key step in turning a raw model into an assistant.

3. Reinforcement learning

From here, the model learns to self-correct. It knows the right answers, but must now discover the best way to formulate them.

🔹 The model is presented with several possible formulations for the same question.
🔹 It is "rewarded" for the best answers (according to quality and relevance criteria), and "penalised" for the weaker ones.
🔹 This trial-and-error optimisation process lets the model improve its "reasoning" over time.

This phase makes models more robust when facing complex cases.

Reinforcement Learning from Human Feedback (RLHF)

Finally, to refine the model's behaviour further, an advanced method is used: RLHF.

🧑‍🏫 Humans evaluate several answers generated for the same question.
🥇 The model is optimised according to this feedback to improve its future behaviour.
📈 This phase lets the model progress without requiring new data: it learns from its mistakes and aligns with human preferences.

In summary

LLMs such as ChatGPT are not "magic". They are the result of a long, structured process:

A massive understanding of language.
Supervised learning with human examples.
A phase of continuous improvement through feedback loops.

👉 It is only after these stages that the models are made available to end users, to guarantee coherent answers that are useful and safe.