Language and LLMs
How machines process language, from tokens and embeddings to the large language models behind ChatGPT, Gemini and Claude.
Imagine someone who has read most of the internet and got extremely good at guessing the next word. Then they were coached to be a helpful assistant. That is roughly a large language model.
Where it fits
Not shown: 17 more kinds and parts. See the full list below.
+All relationships as a list
Inside this domain
- AI Agents and Tool Use
- BERT
- Direct Preference Optimization
- Embedding
- Fine-Tuning
- Google Translate
- GPT
- Hallucination
- In-Context Learning
- Keyboard Autocomplete
- Large Language Model
- LLM Evaluation and Benchmarks
- Perplexity
- Pretraining
- Prompting
- Retrieval-Augmented Generation
- RLHF
- Sampling and Temperature
- Scaling Laws
- Tokenization
- Vector Database
- word2vec
Overview
The path from raw text to a chat assistant:
- Tokenization splits text into pieces (tokens) the model can count.
- Embeddings turn each token into a vector of numbers that captures meaning.
- A Transformer is pretrained on huge amounts of text. Encoder models like BERT learn to fill in blanks; decoder models like GPT learn to predict the next token.
- Scaled up, decoder models become large language models. Sampling settings decide how creative their output is.
- RLHF and related methods tune the model into a helpful, harmless assistant such as Claude.
- RAG connects the model to fresh or private documents so it answers from facts rather than memory.
In practice you shape a model with prompting first, fine-tuning when needed, and tools for actions and fresh information, while watching for hallucination.
Everyday use cases
- Case 01
Chat assistants
Answering questions, drafting text, explaining code.
- Case 02
Search and question answering over documents
Retrieval-augmented generation grounds answers in a company's own files.
- Case 03
Translation and summarization
Turning long or foreign text into something you can read quickly.