Vision-Language Model
A language model that can also see: it takes images (and text) as input and answers questions, describes scenes or reads documents.
Also VLM · image captioning · visual question answering · VQA · multimodal LLM
A chatbot that has been given eyes. Show it a picture, ask anything about it, and it answers in words.
Where it fits
+All relationships as a list
Is a kind of / part of
Kinds and parts
Siblings
Uses / built on
How it works
A common recipe for building a vision-language model:
- Take a pretrained vision encoder (for example from CLIP) that turns an image into a set of feature vectors.
- Add a small projection that maps those vectors into the same space as the language model’s token embeddings.
- Feed the projected image “tokens” into a large language model together with the text.
- Train on image-text data, then on instructions (“What is unusual about this picture?”) so it behaves like an assistant.
To the language model, the image simply looks like a few hundred extra tokens, and attention lets text tokens look at image regions.
Some newer systems, such as Gemini, are trained on mixed images and text from the start instead of attaching vision afterwards. Either way, the same weakness as text-only models applies: they can produce fluent descriptions of details that are not actually there (see hallucination).
Everyday use cases
- Case 01
Explain a screenshot or chart
Ask what an error dialog means or what trend a chart shows.
- Case 02
Read documents
Extract totals from receipts, fields from forms, text from handwritten notes.
- Case 03
Describe images for accessibility
Detailed descriptions of photos for blind and low-vision users.
Famous examples
- 2023Gemini
Google DeepMind's family of AI models and assistant, designed from the start to work across text, images, audio, video and code.
- 2023Claude
Anthropic's family of AI assistants: Transformer-based large language models trained with RLHF and Constitutional AI to be helpful, honest and harmless.
Strengths, weaknesses and when to use it
Strengths
- Flexible: any question about any image, in plain language.
- Reads text in images, charts and diagrams.
Weaknesses
- Can confidently describe things that are not in the image.
- Still weaker than people at precise counting and spatial reasoning.
When to use itWhen you need open-ended understanding of images in combination with language.
Compare with
Check yourself
- Q01
How is a vision encoder commonly connected to a language model?
Show answer
A: Image features are projected into the language model's token embedding space