Multimodal AI
Models that connect different kinds of data, such as images, text and audio, so they can describe pictures, answer questions about them or generate one from the other.
Most early AI could only read, or only see. Multimodal AI can look at a photo and talk about it, like a friend who can both see and speak.
Where it fits
+All relationships as a list
Inside this domain
Overview
Humans combine senses effortlessly. For AI, connecting a picture of a dog to the word “dog” had to be learned.
Three ideas carry most of the field:
- Shared embedding spaces. Contrastive learning trains an image encoder and a text encoder so that matching pairs get similar embeddings. CLIP is the famous example; it powers image search and guides text-to-image generators.
- Vision plus language model. A vision-language model feeds image features into a large language model, so it can describe, reason about and answer questions on images.
- Natively multimodal models. Systems like Gemini are trained on several modalities from the start rather than bolting an image encoder onto a text model afterwards.
Everyday use cases
- Case 01
Ask about a photo
Snap a picture of a broken appliance or a math problem and ask what to do.
- Case 02
Search photos with words
Find "receipts" or "my dog at the beach" in your photo library.
- Case 03
Accessibility
Describe images aloud for blind and low-vision users.