Automation and AI

Multimodal Model

Also called Vision model, multimodal AI

A model that accepts more than one kind of input — text, images, audio, video — and works across them together.

Quick facts: Multimodal Model

Category
Automation and AI
Also called
Vision model, multimodal AI
Level
Intermediate
Affects
Alt text, creative review, product feeds, call summaries
Where to see it
ChatGPT, Claude, Gemini, ad-platform creative tools
In this article4
  1. How a multimodal model works
  2. Why multimodal models matter
  3. Common mistakes with multimodal models
  4. How to act on it

How a multimodal model works

A text-only model can read what you type and nothing else. A multimodal model converts different kinds of input into the same internal form, so a photograph, a screenshot, a recording and a sentence all become something it can process side by side. That is what lets you upload a picture of a product and ask a question about it in words, and get one answer that draws on both.

It is worth being clear about what happens next. The model is still predicting a likely response, not inspecting the object. It describes what an image resembles, based on everything similar it has seen. It reads text in a picture well when the text is clear and less well when it is small, stylised, at an angle, or in a script it has met less often. Its confidence does not drop when its accuracy does.

Why multimodal models matter

Most marketing material is not plain text. It is product photography, screenshots of dashboards, ad creative, recorded calls, PDFs and video. Until recently, using any of that with software meant a person first describing it in words. Multimodal models remove that step, which is why they are useful for jobs that were previously too fiddly to be worth automating.

The practical wins are ordinary ones: drafting alt text for a catalogue of product images, pulling attributes out of supplier photos for a feed, checking a batch of ad creative against brand rules before it goes live, summarising a recorded sales call, or explaining what a chart in a report is showing. These are all review-and-correct tasks, which is exactly where these tools belong.

Common mistakes with multimodal models

Treating a description as a verification is the main one. Asked whether an image shows the right product, a model will answer plausibly rather than carefully, and a near-identical variant — wrong colour, wrong size, wrong model number — is precisely the case it gets wrong. Anything where the detail is the point needs a human check.

The second is privacy. Customer photographs, identity documents, signed contracts and screenshots of an account often contain personal data, and uploading them to a consumer tool is a disclosure decision, not a technical one. The third is cost surprise: images and audio consume far more of a model’s capacity than the same request in words, so bulk processing gets expensive faster than people expect.

How to act on it

Test on your own material before committing. Take a sample of real product images, a call recorded in the accent your customers actually have, and a screenshot from your own reporting, and count the mistakes yourself. Vendor demonstrations use clean, well-lit, English-language examples that look nothing like a supplier’s photo taken on a phone.

Then use it where a wrong answer is cheap. Drafting alt text, first-pass tagging and transcript summaries are safe because errors are visible and quick to correct. Keep sensitive images out of consumer tools entirely, and put a person between the output and the customer, whether the job is content production or automated customer support.

Do and do not

Do

  • Test it on your own photos, recordings and screenshots
  • Use it where a wrong answer is cheap to correct
  • Mask personal details before uploading any image

Do not

  • Rely on it to verify which product variant an image shows
  • Upload customer documents to a consumer account
  • Assume bulk image processing costs the same as text

Questions people ask about this

Can a multimodal model write alt text for my product images?

It can draft it, and drafting is most of the work. What it produces describes what the image looks like, not what the image is for on your page, so it will happily write a fluent description that misses the point of the product or repeats the file name's mistakes. Edit for purpose, include the detail a buyer needs, and check anything where a variant could be confused.

Is it accurate at reading text inside images?

Reasonably, when the text is clear, printed and in a common script. Accuracy falls with small type, decorative fonts, photographs taken at an angle, handwriting and less common scripts, and the model rarely signals that it is unsure. For anything financial or legal, treat what it extracts as a draft to be checked against the original document.

Should I upload customer photos or documents to these tools?

Not to a consumer account. Photographs of people, identity documents, contracts and account screenshots usually contain personal data, and uploading them is a disclosure whatever the tool's convenience. Use a business plan whose data handling terms you have read, remove or mask what is not needed for the task, and write down internally which categories are never uploaded at all.

Related terms

Found this useful?

Share it, or ask an AI to summarise it

Back to the glossary

Knowing the term is the easy part

Applying it to your own site and budget is the work. Book a call and I will tell you what actually applies to you.