Multimodal AI
Multimodal AI is a model that accepts, or produces, more than one kind of data: text together with images, audio or video, inside one system rather than a chain of separate ones. Most current assistants are multimodal on the input side, so a photograph can be part of the question a customer asks.
What the models accept
OpenAI’s images and vision guide describes models that take text and images in the same request, with images supplied as a URL, as base64-encoded data, or as an uploaded file, in PNG, JPEG, WEBP or non-animated GIF. The output is text about the image, or with the image tool, a new or edited image. Other vendors document comparable inputs. The point for a business is not the format list; it is that a customer can now photograph a broken boiler, a rash, or a competitor’s price board and ask “who fixes this near me” without typing a description.
What reaches the model from a page
An assistant that consults the web does so through a crawler, and a crawler receives a page as HTML: text, markup and references to image files. Whether an image is fetched and looked at is a decision of the retrieval system, not of the model, and it is not something a site owner can assume. Information that exists only inside an image, an opening-hours graphic, a scanned menu, a price list as a JPEG, may be perfectly legible to the model if shown it and invisible to the crawler that never was. Alt text, captions and a text version of the same facts make the image redundant, which is the safe state.
One question owners ask
Does a multimodal assistant read the photos on my website?
Only if its retrieval step fetches them, and the crawler documentation this site has read describes fetching pages, not analyzing every image on them. A customer can hand an assistant a photograph directly; a crawler passing through a site receives the image file, its alt text and the text around it. Treat an image as unread until the assistant’s own documentation says otherwise, and put anything the image says into text on the page.
Related terms
Sources
Accepted inputs and formats: OpenAI’s images and vision guide. Read as documentation of one vendor’s API, not as a description of how any assistant’s crawler treats images on a site.