Multimodal: pictures and sound in the same call
A multimodal call is a chat request whose message holds a picture or a sound clip next to the text, sent to a model that can read that kind of input.
Last updated: 12 Oct, 2026 · LiteLLM 1.105
Structured output your code can read shaped the reply. This lesson changes the question: the user message carries an image. Of the three course models, only gemini-2.5-flash-lite accepts one. Both gpt-oss models are text-only, and the lesson shows the call that proves it before showing the call that works.
Syntax: the message content becomes a list of parts instead of one string. Each part has a type.
content = [
{"type": "text", "text": "What colour fills this image?"},
{"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}},
]
messages = [{"role": "user", "content": content}]Checking which models take pictures
LiteLLM keeps a table of what each model accepts. supports_vision() and supports_audio_input() read it, so a program can check before it sends.
import litellm
for model in ["groq/openai/gpt-oss-120b", "groq/openai/gpt-oss-20b", "gemini/gemini-2.5-flash-lite"]:
print(model, "vision:", litellm.supports_vision(model),
"audio:", litellm.supports_audio_input(model))groq/openai/gpt-oss-120b vision: False audio: False groq/openai/gpt-oss-20b vision: False audio: False gemini/gemini-2.5-flash-lite vision: True audio: True
Only Gemini is marked for pictures and sound. The next two examples test that against real calls.
Sending a picture
Making a test picture
A picture sent inside the request is a data URL: the file's bytes, base64 encoded, after a data:image/png;base64, prefix. The helper below builds a solid-colour PNG with the standard library, so no image file is needed.
import base64, struct, zlib
def square(rgb, size=16): # a solid-colour PNG as a data URL
pixels = (b"\x00" + bytes(rgb) * size) * size
def chunk(kind, data):
body = kind + data
return struct.pack(">I", len(data)) + body + struct.pack(">I", zlib.crc32(body))
header = struct.pack(">IIBBBBB", size, size, 8, 2, 0, 0, 0)
png = (b"\x89PNG\r\n\x1a\n" + chunk(b"IHDR", header)
+ chunk(b"IDAT", zlib.compress(pixels)) + chunk(b"IEND", b""))
return "data:image/png;base64," + base64.b64encode(png).decode()Asking Gemini about two pictures
The loop sends a red square and then a blue one with the same question. If the model reads the image, the answers differ.
from litellm import completion
for name, rgb in [("first", (220, 20, 20)), ("second", (20, 20, 220))]:
content = [
{"type": "text", "text": "What colour fills this image? One word."},
{"type": "image_url", "image_url": {"url": square(rgb)}},
]
response = completion(model="gemini/gemini-2.5-flash-lite",
messages=[{"role": "user", "content": content}], max_tokens=100)
print(name, response.choices[0].message.content)first Red second Blue
The answer follows the picture: red for the first square and blue for the second. The text part of the message was identical both times.
Sending the same picture to the text-only models
Both gpt-oss models get the same message. Groq rejects it, and LiteLLM raises the rejection as BadRequestError. The try block prints the status code and Groq's message instead of stopping the program.
import litellm
from litellm import completion
litellm.suppress_debug_info = True # hides LiteLLM's help banner on errors
content = [
{"type": "text", "text": "What colour fills this image? One word."},
{"type": "image_url", "image_url": {"url": square((220, 20, 20))}},
]
for model in ["groq/openai/gpt-oss-20b", "groq/openai/gpt-oss-120b"]:
try:
completion(model=model, messages=[{"role": "user", "content": content}], max_tokens=100)
except litellm.BadRequestError as error:
print(model, type(error).__name__, error.status_code)
print(error.message)groq/openai/gpt-oss-20b BadRequestError 400
litellm.BadRequestError: GroqException - {"error":{"message":"messages[0].content must be a string","type":"invalid_request_error","param":"messages[0].content"}}
groq/openai/gpt-oss-120b BadRequestError 400
litellm.BadRequestError: GroqException - {"error":{"message":"messages[0].content must be a string","type":"invalid_request_error","param":"messages[0].content"}}Each error says messages[0].content must be a string. A text-only model accepts only a string there. A list of parts is refused before any answer is attempted, on the 20b model and on the 120b model alike.
Sending sound
Sound goes in as an input_audio part: the clip's bytes in base64 and its format. Gemini is the only course model marked for audio input.
content = [
{"type": "text", "text": "Transcribe this clip."},
{"type": "input_audio", "input_audio": {"data": audio_base64, "format": "wav"}},
]
response = completion(model="gemini/gemini-2.5-flash-lite",
messages=[{"role": "user", "content": content}], max_tokens=300)Not run here: the course has no recorded clip to send, and a generated beep would not show what the model hears. The call shape is the part to keep. For turning speech into text with a dedicated model, the transcription call in Endpoints beyond chat is the usual route.
Text content vs a list of parts
| A string | A list of parts | |
|---|---|---|
content holds | Text only | Text, images or sound, each as a part |
| Accepted by | Every chat model | Models marked for that kind of input |
| Checked with | Nothing needed | supports_vision() or supports_audio_input() |
| On a text-only model | Works | BadRequestError from the provider |
Where you use multimodal calls
- Reading a screenshot or a receipt. The user sends the picture and the model returns the amounts.
- Describing product photos. One call per image, with a short instruction in the text part.
- Handling voice notes. A short clip goes in as an audio part and comes back as text.
supports_vision() before sending a picture, or the call fails with a 400 from the provider. A router that can send the request to a text-only deployment will hit this error on some calls and not others.Related
- Previous: Structured output your code can read
- Next: CustomLLM: writing a provider of your own
- See also: Providers and keys: the model string for how the model string picks the provider
- Reference: Using vision models in the LiteLLM docs
- Change the second colour to green,
(20, 150, 20), and read the answer. - Change the question to
Is this image bigger than 10 pixels wide? One word.and compare it with a largersizeargument tosquare(). - Send a plain text message, such as
Name one colour., togroq/openai/gpt-oss-120b. The same model answers whencontentis a string.
Every expert started right here.