Multimodal agents — text + image

In one line

Give the model eyes. Send images with your text and it can read, describe, compare, or reason over them.

Key idea

In 1.0 you send mixed input as a list of content blocks inside one message. A text block plus an image block. The image can come three ways:

• URL — point to an image on the web. Easiest and lightest.
• base64 — paste the raw image data. Needs a mime_type like image/png.
• file_id — an id for a file you already uploaded to the provider.

Send an image (URL)

from langchain.agents import create_agent
agent = create_agent(model="claude-sonnet-4-6", tools=[])

message = {
    "role": "user",
    "content": [
        {"type": "text", "text": "What is in this image?"},
        {"type": "image", "url": "https://example.com/photo.jpg"},
    ],
}
result = agent.invoke({"messages": [message]})
print(result["messages"][-1].content)

Send an image (base64)

{"type": "image", "base64": "AAAA....", "mime_type": "image/png"}

Reading the reply

The model also answers in content blocks. An image-making model returns an image block. Read them the same way for every provider:

for block in response.content_blocks:
    if block["type"] == "text":
        print(block["text"])
    elif block["type"] == "image":
        print(block.get("url") or block.get("base64"))

Good use cases

• Read text from a photo or screenshot (OCR).
• Describe a chart or diagram.
• Compare two product images.
• Check a receipt or form.

Watch out

• Not every model takes images. Claude, GPT, and Gemini do. Check the model first.
• Same pattern works for audio and video blocks (model permitting).
• Tools can return images too. MCP tools give back image content blocks the agent can use.

Links

• Messages & multimodal input (docs)
• Models — multimodal output (docs)
• Models (content blocks)

My notes