Build a Multimodal Assistant
Build an assistant that reasons over images and text together using Claude's vision input, then extends that with tool use so it can act on what it sees rather than just describing it.
Prerequisites: An Anthropic API key; basic Python; comfort with JSON-shaped API payloads.
Targeted versions: anthropic Python SDK (current release); Claude Sonnet and Opus tier models with native vision input — verified against Anthropic's API documentation, mid-2026.
"Multimodal" here means an assistant that takes images and text as input in the same conversation and reasons over both jointly — reading a chart and answering questions about its trend, comparing two product photos, or describing a UI screenshot well enough to write code against it. Current Claude models (Sonnet and Opus tier as of mid-2026) accept image input natively in the same message format as text; there's no separate vision endpoint to learn.
You'll build up from a single image-plus-question call to a multi-turn conversation that keeps images in context, then add tool use so the assistant can take action based on what it observes in an image rather than only describing it.
Test what you just learned
Self-testing is one of the best ways to retain new skills. Unlock project quizzes to check your understanding.
Log in to unlock0 / 7 complete