SketchVLM is a training-free, model-agnostic framework from Auburn University, Adobe, and independent researchers (accepted at EMNLP 2026) that lets vision-language models like GPT-5, Gemini-3-Pro, and Claude draw their reasoning directly onto the input image. Instead of only typing an answer, the model produces non-destructive, editable SVG overlays — lines, circles, arrows, labels — that show exactly where it's looking and how it reached its conclusion.

How does SketchVLM work without training?

It overlays a coordinate grid on the image so the model can refer to precise locations (e.g., 'draw a line from x5y6 to x2y3'), then asks the VLM to output SVG strokes that get rendered back onto the picture. Because it's just prompting plus coordinate mapping, it works with any capable VLM through its API — no fine-tuning required. An adaptive grid scales to each image's resolution.

How much does drawing improve accuracy?

Across seven benchmarks (maze navigation, ball-drop trajectory prediction, object counting, part labeling, connect-the-dots, and drawing shapes), SketchVLM improves visual reasoning accuracy by up to +28.5 percentage points and annotation quality by up to 1.48x versus image-editing and fine-tuned sketching baselines — while producing annotations that are more faithful to the model's stated answer.

Where can I try SketchVLM?

There's a live demo at sketch-vlm-demo.vercel.app where you can upload an image, ask a question, and watch the model sketch. Code is open-source at github.com/Brandon-Collins7/sketchvlm, the paper is at arXiv:2604.22875, and three new benchmarks are on HuggingFace under loganbolton/sketchvlm.

SketchVLM: Making Vision Models Draw Their Reasoning Right Onto the Image

By Prahlad Menon Published 2026-10-19 4 min read

Ask a person where the CPU socket is on a motherboard and they won’t just say “upper middle.” They’ll point at it. Circle it. Maybe trace the path from the RAM slots to show you how they know. Humans explain visual reasoning visually. Vision-language models, for all their power, just type a paragraph and leave you to trust them.

SketchVLM — from Auburn University, Adobe, and collaborators, accepted at EMNLP 2026 — closes that gap. It lets a VLM draw its reasoning directly onto the picture.

The Problem With Text-Only Answers

Models like GPT-5, Gemini-3-Pro, and Claude answer image questions in prose. That’s hard to verify. If a model says “the ball will land in the third bin,” you have no idea whether it actually traced the physics or just guessed. The reasoning is a black box, and the only output is a sentence you have to take on faith.

SketchVLM turns that sentence into a visual argument you can check at a glance.

How It Works (And Why It’s Clever)

The whole thing is training-free and model-agnostic — no fine-tuning, no custom model. The trick is two simple ideas:

A coordinate grid. SketchVLM overlays a grid on the input image so the model can name exact locations — “draw a line from x5y6 to x2y3.” An --adaptive-grid option scales the grid to each image’s resolution so cells never get too fine or too coarse.
Editable SVG overlays. Instead of destructively editing pixels, the model emits SVG strokes — lines, circles, arrows, labels — that get rendered back on top of the image. Non-destructive and editable, so you can toggle, tweak, or correct them.

Because it’s all prompting plus coordinate mapping, it plugs into any capable VLM through its API. The repo ships adapters for Claude, GPT, Gemini, and OpenRouter, and the entire pipeline is one entry-point script.

There’s also a multi-turn mode (--mixed-stepwise): the model draws one stroke at a time, sees the updated canvas, and corrects itself — exactly how a person iteratively sketches. That opens the door to genuine human-AI collaboration on a shared canvas.

The Results

Across seven benchmarks spanning visual reasoning (maze navigation, ball-drop trajectory prediction, object counting) and drawing (part labeling, connect-the-dots, shape drawing), SketchVLM delivers:

Up to +28.5 percentage points on visual reasoning accuracy
Up to 1.48× better annotation quality vs. image-editing and fine-tuned sketching baselines
More faithful annotations — the drawing actually matches what the model says

The striking part: making the model draw doesn’t just make its thinking visible, it makes its thinking better. Forcing the model to commit to spatial coordinates seems to ground its reasoning — the same way sketching on a whiteboard sharpens a human’s. Single-turn generation already gets strong results; multi-turn adds collaboration on top.

Why This Matters

This lands right in the middle of two threads we care about: explainable AI and spatial reasoning. We’ve written about why code is often the best action interface for spatial reasoning and about turning video into 4D trajectory fields. SketchVLM is the lightweight, bolt-on cousin: it doesn’t need a new model or a training run, just a grid and an SVG renderer.

For anyone building agents that act on images — robotics, document understanding, UI automation, medical imaging — a model that can show you where it’s looking is a meaningful trust upgrade. “Circle the tumor” beats “there is a mass in the upper-left,” because you can immediately see if it got lost.

It’s open-source, works with the frontier models you already pay for, and you can try it in your browser right now.

Demo: sketch-vlm-demo.vercel.app · Code: github.com/Brandon-Collins7/sketchvlm · Paper: arXiv:2604.22875 · Data: loganbolton/sketchvlm on HuggingFace