Image editing with a vision model and SAM2

• SmolVLM + SAM2
• 8x throughput
• same cost

The problem

Editing images by hand is slow, and every edit starts with the same two questions: what in this image should change, and exactly which pixels is that? A person answers both in seconds. A pipeline has to answer them for every image, without anyone looking.

Two models, two questions

The pipeline split those two questions between two models, each doing the one thing it is good at:

• SmolVLM looks and understands. A small vision-language model reads the image and works out what is in it and what the edit should apply to.
• SAM2 finds the exact shape. Segment Anything 2 turns that into a precise mask: the exact pixels of the object, not a rough box.

image ──▶ SmolVLM ──▶ what to edit ──▶ SAM2 ──▶ mask ──▶ edit ──▶ result

Keeping them separate made each step easy to test on its own. When an edit came out wrong, it was clear whether the model misunderstood the image or the mask missed the object.

8x faster without a bigger bill

The first version handled one image at a time, and most of that time the hardware was waiting: on loading files, on moving data, on the next step to start. The models weren't the bottleneck. The waiting was.

Multithreading fixed that. Several images moved through the pipeline at once, so while one was waiting, another was being worked on. Same machines, same cost, 8x the throughput.

Takeaway

Before paying for faster hardware, check whether the hardware you have is busy. Splitting the work into small steps with one model each made the pipeline easy to debug, and keeping every step busy made it 8x faster for free.