- Redis Pub/Sub
- async
- event-driven
- remote GPU workers
- Dockerized services
The problem
Image models are slow and GPUs are expensive. If the app that receives a request also runs the model, every request waits in line on that one machine, a burst of traffic stalls everything, and adding a second GPU means changing the app.
The models needed to live somewhere else, and requests needed a way to reach them that didn't care how many GPUs there were or how busy they were.
Publish, don't call
The engine was built on Redis Pub/Sub. Instead of calling a model directly, the app publishes a job as a message on a Redis channel and moves on. Workers on remote GPU machines subscribe to the channels for the jobs they can run, do the work, and publish the result on a results channel that the app listens to.
PUBLISH jobs SUBSCRIBE jobs
app ─────────────────────▶ Redis ─────────────────────▶ GPU workers
▲ channels │
└──── SUBSCRIBE results ◀──────────── PUBLISH results ◀──────┘Redis fit because the job was small and the API is smaller: PUBLISH and SUBSCRIBE, with messages delivered in well under a millisecond. Async clients let one process listen on its channels and keep serving requests, without a thread blocked waiting for each result.
Three things got simpler at once:
- The app stays fast. Publishing a job takes milliseconds, however long the model takes.
- The app never waits on a GPU. It hears back when a result event arrives, not by holding a request open.
- Scaling is adding a worker. A new GPU machine subscribes and starts taking work. Nothing else changes.
What Redis Pub/Sub won't do for you
Pub/Sub is fire-and-forget. A message goes to whoever is subscribed at that moment and is not stored anywhere. If no worker is listening, or a worker dies halfway through a job, that message is simply gone. Every subscriber on a channel also gets every message, so sharing work between workers has to be designed, not assumed.
If I built it again, I'd look at Redis Streams with consumer groups: the same Redis, but messages are kept until a worker acknowledges them, and each one goes to exactly one worker.
From notebook to worker
The models started as experiments: notebooks and scripts that worked once, on one machine. To become workers they were rewritten as object-oriented Python, with one class per pipeline and a clear way in and out, and each was packaged in Docker behind an endpoint. Every worker then looked the same to the engine, whatever model it ran inside.
The image-editing pipeline is one of the jobs that ran this way.
Takeaway
When the work is slow and the hardware is scarce, put a message layer between the people asking and the machines doing. The app stays responsive, the GPUs stay busy, and growing means adding workers instead of rewriting code.