# A pub/sub engine for GPU inference

By [Anubhav Choubey](https://anubhavchoubey.com/) · Backend & Systems · 2026-11-03 · 3 min read

> An async, event-driven engine on Redis Pub/Sub that sends inference jobs to remote GPU workers, so slow models never block the app.

```stats
Redis Pub/Sub · async · event-driven · remote GPU workers · Dockerized services
```

## The problem

Image models are slow and GPUs are expensive. If the app that receives a request also runs the model, every request waits in line on that one machine, a burst of traffic stalls everything, and adding a second GPU means changing the app.

The models needed to live somewhere else, and requests needed a way to reach them that didn't care how many GPUs there were or how busy they were.

## Publish, don't call

The engine was built on **Redis Pub/Sub**. Instead of calling a model directly, the app **publishes a job** as a message on a Redis channel and moves on. Workers on remote GPU machines **subscribe** to the channels for the jobs they can run, do the work, and publish the result on a results channel that the app listens to.

```text
            PUBLISH jobs                SUBSCRIBE jobs
app ─────────────────────▶  Redis  ─────────────────────▶ GPU workers
 ▲                          channels                          │
 └──── SUBSCRIBE results ◀──────────── PUBLISH results ◀──────┘
```

Redis fit because the job was small and the API is smaller: `PUBLISH` and `SUBSCRIBE`, with messages delivered in well under a millisecond. Async clients let one process listen on its channels and keep serving requests, without a thread blocked waiting for each result.

Three things got simpler at once:

- **The app stays fast.** Publishing a job takes milliseconds, however long the model takes.
- **The app never waits on a GPU.** It hears back when a result event arrives, not by holding a request open.
- **Scaling is adding a worker.** A new GPU machine subscribes and starts taking work. Nothing else changes.

```callout note
The channel is the contract. The app only knows how to publish a job and read a result; it never knows which worker, which GPU or which model version did the work.
```

## What Redis Pub/Sub won't do for you

Pub/Sub is fire-and-forget. A message goes to whoever is subscribed at that moment and is not stored anywhere. If no worker is listening, or a worker dies halfway through a job, that message is simply gone. Every subscriber on a channel also gets every message, so sharing work between workers has to be designed, not assumed.

```callout warn
Track each job's status outside the channel and time out jobs that never report back. The channel tells you something happened; it doesn't remember that it did.
```

If I built it again, I'd look at **Redis Streams with consumer groups**: the same Redis, but messages are kept until a worker acknowledges them, and each one goes to exactly one worker.

## From notebook to worker

The models started as experiments: notebooks and scripts that worked once, on one machine. To become workers they were rewritten as object-oriented Python, with one class per pipeline and a clear way in and out, and each was packaged in Docker behind an endpoint. Every worker then looked the same to the engine, whatever model it ran inside.

The [image-editing pipeline](/blog/vision-model-image-editing) is one of the jobs that ran this way.

## Takeaway

When the work is slow and the hardware is scarce, put a message layer between the people asking and the machines doing. The app stays responsive, the GPUs stay busy, and growing means adding workers instead of rewriting code.