Producer/Consumer pattern with ZeroMQ

If you’re using ZeroMQ to distribute jobs across a bunch of worker processes, keep in mind that the examples given here…

…don’t include any flow control.  Therefore, if your producer is producing messages much faster than your consumers can consume them, the messages will buffer.  And buffer.  To the point that you will either run out of memory, or your workers will start to page out and their performance will degrade badly.

The good news is, there’s an easy fix for this.  Use ZMQ’s “High Water Mark” feature to implement blocking on PUSH- and PULL-type sockets.  In pyzmq, use the method:

socket.setsockopt(zmq.HWM, LIMIT)

On the sockets for both the producer and consumers.  Where LIMIT is the maximum number of outstanding messages.  I’ve set PRODUCER_LIMIT to be n*CONSUMER_LIMIT where n is the number of consumers, seems to work pretty well.