``baseplate.message_queue``
===========================
This module provides a thin wrapper around POSIX Message queues.
.. note::
This implementation uses `POSIX Message queues`_ and is not portable to
all operating systems.
There are also various limits on the sizes of queues:
* The ``msgqueue`` ``rlimit`` limits the amount of space the user can use on
message queues.
* The ``fs.mqueue.msg_max`` and ``fs.mqueue.msgsize_max`` sysctls limit
the maximum number of messages and the maximum size of each message
which a queue can be configured to have.
.. _POSIX Message queues: http://man7.org/linux/man-pages/man7/mq_overview.7.html
Minimal Example
---------------
Here's a minimal, artificial example of a separate producer and consumer
process pair (run the producer then the consumer):
.. testcode::
# producer.py
from baseplate.message_queue import MessageQueue
# If the queue doesn't already exist, we'll create it.
mq = MessageQueue(
"/baseplate-testing", max_messages=1, max_message_size=1)
message = "1"
mq.put(message)
print("Put Message: %s" % message)
You should see:
.. testoutput::
Put Message: 1
After running the producer once, we have a single message pushed on to our
POSIX message queue. Next up, run the consumer:
.. testcode::
# consumer.py
from baseplate.message_queue import MessageQueue
mq = MessageQueue(
"/baseplate-testing", max_messages=1, max_message_size=1)
# Unless a `timeout` kwarg is passed, this will block until
# we can pop a message from the queue.
message = mq.get()
print("Get Message: %s" % message)
You'll end up seeing:
.. testoutput::
Get Message: 1
The ``/baseplate-testing`` value is the name of the queue. Queues names should
start with a forward slash, followed by one or more characters (but no
additional slashes).
Multiple processes can bind to the same queue by specifying the same queue
name.
Message Queue Default Limits
----------------------------
Most operating systems with POSIX queues include very low defaults for the
maximum message size and maximum queue depths. On Linux 2.6+, you can
list and check the values for these by running::
$ ls /proc/sys/fs/mqueue/
msg_default msg_max msgsize_default msgsize_max queues_max
$ cat /proc/sys/fs/mqueue/msgsize_max
8192
Explaining these in detail is outside the scope of this document, so we'll
refer you to `POSIX Message queues`_ (or ``man 7 mq_overview``) for detailed
instructions on what these mean.
Gotchas
-------
If you attempt to create a POSIX Queue where one of your provided values is
over the limits defined under ``/proc/sys/fs/mqueue/``, you'll probably end
up seeing a vague ``ValueError`` exception. Here's an example::
>>> from baseplate.message_queue import MessageQueue
>>> mq = MessageQueue(
"/over-the-limit", max_messages=11, max_message_size=8096)
Traceback (most recent call last):
File "", line 2, in
File "/home/myuser/baseplate/baseplate/message_queue.py", line 83, in __init__
max_message_size=max_message_size,
ValueError: Invalid parameter(s)
Since the default value for ``/proc/sys/fs/mqueue/msg_max`` on Linux is 10,
our ``max_messages=11`` is invalid. You can raise these limits by doing
something like this as a privileged user::
$ echo "50" > /proc/sys/fs/mqueue/msg_max
CLI Usage
---------
The `message_queue` module can also be run as a command-line tool to consume,
log, and discard messages from a given queue::
python -m baseplate.message_queue --read /queue
or to write arbitrary messages to the queue::
echo hello! | python -m baseplate.message_queue --write /queue
See ``--help`` for more info.
baseplate.message_queue
-----------------------
.. automodule:: baseplate.message_queue
.. autoclass:: MessageQueue
:members:
Exceptions
----------
.. autoexception:: MessageQueueError
.. autoexception:: TimedOutError