mosquitto

rate limit on publishing messages?

Asked by mschulz

Hi Roger

I have been using your mosquitto server code since the release of the first version. Thanks for a fine piece of work.

I am working on a project now which is using MQTT to publish active badge data from the openbeacon.org project. I have purchased 100 badges from Bitmanufactur in Germany and these appear to be working well. I have a previous student project which replaced their custom built Easy Reader hardware (used to collect and transfer the data from each badge to a server process) with an opensource Netduino board + a Sparkfun nRF24L01+ breakout board. This worked well, until we changed our desktop IP network address. That change required a recompilation of the software(??), and the student who reloaded the software started blowing up the Netduinos (a known problem when reflashing memory). Rather than waste anymore time on these (now obsolete) boards, I decided to try to use a Raspberry Pi as the basis of the badge reader.

I now have an Arduino Uno + Sparkfun nRF24L01+ breakout board capturing badge packets and forwarding these over the serial link (115200 baud) to the Raspberry Pi. I have re-writeen the reader software from the Openbeacon, now in Python (and way, way more readable at 2 pages of A4 total). The original software called for the data to be forwarded to downstream software agents on UDP port 2432. I decided to use MQTT and pump this data up to our mosquitto server.

The data from each badge arrives about every 6 seconds.

When I trace the badge data (without sending the data to mosquitto) I can see this happening at about that speed. I have a 'while 1:' loop that reads the badge and then does some processing, then prints the data to the screen. All looks fine at this point.

When I add 'rc = mqttc.loop()' into the loop (leaving 'publish' out for the moment), traces of the badge data are now printed to the screen about once a second - I can match this to the tick of the clock outside my door :-).

DETAILS:
Mosquitto Version 1.1
Python (Mac OS X) Version 2.7

QUESTION:
Does 'mqttc.loop()' introduce a time delay into its response? If so, can I turn it off?

I need to be able to pump the data up to the server at a pretty steady rate. I expect to have up to 100 badges in range of the reader at times, and so need to be able to handle 100 messages per 6 seconds, or 1000 messages per minute. Is this possible?

Yes, I might have to update the 16 MHz Arduino Uno before then, but that plan is in process.

Any advice would be great. Especially if what I am doing has a very obvious error in it. :)

Cheers
Mark

==== python code extract ======

while rc == 0:
    try:
   data = readTagData()

   g_Beacon = bytearray()
   g_hdr = bytearray()

   time_stamp = int(time.time())

   g = struct.Struct('>BBHHLL')
   g_hdr = bytearray(g.pack(protocol, interface, reader_id, message_size, reader_sequence, time_stamp))
   reader_sequence = reader_sequence + 1

   # .icrc16 header checksum
   g_all = struct.Struct('>H')
   crc = 0xffff ^ xxtea.crc16(str(g_hdr[2:]))
   g_Beacon = bytearray(g_all.pack(crc))

   g_Beacon.extend(g_hdr)
# add the body of the header to the CRC16 of the header (wierd layout??)
   s = binascii.b2a_hex(g_hdr)
   print s + data

# mqttc.publish("/openbeacon/LIB/reader1", s + data)

   rc = mqttc.loop()
    except KeyboardInterrupt:
   break

=======================

Question information

Language:
English Edit question
Status:
Answered
For:
mosquitto Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Roger Light (roger.light) said : 		#1

Hi Mark,

Good to hear from you again and it's nice to see you're still using
MQTT and mosquitto.

I've not come across openbeacon before, but it sounds like a pretty
cool project.

> QUESTION:
> Does 'mqttc.loop()' introduce a time delay into its response? If so, can I
> turn it off?

It depends, but yes. Inside loop() there is a call to select(), to
wait on network traffic. If there is outgoing traffic and the socket
can send it or if there is data available to be read, select() will
return immediately and there will be no delay. If neither of these is
the case, then select() will wait for up to 1 second before returning.
Generally speaking, if you're only transmitting data then you will
have a minimum amount of incoming traffic and the publish() call will,
in most cases, transmit all of your data (if you have a large payload
then not all of the data will be transmitted during the publish() call
and so loop() will finish it off). In this case, you end up with no
outgoing traffic and no incoming traffic by the time you hit loop(),
which means there is a delay.

There are two ways to get around this. The first is to reduce the
timeout parameter to loop (default of 1.0). A value of 0 is allowed
and means return immediately with no delay. The second option is to
not use loop() at all, but use loop_start() instead before your while
loop. This starts a separate thread that calls loop() so you never
need to. It will also call reconnect() if the connection drops. You
can use loop_stop() if you want to stop the second thread.

I hope that helps.

Cheers,

Roger

Revision history for this message
mschulz (mark-f-schulz) said : 		#2

Hi Roger

The program I was modifying already had a number of threads running, and i wasn't sure what would result if I introduced yet another thread. I went with setting loop(0).

Everything runs perfectly now.

I'll post notes on the project and leave a copy of the code on Github as soon as I have a stable version running. Should now be real soon.

Thanks again for your help.

Cheers
Mark

>
> There are two ways to get around this. The first is to reduce the
> timeout parameter to loop (default of 1.0). A value of 0 is allowed
> and means return immediately with no delay. The second option is to
> not use loop() at all, but use loop_start() instead before your while
> loop. This starts a separate thread that calls loop() so you never
> need to. It will also call reconnect() if the connection drops. You
> can use loop_stop() if you want to stop the second thread.

Revision history for this message
Roger Light (roger.light) said : 		#3

That's great, definitely use whatever works, I just wanted to make sure you
knew of the threaded option.

Cheers,

Roger

On Sat, Apr 6, 2013 at 5:31 AM, mschulz <
<email address hidden>> wrote:

> Question #225320 on mosquitto changed:
> https://answers.launchpad.net/mosquitto/+question/225320
>
> Status: Answered => Open
>
> mschulz is still having a problem:
> Hi Roger
>
> The program I was modifying already had a number of threads running, and
> i wasn't sure what would result if I introduced yet another thread. I
> went with setting loop(0).
>
> Everything runs perfectly now.
>
> I'll post notes on the project and leave a copy of the code on Github as
> soon as I have a stable version running. Should now be real soon.
>
> Thanks again for your help.
>
> Cheers
> Mark
>
> >
> > There are two ways to get around this. The first is to reduce the
> > timeout parameter to loop (default of 1.0). A value of 0 is allowed
> > and means return immediately with no delay. The second option is to
> > not use loop() at all, but use loop_start() instead before your while
> > loop. This starts a separate thread that calls loop() so you never
> > need to. It will also call reconnect() if the connection drops. You
> > can use loop_stop() if you want to stop the second thread.
>
> --
> You received this question notification because you are a member of
> Mosquitto PPA, which is an answer contact for mosquitto.
>

Can you help with this problem?

Provide an answer of your own, or ask mschulz for more information if necessary.

To post a message you must log in.