PyMQI

Trying to issue MQCMD_STOP_CONNECTION with MQBACF_CONNECTION_ID from MQCMD_INQUIRE_CONNECTION

Asked by Rik Baeten

Briefly summarized I'm writing a script to cleanup orphaned queue manager connections. (By default WMQ never time-outs connections and I'm not willing to change this setting at this moment, since it will impact too much applications not reconnecting automatically. Sometimes abrupt network outages or firewalls can cause stale connections on the WMQ server, which cause occupied connection resources on the queue manager).

Basically I'm calling MQCMD_INQUIRE_CONNECTION to know the existing connections. Afterwards I'm linking this with info (MQCACH_LAST_MSG_DATE) from MQCMD_INQUIRE_CHANNEL_STATUS to know which connections are not in use anymore. Finally I will call MQCMD_STOP_CONNECTION to stop them.

Here's some simplified sample code that just closes the first connection returned by MQCMD_INQUIRE_CONNECTION:
<code>
import pymqi
from CMQC import *
from CMQCFC import *

cd = pymqi.cd()
cd.ChannelName = 'SYSTEM.DEF.SVRCONN'
cd.ConnectionName = 'myserver(1414)'
cd.ChannelType = MQCHT_CLNTCONN
cd.TransportType = MQXPT_TCP
qmgr = pymqi.QueueManager(None)
qmgr.connectWithOptions('QMX', opts=MQCNO_HANDLE_SHARE_NO_BLOCK, cd=cd)
pcf=pymqi.PCFExecute(qmgr)
# Determine connectionID of first connection
# Stopping a random connection is dangerous of course, so change it to another one if required!
connDicts=pcf.MQCMD_INQUIRE_CONNECTION({MQBACF_GENERIC_CONNECTION_ID:pymqi.ByteString(''),
                                    MQIACF_CONNECTION_ATTRS:MQIACF_ALL,
                                    MQIACF_CONN_INFO_TYPE:MQIACF_CONN_INFO_CONN})
connId=connDicts[0][MQBACF_CONNECTION_ID]
# Now close this connection
pcf.MQCMD_STOP_CONNECTION({MQBACF_CONNECTION_ID:pymqi.ByteString(connId)})
</code>

Now my concrete issues are:
1) The connDicts[0][MQBACF_CONNECTION_ID] is converted to ascii, whereas in runmqsc the equivalent runmqsc call (dis conn(*) type(conn) all) returns the connection ID in hex splitted over the CONN and EXTCONN parameters. Of course I can achieve this myself by issuing binascii.hexlify(connDicts[0][MQBACF_CONNECTION_ID]). I'm not sure if you would consider pymqi responsible for issuing the hex-conversion.

In the infocenter I read: "The value for EXTCONN is based on the first sixteen bytes of the ConnectionId converted to its 32–character hexadecimal equivalent. Connections are identified by a 24-byte connection identifier. The connection identifier comprises a prefix, which identifies the queue manager, and a suffix which identifies the connection to that queue manager. By default, the prefix is for the queue manager currently being administered, but you can specify a prefix explicitly by using the EXTCONN parameter. Use the CONN parameter to specify the suffix."
=> In practice I've observed the value returned by binascii.hexlify(connDicts[0][MQBACF_CONNECTION_ID]) corresponds to the concatenation of the EXTCONN, CONN from runmqsc and '0020'-padding.

2) When I issue the MQCMD_STOP_CONNECTION with any of the connectionId values in hex or in ascii I always get a MQRCCF_NONE_FOUND. I've also tried hex and ascii values from the EXTCONN and CONN values from runmqsc.

Question information

Language:
English Edit question
Status:
Solved
For:
PyMQI Edit question
Assignee:
Dariusz Suchojad Edit question
Solved by:
Rik Baeten
Solved:
Last query:
Last reply:
Revision history for this message
Dariusz Suchojad (dsuch) said : 		#1

Hi,

this time again everything looks fine on your end. I'm setting up an environment with the API trace on and will be comparing what gets passed into PCF calls when using PyMQI and what's there when using other languages.

I'll keep you informed.

As to whether PyMQI should un-/hexlify the values back and forth, well, I'm not sure yet, let's first see what the correct form accepted by MQAI calls is.

Revision history for this message
Hannes Wagener (johannes-wagener) said : 		#2

Hi,

Think Found it.

Have a look at and compile pymqe.c in branch: lp:~johannes-wagener/pymqi/byte-string-fix

I managed to solve this one differently though. Over the weekend I wanted to see if I could send and receive PCF messages with pymqi using normal puts and gets to the command queue. After a (very brief) investigation I realised that pymqi does not support the PCF header or the PCF type structures - after which I promptly coded and tested them. I did not think they would come in handy as soon as they did(to help solve this bug was their first application).

This bug had some sides to it. This is how the analysis went:

1.) I wrote a script(test_stop_conn_pcf_put_get.py) to use the new PCF structures I defined to test INQUIRE_CONNECTION and STOP_CONNECTION with PCF messages put to the "SYSTEM.ADMIN.COMMAND.QUEUE" queue(the script looks for a specific connection and tries to stop it). It worked like a charm. From this I realised that INQUIRE_CONNECTION and STOP_CONNECTION wants the MQBACF_CONNECTION_ID in it's byte string form and not hexlified. I then modified your script(see test_stop_conn_mqai.py) to try and stop the same connection as my PCF script.

2.) I then set out to figure out where it was going wrong. At first the CONNECTION_ID being returned look ok so I figured something must be going wrong during the set of it before the INQUIRE or STOP PCF calls.

The following code confused me because I could not understand how it was getting to the "value" attribute of the "ByteString" object.

<code>
strByteArg = (MQBYTE *)value;
mqAddByteString(adminBag, paramType, PyObject_Length(value), strByteArg, &compCode, &compReason);
</code>

After a few tries I realised that that my "C" is really rusty and that it was not getting to the "value" attribute of the ByteString object. I changed the code to the below - It works - but please confirm that it's correct.

<code>
strByteArg = (MQBYTE *)PyString_AsString(PyObject_GetAttrString(value, "value"));
mqAddByteString(adminBag, paramType, PyObject_Length(value), strByteArg, &compCode, &compReason);
<code>

3.) After this fix I was still getting MQRCCF_NONE_FOUND. After some trial and error I realised that the length of the MQBACF_CONNECTION_ID returned in my script was longer than than the one in your script.

In your script:
CONN ID LEN: 22
CONN ID: 414d514354535450594d5149202020201644fd4c0840

in my script:
CONN ID LEN: 24
CONN ID: 414d514354535450594d5149202020201644fd4c08400020

I realised immediately that your id was being terminated at the first null. The code below where the bytestring was being set was the only place it could be going wrong:

<code>
itemByteStrVal[byteStrLength] = 0;

pyItemStrVal = PyString_FromString(itemStrVal);
PyDict_SetItem(resultsDict, key, pyItemStrVal);
</code>

I looked up PyString_FromString in the python manual to see if they mention anything about null-terminated strings and immediately noticed his brother PyString_FromStringAndSize and tried it.

<code>
/*itemByteStrVal[byteStrLength] = 0;*/
/*byte strings may contain nulls */
pyItemStrVal = PyString_FromStringAndSize((char *)itemByteStrVal, byteStrLength);
PyDict_SetItem(resultsDict, key, pyItemStrVal);
</code>

After this it worked first time!

Dariusz - My "C" is really rusty and my knowledge of the Python C api was picked up from copying and pasting from pymqi - so please verify that you are happy with what I have done.

I you do not mind - please merge my changes in pymqe.c if you are happy with them - but please register a branch for the new PCF structures I defined in pymqi.py. One or two of them still needs to be tested to MQ and I still have some code outstanding on the get and put of PCF messages via "pymqi.Queue".

This one kept me up. Going to bed!

Cheers,

H.

Revision history for this message
Dariusz Suchojad (dsuch) said : 		#3

Wow Hannes, thanks a heap! Yesterday night I went as far as fixing the mqAddByteString stuff and noticing a difference in the length of connection IDs but it was way too late for doing anything about it.. The changes look fine, the only thing I'd like to make sure before pushing it to trunk is whether there aren't any Py_XDECREF missing - I'm in a bit hurry right now but I'm going to do it later on today.

Revision history for this message
Dariusz Suchojad (dsuch) said : 		#4

Rik - I've merged the fix by Hannes and all seems fine. Can you try the current trunk (r38) now?

Hannes - in r38 I've cherrypicked only parts of the changes from your branch, those that were related to the Rik's question. Let's merge the rest in bug #686286, OK? As to what I've changed as compared to the code of yours - the call to PyObject_GetAttrString returns a reference to the object and that reference needs to be released by the caller, which is pymqe in that case, so I added a Py_XDECREF(byteStringValue); in there.

Revision history for this message
Hannes Wagener (johannes-wagener) said : 		#5

Oh Shucks! So sorry. I did not mean to step on your toes or waste your time. I should have told you that I am looking at it.

Ok. I understand Py_XDECREF. Thanks. My "C" is very rusty and my experience with the python C api is minimal. I have added the new MQSUB/MQSUBRQ calls for pub/sub - but I just copied the code from the other calls.

What do you think of the PCF messages via put/get? Can you register an official branch for me? I want to include the test suite and the new PCF structures.

Please advise.

H.

Revision history for this message
Dariusz Suchojad (dsuch) said : 		#6

All's good, I'm new to the whole byte strings business so I certainly don't fell like I've wasted any time.

Regarding the new branch - can you try registering it yourself, giving 'pymqi-dev' as its owner? Of course, not that I don't want to do it myself, but you're a member of the PyMQI Dev team and I'm curious myself whether you have the necessary power for creating new branches owned by the whole team - would be handy if you had (that would be one of the points of joining a Launchpad team if you ask me). If not, just let me know and I'll create it right away.

Revision history for this message
Rik Baeten (rik-baeten) said : 		#7

Hey Dariusz and Hannes,

Sorry for the rather late reaction. I have cancelled the use of MQCMD_STOP_CONNECTION, since it doesn't clean up channel instances or sockets. It merely blocks an active connection. Instead I will implement the KeepAlive setting on our queue manager(s): this will automatically clean the orphaned channels.

For more information, see:
http://www.mqseries.net/phpBB2/viewtopic.php?t=55987

I haven't tried out your fix, but I suppose it works.

Thanks very much for your assistance,
Rik