How do you use the pickle format?

Asked by Bill Wheatley on 2011-06-02

In a prior question you talked about pickle format
https://answers.launchpad.net/graphite/+question/121579

Though you did not really go into details on how to get it to work.

I was passing pickled data (i think) to port 2004 of the graphite server.
#!/usr/bin/perl

use strict;
use warnings;

use IO::Socket;

my $socket = new IO::Socket::INET (
    PeerAddr => '192.168.122.190',
    PeerPort => '2004',
    Proto => 'tcp',
);

my @time = localtime;
my $now = time;

die "Unable to open socket: $!" unless ( $socket );

printf $socket "(lp0\n(lp1\nS'test.bill.pickle1'\np2\naI11\na$now\naa(lp3\nS'test.pickle2'\np4\naI22\naI1306891549\naa(lp5\nS'test.pickle3'\np6\naI33\naI1306891549\naa.";

close $socket;

Then in the listener.log i see that it receives the pickle connection. ( i think)

02/06/2011 01:50:15 :: MetricPickleReceiver connection with 192.168.122.7:57156 established
02/06/2011 01:50:15 :: MetricPickleReceiver connection with 192.168.122.7:57156 closed cleanly

Put i don't see anything in storage/whisper

Question information

Language:
English Edit question
Status:
Solved
For:
Graphite Edit question
Assignee:
No assignee Edit question
Solved by:
Dave Rawks
Solved:
Last query:
Last reply:
Revision history for this message
Jon Stevens (latchkey-gmail) said :
#1

I guess I should ask... why are you bothering? There is a perfectly valid/easy format for you to write to the socket to already. Just use that.

Revision history for this message
Best Dave Rawks (drawks) said :
#2

the pickle format is quite a bit more efficient in on the carbon side of things, but if you are writing your metric collector or proxy in perl I'm thinking pickle is not what you want to do. At any rate in python you want something like

        data = [ metric_name, [(timestamp,value) for record in records] ]
        serializedData = pickle.dumps(data, protocol=-1)
        prefix = struct.pack("!L", len(serializedData))
        message = prefix + serializedData

then use socket comms to send the message object.

Revision history for this message
Bill Wheatley (bwheatley) said :
#3

Thanks Dave Rawks, that solved my question.

Revision history for this message
Bill Wheatley (bwheatley) said :
#4

Thanks for the useful feedback dave. We just wanted to start with pickled data to pass it in to graphite then see how it handled the data. Once we got that we could use the Perl lib Python::Serialize::Pickle to pickle the data in perl.