TCP packet size?

This is probably due to a protocol error. Can you show me the client code that sends the data? If you are sending multiple lines of data over a persistent connection you have to make sure every line is terminated by a newline character otherwise issues like this can occur.

in the initializer, i create a BufferedStream like so (C# code):

IPAddress addr;
if (!IPAddress.TryParse(server_name, out addr))
{
 addr = Dns.GetHostAddresses(server_name)[0];
}
IPEndPoint endPoint = new IPEndPoint(addr, port);
sock = new Socket(endPoint.AddressFamily, SocketType.Stream, ProtocolType.Tcp);
sock.NoDelay = true;
sock.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.SendTimeout, 3000);

sock.Connect(endPoint);

outputStream = new BufferedStream(new NetworkStream(sock));

Then in the function that periodically writes the stats, I do the following:

StringBuilder statsToSend = new StringBuilder();
foreach (KeyValuePair<string, int> stat in stats) // where stats is a dictionary of stats names and corresponding values
{
 if (IsValidName(stat.Key))
 {
  statsToSend.AppendFormat("{0}.{1}.{2} {3} {4}\n", periodString, this.qualifiedHostname, stat.Key, stat.Value, now);
 }
}

byte[] buffer = Encoding.ASCII.GetBytes(statsToSend.ToString());

try
{
 outputStream.Write(buffer, 0, buffer.Length);
 outputStream.Flush();
}
catch (Exception ex)
{
 Log.Info("Failed to write to CarbonClient:");
 Log.Info(ex);
}

Hm.... I'm not a C# guy but one theory would be that maybe the outputStream.Write() call is only writing some of the data. In some languages write() returns how many of the bytes it actually wrote and does not guarantee all will be written. Just a guess.

Barring that, one thing that might help debug the issue would be to log everything you're sending. Also just to make sure this code is single-threaded right?

Also, can you provide a specific example of a metric path (a.b.c.etc...) as you expect to see it and as it actually appears?

Unfortunately, our write() is a void method. I also logged what I was sending and it looked fine.

The code is multi-threaded but it's thread safe, i have locks everywhere.

Here's an example of an expected metric vs an actual output:

expected: s15.imeem.delta.oak1.i1.srv0205-17.plt.aggCount.song
actual: lta.oak1.i1.srv0205-17.plt.aggCount.song

also, reducing the number of lines written (breaking the stats into batches) seems to alleviate the problem, hence the packet size question.

I see. Packet size generally isn't an issue with TCP because when you send more data than can fit in one packet the data is transparently broken into multiple packets. Our Graphite deployment has thousands of clients that send lots of data (some in very large batches) so I doubt this is the cause.

Just for the heck of it could you run your client app with only one thread to see if the problem goes away? And when you say sending less lines alleviates the problem, do you mean you are sending less data overall or do you just mean that you run the method that sends the data more frequently?

It turned out to be a threading issue after all. I wasn't locking the output stream. The problem is solved now. Thanks for your help.

katya,
Can you please post your C# fix for sending tcp packets? I am encountering a similar situation.

Thanks.

Piyush Shah,

It was not a TCP packet problem, but a threading issue - multiple threads writing to the output stream without a lock.