All upload failed after RescueTime maintenance in Feb 25, 2009

Asked by Donny Kurnia on 2009-02-28

Hi,

On Feb 25 RescueTime do the maintenance to their system. After that, all upload from my computer failed.

This is the reply before RescueTime went down

$ ls -al /home/donny/.rescuetime/replys/332.upload.encoded.data
-rw------- 1 donny donny 114 2009-02-25 07:36 /home/donny/.rescuetime/replys/332.upload.encoded.data
$ cat /home/donny/.rescuetime/replys/332.upload.encoded.data
<hash><successes>49</successes><failures>0</failures><total-records>49</total-records><notices>[]</notices></hash>

333 until 338 replys is html:
We're currently down for system-wide upgrades to make everything better
  as of 17:11 PST.

Then

$ ls -al /home/donny/.rescuetime/replys/339.upload.encoded.data
-rw------- 1 donny donny 100 2009-02-25 11:13 /home/donny/.rescuetime/replys/339.upload.encoded.data
$ cat /home/donny/.rescuetime/replys/339.upload.encoded.data
<hash><successes>0</successes><total-records>0</total-records><error>bad value sent</error></hash>

After 339, all replys are the same, "bad value sent".

Is this because RescueTime change their API? Anyone have the same problems and have the solution?

Question information

Language:
English Edit question
Status:
Solved
For:
RescueTime Linux Uploader Edit question
Assignee:
No assignee Edit question
Solved by:
Donny Kurnia
Solved:
2009-03-04
Last query:
2009-03-04
Last reply:
2009-03-04
rssaddict (simoneau-louis) said : #1

Same problem here. I'm getting a 404 error for api.rescuetime.com

What happened to the API?

I am getting the exact same thing. Everything else seems to be running fine.

Elliot Murphy (statik) said : #3

I pushed revision 91 with the URL update that Jason provided. Can you let us know if it works?

Still failing:

<hash><successes>0</successes><total-records>0</total-records><error>bad value sent</error></hash>

The only file modified with 91 was uploader.py, correct? Dropping that in and rebuilding should have worked?

Elliot Murphy (statik) said : #6

It should have, but now the message from Brian is that the yaml is invalid:

http://blog.rescuetime.com/2009/03/02/linux-data-collector-needs-your-support/ (see comment 3)
"Hi Elliot - the URLs are the same, the problem is the YAML being sent doesn’t seem to be correct. We’re using a strict YAML parser now and the format of what the linux collect sends is being rejected."

This is probably easy to fix, but I'm buried in fixing some other bugs in my $DAYJOB. Any info that you can provide is most welcome, and I'll try to apply patches promptly.

Hey there guys, I finally got a bit of a breather and am helping dig into why the Linux uploader fails after our latest release.

I don't think there is anything specific that the Linux uploader is doing in correctly, and I have sample YAML data from the Linux uploader on my box that duplicates the issue. We're trying to track down why our new API code thinks the YAML data is not valid - from my quick check, it appears to be fine to me. My guess right now is that is an encoding issue that we should be handling better on our side, but I'll let you know what we find out.

Thank you guys!

Elliot Murphy (statik) said : #8

removing the urlescaping seems to have helped make things work again.

--
Elliot Murphy

On Mar 3, 2009, at 17:40, "Joe \(RescueTime\)" <<email address hidden>
 > wrote:

> Question #62639 on RescueTime Linux Uploader changed:
> https://answers.launchpad.net/rescuetime-linux-uploader/+question/
> 62639
>
> Joe (RescueTime) proposed the following answer:
> Hey there guys, I finally got a bit of a breather and am helping dig
> into why the Linux uploader fails after our latest release.
>
> I don't think there is anything specific that the Linux uploader is
> doing in correctly, and I have sample YAML data from the Linux
> uploader
> on my box that duplicates the issue. We're trying to track down why
> our
> new API code thinks the YAML data is not valid - from my quick
> check, it
> appears to be fine to me. My guess right now is that is an encoding
> issue that we should be handling better on our side, but I'll let you
> know what we find out.
>
> Thank you guys!
>
> --
> You received this question notification because you are a member of
> Rescuetime-developers, which is an answer contact for RescueTime Linux
> Uploader.
>

Donny Kurnia (donnykurnia) said : #9

Can you provide example of working yaml data?

This is data from my computer that failed.

http://pastie.org/405874

Having another sample is very helpful.

I think the encoding on the os_username is what is causing our new API to choke on the files. We're putting some changes in our Test environment tomorrow for verification and will see if this is something we can fix on our side without causing other issues.

Donny Kurnia (donnykurnia) said : #11

I'm testing the 92 release now. I'll report it's result in the next one hour.

Elliot Murphy (statik) said : #12

Here is an example of working and failing data, note the only difference
is the urlencoding of the values.

http://pastie.org/406771

--
Elliot Murphy | https://launchpad.net/~statik/

What if you enquoted the username, so it is treated as a string literal?

Elliot Murphy (statik) said : #14

On 03/03/2009 10:07 PM, mvandemar wrote:
> mvandemar proposed the following answer:
> What if you enquoted the username, so it is treated as a string literal?
>

Thats probably a good idea. So, put quotes around the username, but
don't encode the backslash, and put back the urlencoding of the window
titles?
--
Elliot Murphy | https://launchpad.net/~statik/

Donny Kurnia (donnykurnia) said : #15

Result:

$ cat replys/589.upload.encoded.data
<hash><successes>0</successes><total-records>0</total-records><error>error parsing YAML</error></hash>

Failed YAML sample:

http://pastie.org/406790

Donny Kurnia (donnykurnia) said : #16

Elliot, I saw your sample data.

Release 92 have the correct os_username, and window_title, but in my example, app_name have spaces. Will adding quote around app_name value solve this?

Ok, I got one to work by adjusting it manually. Enquoting the username didn't help, but escaping the percent signs did. This example succeeded using the 91 release (I manually escaped the %'s before uploading):

http://pastie.org/406798

I just tested again. It looks like the only field that needs escaping is in fact the os-username... if the app name has unescaped percent signs in it there is no fail. This is another example of a successful upload:

http://pastie.org/406806

Although when I look at donnykurnia's last example, there are no unescaped percent signs in his username, but there are @ signs in the app names, so not sure if there is an issue with those or not...

And I just realised that there @ signs in my app names, but they are encoded as %40... so it looks like if you put the url encoding back, then escape the percent signs out of the os-username field, it should work. I think. :)

Donny Kurnia (donnykurnia) said : #20

Got success result

$ cat replys/592.upload.encoded.data
<hash><successes>62</successes><total-records>62</total-records></hash>

Content of file finished/592.upload.encoded.data

http://pastie.org/406829

I change the file uploader.py at line 298
Release 92:
out += ' app_name: %s\n' % str(self.app_name)

Become
out += " app_name: '%s'\n" % str(self.app_name)

It seems quoting app_name value make the YAML valid. No need to change it back to escaped value.

Elliot Murphy (statik) said : #21

You guys rock! I'm pushing out a 93 release now, with the changes proposed by donny. Testing appreciated.

Elliot - the uploader works like a charm on my end. :)

Donny Kurnia (donnykurnia) said : #23

Elliot, today I got failed upload

This is the failed YAML data:
http://pastie.org/407124

The window title contain single quotes. So it need escaping the single quote found in window_title value.

I also do some experiment today uploading failed YAML data. First I change the os_username become new format, then I put quotes around app_name. All data uploaded successfully.

I think we can go back to escaped YAML data for app_name and window_title value, as long as it wrapped inside single quotes. I think the only need change with the new RescueTime server parser is os_username.

I will do more experiment on this.

Donny Kurnia (donnykurnia) said : #24

This is old YAML data that successfully uploaded:
http://pastie.org/407127

Hey guys, Mark here. Notorious author of the new API collector code.

Looks like you are on the right track. The specific error the YAML parser was getting was multiple entries in field expecting one entry (presumably because escaping issue led to spaces or quotes where inappropriate).

What has happened is we have introduced a layer of asynchrony to improve general server performance and especially client response times (your uploads are quick now, even if quickly rejected :) ).
The flip side is we are also positioning ourselves to a change in the send format of data (the current method will continue to be supported also) from YAML to CSV. For this reason, the inbound YAML now gets translated to CSV, so that we didn't have to implement the asynchronous processor to support 2 data formats. This leads to escaping madness.
However, the windows/mac clients that use the yaml lib to package the data seem to be fine.

I do not explicitly url unescape the os_username, but the parser seems handle this fine.

I'm hopefully going to have some time to toss in assistance here later this week. My suggestion, and what I would implement, is instead of trying to build valid YAML by string contatenation, build a python dict object and let the python yaml parser build it for you. That should eliminate all questions of valid YAML escaping.

--Mark