Move widelands bugs into launchpad tracker

Asked by SirVer on 2010-02-10

In our continuous strive to move to launchpad, we would like to move the bug tracker also away from sourceforge. I read that an XML format is needed for launchpad, but I trust that importing from sourceforge is so important that you already have scripts to handle their format.

You can find a full XML export from the widelands project on sourceforge here:

http://widelands.org/~sirver/widelands_export.xml

This contains all information including the trackers. Could you import them into the widelands project on launchpad and enable bug tracking for it?

I can also try to convert the format to launchpads XML format, but I cross my fingers that you can manage ;)

Cheers,
Holger

Question information

Language:
English Edit question
Status:
Solved
For:
Launchpad itself Edit question
Assignee:
Graham Binns Edit question
Solved by:
Graham Binns
Solved:
2010-05-07
Last query:
2010-05-07
Last reply:
2010-05-07

Graham is the man for this.

Graham Binns (gmb) said : #2

Hi,

Here's how bug imports generally work.

 1. You provide the XML (already done, thank you)
 2. We do a test import on a local instance to ensure that that XML is importable.
 3. We do a test import, usually on a Launchpad instance running in Amazon EC2, so that you can check that the imported data look sane.
 4. We do the import on staging.launchpad.net to ensure that it works when imported into the full Launchpad dataset
 5. Assuming that everything goes well in step 4 we do the import on the Launchpad production database.

I'll try to get the local import done today. Please let me know when you're likely to be free to review the import on an ec2 instance.

Regards,

Graham

Graham Binns (gmb) said : #3

> I read that an XML format is needed for launchpad, but I trust that importing from sourceforge is so important that you already have scripts to handle their format.

I skipped over this bit when reading your initial question, and shouldn't have.

Our original importer for SourceForge no longer works. This is because it was written before SourceForge XML exports were available. It relies on screen-scraping and so the update to SourceForge's templates broke it completely.

We do want to work towards having an importer that will handle SourceForge XML natively. However, at the moment we don't have the resources to work on this problem (as you said, it's important, but there are many things in the Launchpad Bug Tracker that are important to work on and only five developers contracted to work on them full time).

In the meantime, we have an interchange schema for Bug imports, which can be found at https://help.launchpad.net/Bugs/ImportFormat. If you are able to convert your data to this format then we'll be able to work with the converted version of your data to perform the import. Similarly, we're willing to work with you or other community members if you want to produce a script which will do the conversion from SourceForge -> Launchpad XML; we'll help as much as possible.

Please let me know how you'd like to proceed with this.

SirVer (sirver) said : #4

Graham, thanks for the reply. I will try to come up with a conversion script, we have tons of bugs to transfer.

SirVer (sirver) said : #5

Graham,

I converted the sf.net XML into one conforming to
https://help.launchpad.net/Bugs/ImportFormat

The resulting file can be found here
http://widelands.org/~sirver/wl_launchpadxmlimport.xml

I am unsure if everything is alright since I cannot properly test it. The script I use is highly automated and I added it as a launchpad project in the hope that it will be useful for you/launchpad/others. You can find it under:

https://edge.launchpad.net/sfbugs2launchpad

I'd like to push this forward so that our bugs are imported in roughly two weeks. Importing the bugs from sf.net is a showstopper for our next release.

Thanks for your help.

Holger

Graham Binns (gmb) said : #6

On 2 March 2010 13:52, SirVer <email address hidden> wrote:
> Question #100549 on Launchpad Bugs changed:
> https://answers.edge.launchpad.net/malone/+question/100549
>
>    Status: Answered => Open
>
> SirVer is still having a problem:
> Graham,
>
> I converted the sf.net XML into one conforming to
> https://help.launchpad.net/Bugs/ImportFormat
>
> The resulting file can be found here
> http://widelands.org/~sirver/wl_launchpadxmlimport.xml
>
> I am unsure if everything is alright since I cannot properly test it.
> The script I use is highly automated and I added it as a launchpad
> project in the hope that it will be useful for you/launchpad/others. You
> can find it under:
>
> https://edge.launchpad.net/sfbugs2launchpad
>
> I'd like to push this forward so that our bugs are imported in roughly
> two weeks. Importing the bugs from sf.net is a showstopper for our next
> release.
>

Thanks for all the hard work. We really appreciate you pushing the
code up to Launchpad, too. That's really great.

I'll take a look at the data within the next 24 hours and get back to
you if there are any problems. Presuming we can resolve any issues
quickly it should only take a couple of days to get the import
completed (though we're rolling out a new version of Launchpad to our
productions servers this week, which could hold up the import for a
while).

I see no reason why we can't get the import complete within your time
frame, though. I'll get back to you as soon as I can.

SirVer (sirver) said : #7

Graham, I did not hear from you. Were there any problems with the data?

Graham Binns (gmb) said : #8

On 5 March 2010 13:23, SirVer <email address hidden> wrote:
> Question #100549 on Launchpad Bugs changed:
> SirVer is still having a problem:
> Graham, I did not hear from you. Were there any problems with the data?
>

I'm working on doing a local import at the moment, now that the
rollout is out of the way. I'll let you know if anything comes up.

Graham Binns (gmb) said : #9

On 5 March 2010 14:33, Graham Binns
<email address hidden> wrote:
> I'm working on doing a local import at the moment, now that the
> rollout is out of the way. I'll let you know if anything comes up.

I've done the local import and it's worked fine after some tweaking.
The only error that I ran into was that all the dates are missing a
trailling "Z". However, a quick sed fixes that problem; I'll file a
bug on the importer script.

Please let me know the next time you're online and I'll get the bugs
imported into our staging instance or an EC2 instance for you to check
to see whether they look okay, then once you're happy with that we can
proceed to the production import.

Graham Binns (gmb) said : #10

On 5 March 2010 14:33, Graham Binns
<email address hidden> wrote:
> I'm working on doing a local import at the moment, now that the
> rollout is out of the way. I'll let you know if anything comes up.

I've done the local import and it's worked fine after some tweaking.
The only error that I ran into was that all the dates are missing a
trailling "Z". However, a quick sed fixes that problem; I'll file a
bug on the importer script.

Please let me know the next time you're online and I'll get the bugs
imported into our staging instance or an EC2 instance for you to check
to see whether they look okay, then once you're happy with that we can
proceed to the production import.

SirVer (sirver) said : #11

Please proceed with the import. I will be online for another 2 hours.

The dates where formatted using datetime.isoformat(); obviously that is not the correct format. What should be the date format? What is this Z meant to mean?

Graham Binns (gmb) said : #12

On 5 March 2010 15:31, SirVer <email address hidden> wrote:
> Question #100549 on Launchpad Bugs changed:
> https://answers.launchpad.net/malone/+question/100549
>
>    Status: Answered => Open
>
> SirVer is still having a problem:
> Please proceed with the import. I will be online for another 2 hours.
>
> The dates where formatted using datetime.isoformat(); obviously that is
> not the correct format. What should be the date format? What is this Z
> meant to mean?

The "Z" stands for "Zulu" (i.e. UTC). Thinking about it some more, and
doing some research, I've found out two things:

 1. The date should be formatted in the form*
[-]CCYY-MM-DDThh:mm:ss[Z|(+|-)hh:mm]
 2. I think the SF servers are in pacific time, so the dates should
end "Z -08:00"

I'll re-do the import locally with this format to check that Launchpad
handles it correctly.

* See http://books.xmlschemata.org/relaxng/ch19-77049.html for details
of the xsd:dateTime format.

Graham Binns (gmb) said : #13

The import to staging is now complete (sorry it took longer than expected). You should be able to view the bugs at https://bugs.staging.launchpad.net/widelands (though you'll need to set Widelands to use launchpad for bug tracking before you'll be able to see them).

Please take a look and let me know if the import looks okay. There's a chance the import will get clobbered because the staging DB gets refreshed from production on a daily basis. If that happens we'll re-run the import on Monday morning for you.

SirVer (sirver) said : #14

Graham,

thanks for your work so far. I am currently browsing the bugs and told my team to do so as well. The first issue that I came across was that attachements weren't included. See:

https://bugs.staging.launchpad.net/widelands/+bug/529476

and the original:
http://sourceforge.net/tracker/index.php?func=detail&aid=2928937&group_id=40163&atid=427221

filename and file size seems to be correct, but the links are broken.

SirVer (sirver) said : #15

Graham,

the only issue we had with the bug imports are the missing attachments. I thought the problem might come from the prettyprinting of the xml and therefore removed all prettyprintings. I also exported the current state of the bug tracker as it is on sf.net. You can find the current file under

http://widelands.org/~sirver/wl_launchpadxmlimport.xml

Please look into the attachment issue and try to import the new file.

One more question: What will happen to the bugs we have already filed on launchpad, will they get lost when the import is done?

Cheers,
Holger

Graham Binns (gmb) said : #16

On 8 March 2010 08:43, SirVer <email address hidden> wrote:
> Question #100549 on Launchpad Bugs changed:
> https://answers.edge.launchpad.net/malone/+question/100549
>
> SirVer gave more information on the question:
> Graham,
>
> the only issue we had with the bug imports are the missing attachments.
> I thought the problem might come from the prettyprinting of the xml and
> therefore removed all prettyprintings. I also exported the current state
> of the bug tracker as it is on sf.net. You can find the current file
> under
>
> http://widelands.org/~sirver/wl_launchpadxmlimport.xml

Thanks. I'll re-do the import locally and let you know how it goes
before getting it re-imported on staging.

>
> Please look into the attachment issue and try to import the new file.
>
> One more question: What will happen to the bugs we have already filed on
> launchpad, will they get lost when the import is done?
>

No, those bugs will stay as they are and won't be overwritten (we
never delete or overwrite bugs except in exceptional circumstances).

--
Graham Binns | PGP Key: EC66FA7D

Graham Binns (gmb) said : #17

The re-import to staging has been completed. Hopefully it won't get
stomped on by the staging restore until you have chance to take a
look.

It looks as though your work on the attachments has paid off; I've
only checked a few, though, so if you see any problems please let me
know.

Cheers,

Graham

SirVer (sirver) said : #18

Graham,

everything looks fine at this point. I updated the file under

http://widelands.org/~sirver/wl_launchpadxmlimport.xml

to the latest export from sf.net. Please use this for import. From our side we can proceed to import into the production site.

Thanks for all your help!

Cheers,
Holger

Graham Binns (gmb) said : #19

On 10 March 2010 09:43, SirVer <email address hidden> wrote:
> Question #100549 on Launchpad Bugs changed:
> https://answers.launchpad.net/malone/+question/100549
>
>    Status: Answered => Open
>
> SirVer is still having a problem:
> Graham,
>
> everything looks fine at this point. I updated the file under
>
> http://widelands.org/~sirver/wl_launchpadxmlimport.xml
>
> to the latest export from sf.net. Please use this for import. From our
> side we can proceed to import into the production site.
>

Brilliant. I'll get that done for you today.

--
Graham Binns | PGP Key: EC66FA7D

Best Graham Binns (gmb) said : #20

The import to production is now complete. Please let me know if you
spot any problems with it.

Graham Binns (gmb) said : #21

Marking solved as there are no apparent problems with the import to production.

SirVer (sirver) said : #22

Oh yes, sorry Graham. I forget about this question, I wanted to wait a while if any problems surface, but all seems fine. Thanks for the support, this all went as smoothly as one could hope for!

SirVer (sirver) said : #23

Thanks Graham Binns, that solved my question.