Requesting import of bug history into IVLE project

Asked by David Coles on 2009-11-25

We're in the final stages of migrating our project away from Google Code, and one of the last things left to do is move the old bug tracker issues across. Thankfully all new bugs are being tracked on Launchpad now.

I've gone through and scraped our old bug tracker and produced an XML with the Launchpad Import Format. It includes full names and email addresses of the users involved and the Launchpad usernames where known. I've run it through a RELAX NG validator, so it should be syntactically correct.

Could we have the attached bug history imported into our project. Hopefully this won't cause any issues with existing bugs.

Thanks!
David

Question information

Language:
English Edit question
Status:
Solved
For:
Launchpad itself Edit question
Assignee:
Graham Binns Edit question
Solved by:
Graham Binns
Solved:
2009-12-05
Last query:
2009-12-05
Last reply:
2009-12-03
Whiteboard:
al-maisan; Thu, 26 Nov 2009 15:10:01 +0100 assigned to gmb
David Coles (dcoles) said : #1
Best Graham Binns (gmb) said : #2

Hi David,

Here's how the process usually works for imports:

 1. We do a test import locally to make sure that the data look sane.
 2. We do the import on a temporary Launchpad instance hosted on Amazon EC2 (we occasionally use staging for this, but since it gets reset every 24 hours that's usually a bad idea). This is so that you can check the result of the import.
 3. Once you're happy with the import we'll do it on staging to check that there are no collisions with production data
 4. Once the staging import checks out we perform it on production.

I'll crack on with steps 1 and (hopefully) 2 today.

David Coles (dcoles) said : #3

Hi Graham,

That sounds like a great plan. Thanks for looking after this for us.
If they're any issues with the data let me know and I'll see about fixing it up as soon as possible.

Cheers,
David

Graham Binns (gmb) said : #4

I've done the import locally and I'm repeatedly running into a problem with Launchpad telling me that the first bug has already been imported, which it hasn't, but at the moment I'm unclear as to why the error's occurring (i.e. it could be the data or it could be the import script).

I'm not going to be able to take a close look at the problem until tomorrow as I'm currently working on some release-critical fixes for the next release of LP, which happens on Wednesday. I'll pick this up tomorrow and let you know if I find any more problems.

William Grant (wgrant) said : #5

The import runs fine for me, except for the space after all instances of Steven's email address. David, can you correct that?

William Grant (wgrant) said : #6

Also, I'd prefer it if you mapped to Triaged rather than Confirmed.

David Coles (dcoles) said : #7

Shall do. I've also fixed it up so we skip my "test" bug, ignore "no comment for this change" comments, do a better mapping of milestones, don't double escape comment text and fixed up Steven's email.

Re-uploaded to the same location.

Graham Binns (gmb) said : #8

2009/12/3 David Coles <email address hidden>:
> Question #91604 on Launchpad itself changed:
> https://answers.edge.launchpad.net/launchpad/+question/91604
>
> David Coles posted a new comment:
> Shall do. I've also fixed it up so we skip my "test" bug, ignore "no
> comment for this change" comments, do a better mapping of milestones,
> don't double escape comment text and fixed up Steven's email.
>
> Re-uploaded to the same location.
>

Thanks David, and thanks William for taking a crack at this whilst I
had my hands full with release-critical gubbins.

Turns out that I encounter the "imported before" error on one machine
only (goodness knows why), so I'll re-do the local import with the new
version of the data today and hopefully be able to get an import up on
an EC2 instance this afternoon.

Graham Binns (gmb) said : #9

So, as William pointed out in IRC, this is looking good locally, and since David has already seen it running on an LP instance there's no need to go to EC2 with this. We'll proceed straight to staging and hopefully have it up and running there some time early this afternoon.

David Coles (dcoles) said : #10

Excellent. I'm happy with that. Once it's uploaded William and I will have a final check through, but otherwise it's looking really good so far. Much nicer than digging through Google Code. *grin*

Graham Binns (gmb) said : #11

Import to staging complete. Everything went well. If you've time, feel free to give it one last check over. We'll do the production import tomorrow (we're a bit short of sysadmins today and they've got lots to deal with already).

David Coles (dcoles) said : #12

Yup. Everything looks fine to me. I'm happy for it to be imported to production.

Graham Binns (gmb) said : #13

The import to production is complete. Please let me know if there are any problems.

David Coles (dcoles) said : #14

Thanks Graham Binns, that solved my question.

David Coles (dcoles) said : #15

Urgh. Cheesy autoreply.

But yes. Thanks for the help. It's looking pretty good so far and had no problem working with the new bugs. Even nicely merged non-lp users. Excellent.

I'll have to clean up the Google Code export script I wrote and chuck it somewhere. Certainly a lot better than doing it by hand.

Graham Binns (gmb) said : #16

2009/12/5 David Coles <email address hidden>:
> I'll have to clean up the Google Code export script I wrote and chuck it
> somewhere. Certainly a lot better than doing it by hand.

It would be really great if you could do that and then put it on LP
under an open source license (we generally use GPLv3 for such things).
We've already got an exporter for Trac so it would be nice to have one
for Google Code, too. Maybe I'll even get around to cleaning up the
very crufty Bugzilla exporter and the very broken Sourceforge one,
too, some day.

Anyway, glad to be of help.

Graham

David Coles (dcoles) said : #17

I've placed the code we used to export the bug history from Google Code at https://code.launchpad.net/~dcoles/+junk/gc-bug-exporter - probably not worth creating a full project for it. It's licenced under the GPLv3 so should be fine if anyone wants to hack on it. Hopefully it'll come in use for some other people.

Cheers,
David

Graham Binns (gmb) said : #18

2009/12/5 David Coles <email address hidden>:
> Question #91604 on Launchpad itself changed:
> https://answers.launchpad.net/launchpad/+question/91604
>
> David Coles posted a new comment:
> I've placed the code we used to export the bug history from Google Code
> at https://code.launchpad.net/~dcoles/+junk/gc-bug-exporter - probably
> not worth creating a full project for it. It's licenced under the GPLv3
> so should be fine if anyone wants to hack on it. Hopefully it'll come in
> use for some other people.
>

That's brilliant, thank you!

For the sake of tracking bugs and questions and so on I've created
https://edge.launchpad.net/google-launchpad-migrator, which puts it on
a level with the Trac Launchpad Migrator.

Again, thanks very much for doing this work. I'm sure it will be very
useful to people wanting to migrate to Launchpad from Google Code in
future.

Cheers,

Graham

David Coles (dcoles) said : #19

On Mon, 2009-12-07 at 10:57 +0000, Graham Binns wrote:
> Your question #91604 on Launchpad itself changed:
> That's brilliant, thank you!
>
> For the sake of tracking bugs and questions and so on I've created
> https://edge.launchpad.net/google-launchpad-migrator, which puts it on
> a level with the Trac Launchpad Migrator.
>
> Again, thanks very much for doing this work. I'm sure it will be very
> useful to people wanting to migrate to Launchpad from Google Code in
> future.
>
> Cheers,
>
> Graham

Excellent! I wasn't quite sure if it was big enough to be a full project
(though I've already made the mistake of not using Bazaar on it until I
finished!) I'll be sure to keep an eye on it then!

Cheers,
David

David Coles (dcoles) said : #20

Hi Graham,

Just a quick request. The branch that I uploaded still had the personal details of some of our developers (private email addresses) who would rather that not be included as an example. I've dummied up some example configuration and recreated the branch: lp:~dcoles/google-launchpad-migrator/gc-bug-exporter

Would it be possible to replace the contents of lp:google-launchpad-migrator with this new version.

Thanks,
David

Graham Binns (gmb) said : #21

Oh, hurrah, it looks like I've just managed to send an empty message to this question by email. Apologies for any ensuing spam...

> Would it be possible to replace the contents of lp:google-launchpad-migrator with this new version.

Certainly. I've deleted the old branch and pushed the new one. Thanks for fixing the problem.

Graham Binns (gmb) said : #22

2009/12/7 David Coles <email address hidden>:
> migrator/gc-bug-exporter

--
Graham Binns | PGP Key: EC66FA7D

David Coles (dcoles) said : #23

On Tue, 2009-12-08 at 07:44 +0000, Graham Binns wrote:
> Your question #91604 on Launchpad itself changed:
> https://answers.edge.launchpad.net/launchpad/+question/91604
>
> Graham Binns posted a new comment:
> Oh, hurrah, it looks like I've just managed to send an empty message to
> this question by email. Apologies for any ensuing spam...

No worries. I tend to do that a lot with Evolution myself.

> > Would it be possible to replace the contents of lp:google-launchpad-
> migrator with this new version.
>
> Certainly. I've deleted the old branch and pushed the new one. Thanks
> for fixing the problem.

Cheers. That's great. Thanks again!