stuck git import for a few days

Hi Peter,

I see that this code import was tried a few times, each time the same error appears for different code-import workers, and those workers are working OK for other projects. So this seems weirdly very specific for this particular import, I'm not 100% sure how we can help currently, or if this is indeed an issue from Launchpad or not.

It was working on the 2nd of October (this Monday), then it started failing. I'm guessing the launchpad configuration for the code import has remained the same?

I also noticed that there was exactly 1 commit between things working vs. not working, though I don't see how that could be the root of the issue.

I'll ask if anyone else in the team has any other ideas

Ines,

indeed the launchpad configuration hasn't changed.

FTR. this is the MR where things stopped working: https://review.opendev.org/c/openstack/charm-ceph-mon/+/897011

It's a one line change of a parameter, can't see how this could affect things.

Small update here, I tried doing the same code import in our qastaging environment and it succeeded: https://code.qastaging.launchpad.net/~ines-almeida/test-project-ines/+git/test-project-ines

This is more proof that this is an issue within production (given the configuration didn't change, then it shouldn't be due to the configuration in production either)

@Ines thanks for the update

@Juergen, I see the import was retried and failed at 2023-10-09 06:52:34
http://launchpadlibrarian.net/691047798/openstack-charmers-charm-ceph-mon-+git-charm-ceph-mon.log

The import seems to be running into a deadlock between "git-remote-https" and "git fetch-pack". It looks quite like https://github.com/git/git/commit/b37fd14beb39b9f545bd72e42e1bdbb00bad4b3d, and I wonder if we should try cherry-picking that patch into the git backport that we're running.

Interesting!

From the linked commit it says this happens "when the server side prematurely throws an error and disconnects", does this mean we're running into errors when fetching from upstream?

Fwiw I'm of course very much +1 on trying that cherry-pick if it helps us get unstuck.

Thanks!

I didn't see any such errors, but it's always possible that upstream's analysis of the exact set of situations that can result in this bug is incomplete. (Alternatively, my educated guess could be wrong.)

update
see https://chat.canonical.com/canonical/pl/f8ftxxyitpdapekgdj1sjbfwmo (Canonical internal)

I backported what looked like a relevant fix upstream to the bionic package that we are using in the code import workers. But that didn't solve the problem and the code import is still timing out. Colin from my team has suggested backporting the jammy version of git to bionic (if at all that is possible) to see if that resolves this issue. So fixing this is going to take more time.

Hey @Guruprasad, thanks for the update (and bummer about that fix).

Just for avoidance of doubt, is the git backport jammy->bionic something you're considering implementing?

Cheers!

Hi Peter, yes, I am going to try backporting the jammy version of git to bionic, if at all that is possible.

Any thoughts on deleting the repository and re-importing it? Is that a potential option that might get everything by?

Billy, no, that won't help given that we were able to reproduce the failure in qastaging.

I backported jammy's git package to bionic, upgraded the qastaging codeimport workers to use it, and retried the import of the same repository (https://code.qastaging.launchpad.net/~ines-almeida/test-project-ines/+git/test-project-ines) on the qastaging instance. After multiple failures similar to what we saw in the production environment before today's upgrade, the import succeeded on the first try. Let us keep it running for a few more iterations to confirm that this is not a random success and then upgrade the production codeimport workers to use this version.

> After multiple failures similar to what we saw in the production environment before today's upgrade, the import succeeded on the first try.

that's great news, thanks for working on this issue, Guruprasad.

> Let us keep it running for a few more iterations to confirm that this is not a random success and then upgrade the production codeimport workers to use this version.

do you think before end of this week we could get this roll over to production?

Fantastic, thanks a ton Guruprasad!

Thanks Guruprasad, that solved my question.