use persistent svn_ra sessions and reconnect on failure

Bug #120992 reported by Michael Hudson-Doyle
50
This bug affects 5 people
Affects Status Importance Assigned to Milestone
Launchpad CSCVS
Won't Fix
High
Unassigned

Bug Description

Making the initial import of a large repository requires something in the region of 10K connections to the server to all go through successfully. A server doesn't have to be all that flaky to drop just one of these connections, and if it's towards the end, we currently (to the best of my knowledge) have to start over again. If instead we could restart the import at that 95% point (which certainly ought to be possible), we'd be much more likely to be able to complete the import.

I'm looking at you, google code hosting.

Tags: svn
Revision history for this message
David Allouche (ddaa) wrote :

The svn import code of cscvs should:

 * Use the svn_ra API instead of the svn_client API, to use persistent sessions for multiple queries and avoid creating that silly number of connections.
 * Be able to create a new session, with increasing retry delays if the server has a hiccup and the session errors out, just as its native cvs pserver client does.

That should adequately address the issue with flaky svn servers, and would also make svn imports much faster and put less load on the community servers we import from.

Handling flaky servers by allowing importd to restart failed imports would probably be less programming work, but it would require more handiwork and I do not think it would be a Good Thing.

I suggest changing the title of this bug to "use persistent svn_ra sessions and reconnect on failure" and retargetting to launchpad-cscvs.

David Allouche (ddaa)
Changed in launchpad-bazaar:
importance: Undecided → High
status: New → Confirmed
Revision history for this message
Rolf Leggewie (r0lf) wrote :

are we going to see an improvement here soon?

Revision history for this message
Pau Arumi (parumi) wrote :

I second that fixing this is very needed.

The import of our (community) project persistently failed during a week after 26-28 hours downloading:
https://code.launchpad.net/~vcs-imports/clam/trunk

The story behind my interest is that this bug contributed to our servers failure and our sysadmin understandably got a bit angry with us.
Moreover, the user is not allowed to stop the automatic retries. Asking to stop the imports in the "questions" didn't either work, which made the problem continue several more days (till found help at #launchpad).

Thanks!

Paul Hummer (rockstar)
Changed in launchpad-cscvs:
assignee: nobody → rockstar
Revision history for this message
Joey Stanford (joey) wrote :

Hi Paul,

Do we have word yet when this will be scheduled? I know you've looked into this recently and have an idea on how to make this change. The code is a bit complex so that's not easy task. :-)

Thanks,

Joey

Revision history for this message
Paul Hummer (rockstar) wrote : Re: [Bug 120992] Re: use persistent svn_ra sessions and reconnect on failure

> Do we have word yet when this will be scheduled?

We've talked about this, and expect it to land in 2.1.10

--
Paul Hummer
Launchpad Code Team

Paul Hummer (rockstar)
Changed in launchpad-cscvs:
status: Confirmed → In Progress
Revision history for this message
Paul Hummer (rockstar) wrote :

I've linked this bug to a branch that, while not immediately providing a fix, lays a groundwork to stop using the client interface when not needed. This will reduce memory consumption and simplify the amount of concurrent threads making connections.

Revision history for this message
Dominic Evans (oldmanuk) wrote :

I presume this is the cause of failures on initial import:
PROPFIND of '/svnroot/zquake/trunk/zquake/r_main.c': SSL negotiation failed: Connection reset by peer (https://zquake.svn.sourceforge.net)
Import failed:

[SRC] http://launchpadlibrarian.net/19837698/zquake-trunk-log.txt

a shame that once it has imported several changesets, it then rolls back after a single failure.

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

Yes to all the above.

It's annoying that sf seem to have joined the ranks of unreliable servers, they used to be rock solid.

Paul Hummer (rockstar)
Changed in launchpad-cscvs:
status: In Progress → Triaged
Paul Hummer (rockstar)
Changed in launchpad-cscvs:
assignee: rockstar → nobody
Revision history for this message
Paul Hummer (rockstar) wrote :

A branch has been reviewed and will be landed soon that will start the process of fixing this. While the review was being done, it was discovered that this error has to do with pysvn.client's cat method, which will be swapped out for the libsvn equivalent in another branch.

I just wanted to update all subscribers that this bug isn't being ignored.

Revision history for this message
Dominic Evans (oldmanuk) wrote :

Hey Paul, any update on when the fix for this will get pushed?

Two new vcs-imports from sourceforge both failing today

https://code.launchpad.net/~vcs-imports/recordmydesktop/upstream
https://code.launchpad.net/~vcs-imports/gtk-recordmydesktop/upstream

tags: added: svn
Revision history for this message
Robert Collins (lifeless) wrote :

For LP itself, we're not going to fix this: we're using bzr-svn instead for imports from svn.

Changed in launchpad-cscvs:
status: Triaged → Won't Fix
Revision history for this message
Christian Reis (kiko) wrote :

And does bzr-svn address the issues seen in this bug?

Revision history for this message
Robert Collins (lifeless) wrote :

@kiko its a very different stack making very different calls; we have very reliable imports with it. If a similar defect exists, its still a different defect.

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

Yes.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.