Reducing traffic to people.samba.org

Asked by Jelmer Vernooij on 2007-06-17

I have a lot of branches hosted on people.samba.org, most registered on launchpad.

Launchpad tries to poll the status of these branches regularly, resulting in a lot of traffic (over 200k HTTP requests on a daily basis). Would it be possible to increase the interval at which launchpad polls the status or disable polling for people.samba.org temporarily ?

Question information

Language:
English Edit question
Status:
Answered
For:
Launchpad itself Edit question
Assignee:
No assignee Edit question
Last query:
2007-06-19
Last reply:
2007-06-20
Jelmer Vernooij (jelmer) said : #1

The amount of traffic doesn't appear to be a big deal at the moment, though it would be nice to get it down (especially as I register more branches...). The number of HTTP requests is much higher than it should be.

My sniffer tells me bzr does 8 HTTP requests for a no-op pull. spiv mentions branches are polled every 6 hours. That would suggest I have:

200k / (24 / 6) / 8 = 6250 branches

there may be stale locks around - maybe the mirror code is spinning checking for the locks to be released?

Robert Collins (lifeless) said : #2

read only requests don't look at locks. I'd expect to see the following for a no-op pull:
branch-format
branch/format
repository/format
branch/revision-history (format 5 branches)
or the format 6 equivalent

If a branch fails I think it gets tried again more often.

Can you get a log of the requests for your branches to analyse?

Robert Collins (lifeless) said : #3

the ui confused me, this is not answered.

Jelmer Vernooij (jelmer) said : #4

I've uploaded the relevant part of the logs for june 16 on http://samba.org/~jelmer/vostok_access.log.gz. It does seem to be hitting an awful lot of kndx files.

Jelmer Vernooij (jelmer) said : #5

Also, note that this is just hitting one of the repositories on that server. There are dozens of other branches registered that use different repositories.

Robert Collins (lifeless) said : #6

Does that branch contain a broken unicode xml as you had previously? That would cause the mirror to fail, and thus its actually doing a full copy; its not waiting 24 hours because it hasn't *finished* a mirror.

Robert Collins (lifeless) said : #7

82.211.81.156 - - [16/Jun/2007:00:05:07 -0500] "GET /bzr/jelmer/samba/.bzr/repository/knits/34/svn-v1%253a6%400c0555d6-39d7-0310-84fc-f1cc0bd64818-branches%25252f%2553%2541%254d%2542%2541_4_0-source%25252fsmb_server%25252ftrans2.c.kndx HTTP/1.1" 200 4669 "-" "bzr/0.16.0 (urllib)"
82.211.81.156 - - [16/Jun/2007:00:06:09 -0500] "GET /bzr/jelmer/samba/4.0-signalreplace/.bzr/branch-format HTTP/1.1" 200 35 "-" "bzr/0.16.0 (urllib)"
82.211.81.156 - - [16/Jun/2007:00:06:09 -0500] "GET /bzr/jelmer/samba/4.0-signalreplace/.bzr/branch/format HTTP/1.1" 200 26 "-" "bzr/0.16.0 (urllib)"
82.211.81.156 - - [16/Jun/2007:00:06:09 -0500] "GET /bzr/jelmer/samba/4.0-signalreplace/.bzr/repository/format HTTP/1.1" 404 406 "-" "bzr/0.16.0 (urllib)"
82.211.81.156 - - [16/Jun/2007:00:06:09 -0500] "GET /bzr/jelmer/samba/.bzr/branch-format HTTP/1.1" 200 35 "-" "bzr/0.16.0 (urllib)"
82.211.81.156 - - [16/Jun/2007:00:06:10 -0500] "GET /bzr/jelmer/samba/.bzr/repository/format HTTP/1.1" 200 34 "-" "bzr/0.16.0 (urllib)"
82.211.81.156 - - [16/Jun/2007:00:06:10 -0500] "HEAD /bzr/jelmer/samba/.bzr/repository/shared-storage HTTP/1.1" 200 - "-" "bzr/0.16.0 (urllib)"
82.211.81.156 - - [16/Jun/2007:00:06:10 -0500] "GET /bzr/jelmer/samba/4.0-signalreplace/.bzr/branch/revision-history HTTP/1.1" 200 643021 "-" "bzr/0.16.0 (urllib)"
82.211.81.156 - - [16/Jun/2007:00:06:11 -0500] "GET /bzr/jelmer/samba/.bzr/repository/revisions.kndx HTTP/1.1" 200 5232743 "-" "bzr/0.16.0 (urllib)"

I think the timings in the above support my hypothesis.

Robert Collins (lifeless) said : #8

Our puller is getting this error:
RevisionNotPresent: Revision {svn-v1:2044@0c0555d6-39d7-0310-84fc-f1cc0bd64818-branches%2fSAMBA_4_0} not present in KnitVersionedFile(http://people.samba.org/bzr/jelmer/samba/.bzr/repository/knits/34/svn-v1%253a6%400c0555d6-39d7-0310-84fc-f1cc0bd64818-branches%25252f%2553%2541%254d%2542%2541_4_0-source%25252fsmb_server%25252ftrans2.c).

The frequency is high as we presume errors are transient things we should quickly catch when they get corrected.

Can you help with this problem?

Provide an answer of your own, or ask Jelmer Vernooij for more information if necessary.

To post a message you must log in.