Launchpad itself

Reducing traffic to people.samba.org

Asked by Jelmer Vernooij on 2007-06-17

I have a lot of branches hosted on people.samba.org, most registered on launchpad.

Launchpad tries to poll the status of these branches regularly, resulting in a lot of traffic (over 200k HTTP requests on a daily basis). Would it be possible to increase the interval at which launchpad polls the status or disable polling for people.samba.org temporarily ?

Question information

Language:: English Edit question

Status:: Answered

For:: Launchpad itself Edit question

Assignee:: No assignee Edit question

Last query:: 2007-06-19

Last reply:: 2007-06-20

Link existing bug

Jelmer Vernooij (jelmer) said on 2007-06-17:

The amount of traffic doesn't appear to be a big deal at the moment, though it would be nice to get it down (especially as I register more branches...). The number of HTTP requests is much higher than it should be.

My sniffer tells me bzr does 8 HTTP requests for a no-op pull. spiv mentions branches are polled every 6 hours. That would suggest I have:

200k / (24 / 6) / 8 = 6250 branches

there may be stale locks around - maybe the mirror code is spinning checking for the locks to be released?

Robert Collins (lifeless) said on 2007-06-18:

read only requests don't look at locks. I'd expect to see the following for a no-op pull:
branch-format
branch/format
repository/format
branch/revision-history (format 5 branches)
or the format 6 equivalent

If a branch fails I think it gets tried again more often.

Can you get a log of the requests for your branches to analyse?

Robert Collins (lifeless) said on 2007-06-18:

the ui confused me, this is not answered.

Jelmer Vernooij (jelmer) said on 2007-06-19:

I've uploaded the relevant part of the logs for june 16 on http://samba.org/~jelmer/vostok_access.log.gz. It does seem to be hitting an awful lot of kndx files.

Jelmer Vernooij (jelmer) said on 2007-06-19:

Also, note that this is just hitting one of the repositories on that server. There are dozens of other branches registered that use different repositories.

Robert Collins (lifeless) said on 2007-06-20:

Does that branch contain a broken unicode xml as you had previously? That would cause the mirror to fail, and thus its actually doing a full copy; its not waiting 24 hours because it hasn't *finished* a mirror.

Robert Collins (lifeless) said on 2007-06-20:

82.211.81.156 - - [16/Jun/2007:00:05:07 -0500] "GET /bzr/jelmer/samba/.bzr/repository/knits/34/svn-v1%253a6%400c0555d6-39d7-0310-84fc-f1cc0bd64818-branches%25252f%2553%2541%254d%2542%2541_4_0-source%25252fsmb_server%25252ftrans2.c.kndx HTTP/1.1" 200 4669 "-" "bzr/0.16.0 (urllib)"
82.211.81.156 - - [16/Jun/2007:00:06:09 -0500] "GET /bzr/jelmer/samba/4.0-signalreplace/.bzr/branch-format HTTP/1.1" 200 35 "-" "bzr/0.16.0 (urllib)"
82.211.81.156 - - [16/Jun/2007:00:06:09 -0500] "GET /bzr/jelmer/samba/4.0-signalreplace/.bzr/branch/format HTTP/1.1" 200 26 "-" "bzr/0.16.0 (urllib)"
82.211.81.156 - - [16/Jun/2007:00:06:09 -0500] "GET /bzr/jelmer/samba/4.0-signalreplace/.bzr/repository/format HTTP/1.1" 404 406 "-" "bzr/0.16.0 (urllib)"
82.211.81.156 - - [16/Jun/2007:00:06:09 -0500] "GET /bzr/jelmer/samba/.bzr/branch-format HTTP/1.1" 200 35 "-" "bzr/0.16.0 (urllib)"
82.211.81.156 - - [16/Jun/2007:00:06:10 -0500] "GET /bzr/jelmer/samba/.bzr/repository/format HTTP/1.1" 200 34 "-" "bzr/0.16.0 (urllib)"
82.211.81.156 - - [16/Jun/2007:00:06:10 -0500] "HEAD /bzr/jelmer/samba/.bzr/repository/shared-storage HTTP/1.1" 200 - "-" "bzr/0.16.0 (urllib)"
82.211.81.156 - - [16/Jun/2007:00:06:10 -0500] "GET /bzr/jelmer/samba/4.0-signalreplace/.bzr/branch/revision-history HTTP/1.1" 200 643021 "-" "bzr/0.16.0 (urllib)"
82.211.81.156 - - [16/Jun/2007:00:06:11 -0500] "GET /bzr/jelmer/samba/.bzr/repository/revisions.kndx HTTP/1.1" 200 5232743 "-" "bzr/0.16.0 (urllib)"