bazaar behind a firewall is so slow

On Wed, Jun 25, 2008 at 11:18 AM, xuekun
<email address hidden> wrote:
> New question #37308 on Bazaar:
> https://answers.launchpad.net/bzr/+question/37308
>
> I am behind a corporate firewall. So I set http_proxy environment. Then I used
> "bzr branch http://bazaar.launchpad.net/~mysql/mysql-server/mysql-6.0
> mysql-6.0" to download mysql 6.0 source code. However it has not been
> finished yet for three days!!! So slow.
>
> I am wondering if it is bazaar problem or my proxy server problem.

That does sound pretty slow unless your network is extremely
high-latency. It would be useful if you could either run with -Dhpss
(case sensitive) and send part of that log, or record a network trace
using eg

  tcpdump -p -w bzr.pcap -s0 tcp port 80

Change 80 to whatever is your proxy server port.

This will include any other http traffic so you might want to avoid
including any sensitive traffic. Let that run for a while. Then
please attach the capture to this ticket.

--
Martin <http://launchpad.net/~mbp/>

Thanks. I used -Dhpss and below was the log.

From the log, I found some things.
1. "failed to import pycurl". Is pycurl necessary needed?
2. The time stamp looks slow, but should be ok. The problem maybe is after some times, time out happened, then quit.
3. Then I retry. And bzr seems to retrieve from the repository begining, instead of from the breaking point. So I wonder if bzr support "resume broken download"?

0.043 encoding stdout as sys.stdout encoding 'UTF-8'
0.043 bzr arguments: [u'branch', u'http://bazaar.launchpad.net/~mysql/mysql-server/mysql-6.0-falcon', u'mysql-6.0-falcon']
0.043 looking for plugins in /root/.bazaar/plugins
0.043 looking for plugins in /usr/lib64/python2.4/site-packages/bzrlib/plugins
0.043 Plugin name __init__ already loaded
0.043 Plugin name __init__ already loaded
0.046 looking for plugins in /usr/lib/python2.4/site-packages/bzrlib/plugins
0.053 encoding stdout as sys.stdout encoding 'UTF-8'
0.072 failed to import pycurl: No module named pycurl
0.072 failed to instantiate transport <bzrlib.registry._LazyObjectGetter object at 2aaaaf5da460, module='bzrlib.transport.http._pycurl' attribute='PyCurlTransport'> for 'http://bazaar.launchpad.net/~mysql/mysql-server/mysql-6.0-falcon': <bzrlib.errors.DependencyNotPresent instance at 0x2aaab1060bd8>
96.203 Using fetch logic to copy between KnitPackRepository('http://bazaar.launchpad.net/%7Emysql/mysql-server/mysql-6.0-falcon/.bzr/repository/')(<RepositoryFormatKnitPack1>) and KnitPackRepository('file:///root/mysql/mysql_auto_test/shared/.bzr/repository/')(<RepositoryFormatKnitPack1>)
96.204 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 2 collapsed 2
1666.489 http readv of bdc6e0637a4fb2dbb350b797930286fa.rix offsets => 1 collapsed 1
1694.377 http readv of 4811c954b92388f57efcfed02c3cc9f1.rix offsets => 1 collapsed 1
1696.081 http readv of 35e689f261296975394e2c888766593a.rix offsets => 1 collapsed 1
1697.793 http readv of 2b47f16144cbb06b61a0dc47403502b8.rix offsets => 1 collapsed 1
1718.946 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 1 collapsed 1
1725.727 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 1 collapsed 1
1764.035 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 1 collapsed 1
1801.148 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 1 collapsed 1
1880.286 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 1 collapsed 1
1933.278 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 1 collapsed 1
2007.933 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 1 collapsed 1
2024.159 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 1 collapsed 1
2068.556 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 2 collapsed 2
2178.563 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 2 collapsed 2
2278.438 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 1 collapsed 1
2320.014 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 4 collapsed 4
2475.164 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 1 collapsed 1
2523.279 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 1 collapsed 1
2591.572 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 2 collapsed 2
2726.370 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 2 collapsed 2
2796.388 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 2 collapsed 2
2875.238 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 4 collapsed 4
3079.222 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 1 collapsed 1
3130.691 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 2 collapsed 2
3211.115 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 1 collapsed 1
3245.180 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 1 collapsed 1
3315.571 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 1 collapsed 1
3369.849 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 1 collapsed 1
3440.412 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 2 collapsed 2
3533.140 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 1 collapsed 1
3637.010 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 1 collapsed 1
3639.674 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 3 collapsed 3
3769.350 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 3 collapsed 3
3910.958 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 2 collapsed 2
4004.701 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 4 collapsed 4
4162.795 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 3 collapsed 3
4344.323 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 2 collapsed 2
4429.438 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 4 collapsed 4
4576.718 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 1 collapsed 1
4618.725 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 2 collapsed 2
4696.807 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 1 collapsed 1
4838.298 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 2 collapsed 2
4858.370 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 2 collapsed 2
4937.672 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 1 collapsed 1
4997.801 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 1 collapsed 1
5053.498 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 2 collapsed 2
5152.279 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 1 collapsed 1
5196.432 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 1 collapsed 1
5244.908 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 2 collapsed 2
5302.949 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 1 collapsed 1
5350.593 http readv of ffe7ea0245adbf3458c9dfaa067b3857.rix offsets => 1 collapsed 1
6251.494 Traceback (most recent call last):
  File "/usr/lib64/python2.4/site-packages/bzrlib/commands.py", line 846, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib64/python2.4/site-packages/bzrlib/commands.py", line 797, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib64/python2.4/site-packages/bzrlib/commands.py", line 499, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib64/python2.4/site-packages/bzrlib/builtins.py", line 942, in run
    hardlink=hardlink)
  File "/usr/lib64/python2.4/site-packages/bzrlib/bzrdir.py", line 1006, in sprout
    result_repo.fetch(source_repository, revision_id=revision_id)
  File "/usr/lib64/python2.4/site-packages/bzrlib/repository.py", line 956, in fetch
    return inter.fetch(revision_id=revision_id, pb=pb, find_ghosts=find_ghosts)
  File "/usr/lib64/python2.4/site-packages/bzrlib/decorators.py", line 165, in write_locked
    return unbound(self, *args, **kwargs)
  File "/usr/lib64/python2.4/site-packages/bzrlib/repository.py", line 2721, in fetch
    find_ghosts=find_ghosts).get_keys()
  File "/usr/lib64/python2.4/site-packages/bzrlib/decorators.py", line 127, in read_locked
    return unbound(self, *args, **kwargs)
  File "/usr/lib64/python2.4/site-packages/bzrlib/repository.py", line 2747, in search_missing_revision_ids
    return self._walk_to_common_revisions([revision_id])
  File "/usr/lib64/python2.4/site-packages/bzrlib/repository.py", line 2347, in _walk_to_common_revisions
    next_revs, ghosts = searcher.next_with_ghosts()
  File "/usr/lib64/python2.4/site-packages/bzrlib/graph.py", line 1195, in next_with_ghosts
    self._advance()
  File "/usr/lib64/python2.4/site-packages/bzrlib/graph.py", line 1205, in _advance
    found, ghosts, next, parents = self._do_query(self._next_query)
  File "/usr/lib64/python2.4/site-packages/bzrlib/graph.py", line 1228, in _do_query
    parent_map = self._parents_provider.get_parent_map(revisions)
  File "/usr/lib64/python2.4/site-packages/bzrlib/graph.py", line 132, in get_parent_map
    new_parents = self._real_provider.get_parent_map(needed)
  File "/usr/lib64/python2.4/site-packages/bzrlib/repofmt/pack_repo.py", line 1932, in get_parent_map
    for index, key, value, refs in index.iter_entries(search_keys):
  File "/usr/lib64/python2.4/site-packages/bzrlib/index.py", line 1079, in iter_entries
    for node in index.iter_entries(keys):
  File "/usr/lib64/python2.4/site-packages/bzrlib/index.py", line 478, in iter_entries
    return (result[1] for result in bisect_multi_bytes(
  File "/usr/lib64/python2.4/site-packages/bzrlib/bisect_multi.py", line 50, in bisect_multi_bytes
    search_results = content_lookup(search_keys)
  File "/usr/lib64/python2.4/site-packages/bzrlib/index.py", line 680, in _lookup_keys_via_location
    self._read_and_parse(readv_ranges)
  File "/usr/lib64/python2.4/site-packages/bzrlib/index.py", line 962, in _read_and_parse
    for offset, data in readv_data:
  File "/usr/lib64/python2.4/site-packages/bzrlib/transport/http/__init__.py", line 245, in _readv
    for cur_coal, rfile in self._coalesce_readv(relpath, coalesced):
  File "/usr/lib64/python2.4/site-packages/bzrlib/transport/http/__init__.py", line 337, in _coalesce_readv
    for c, rfile in get_and_yield(relpath, ranges):
  File "/usr/lib64/python2.4/site-packages/bzrlib/transport/http/__init__.py", line 300, in get_and_yield
    code, rfile = self._get(relpath, coalesced)
  File "/usr/lib64/python2.4/site-packages/bzrlib/transport/http/_urllib.py", line 120, in _get
    response = self._perform(request)
  File "/usr/lib64/python2.4/site-packages/bzrlib/transport/http/_urllib.py", line 79, in _perform
    response = self._opener.open(request)
  File "/usr/lib64/python2.4/urllib2.py", line 364, in open
    response = meth(req, response)
  File "/usr/lib64/python2.4/site-packages/bzrlib/transport/http/_urllib2_wrappers.py", line 1333, in http_response
   code, msg, hdrs)
  File "/usr/lib64/python2.4/urllib2.py", line 402, in error
    return self._call_chain(*args)
  File "/usr/lib64/python2.4/urllib2.py", line 337, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.4/site-packages/bzrlib/transport/http/_urllib2_wrappers.py", line 1349, in http_error_default
    'Unable to handle http code %d: %s'
InvalidHttpResponse: Invalid http response for http://bazaar.launchpad.net/%7Emysql/mysql-server/mysql-6.0-falcon/.bzr/repository/indices/ffe7ea0245adbf3458c9dfaa067b3857.rix: Unable to handle http code 504: Gateway Time-out

Thanks for the details.

As a workaround, you can try accessing the branch using bzr+ssh://bazaar.launchpad.net/~mysql/mysql-server/mysql-6.0 which may avoid problems with the firewall.

Could you give me more info how to use "bzr+ssh"? Is it possible to
bypass the corporate firewall?

It failed with "bzr branch
bzr+ssh://bazaar.launchpad.net/~mysql/mysql-server/mysql-6.0 -Dhpss".

The error are:
ssh: connect to host bazaar.launchpad.net port 22: Connection timed out
bzr: ERROR: Connection closed: please check connectivity and permissions

I also noticed that "bzr+ssh" doesn't support "~" in URL scheme
(https://bugs.launchpad.net/bzr/+bug/109143).

Thx, Xuekun

Finally I downloaded the files from a US machine. I'm in China, far from Europe. :-)

However I still need to say: Thank you!

BTW. One suggestion is I hope bazaar support such "resume broken download" feature. :-)