Launchpad itself

bzr fails, perhaps account is locked

Asked by Alexey Bychko

since yesterday we're not able to use our dev bot account for bazaar. we can't do "bzr branch" in our CI system, manual repo fetch doesn't work also.

all info on this is short failure message:

-bash-4.2$ bzr branch lp:~sergei.glushchenko/percona-xtrabackup/2.1-ST43302-xb-bug1340717
ssh: connect to host bazaar.launchpad.net port 22: Connection refused
ConnectionReset reading response for 'BzrDir.open_2.1', retrying
ssh: connect to host bazaar.launchpad.net port 22: Connection refused
bzr: ERROR: Connection closed: Unexpected end of message. Please check connectivity and permissions, and report a bug if problems persist.

but it works perfectly in the same time on the same servers with other accounts

Question information

Language:
English Edit question
Status:
Answered
For:
Launchpad itself Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:

This question was originally filed as bug #1358253.

Revision history for this message
William Grant (wgrant) said : 		#1

Yes, it looks like ~lp-dev-merge-bot (is that your Jenkins?) had 50 connections open (the maximum we allow) each using several gigabytes of RAM. Can you convince it to not make so many simultaneous connections, or work out what it's doing to use so much RAM?

Revision history for this message
Alexey Bychko (abychko) said : 		#2

thank you for your response.
our Jenkins is a large system - you can take a look at http://jenkins.percona.com and scroll down to see the tasks.
we're still trying to use single connection for our matrix jobs, but unfortunately last time it broke our build process and was rolled back. at the moment we're preparing some releases, so ability to build is important for us. we can continue our Jenkins adjustments
after all urgent work.

thank you

Revision history for this message
William Grant (wgrant) said : 		#3

I've unblocked you at the firewall again, but with a lower concurrent connection limit for now (20 rather than 50). Let's see if that causes any problems.

Revision history for this message
Alexey Bychko (abychko) said : 		#4

thank you very much

Revision history for this message
Alexey Bychko (abychko) said : 		#5

just a question. may it be a cause if we have periodic poll for changes in many tasks?

Revision history for this message
William Grant (wgrant) said : 		#6

I don't know how your Jenkins bzr integration works. Normal clients just connect for a second or two every few minutes to check the latest revision, which isn't a problem. But I recall that an old version of some Jenkins plugin ended up maintaining a persistent connection for every branch it was monitoring, which doesn't work very well.

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) said : 		#7

I am getting

 "bzr: ERROR: Connection error: while sending GET /~percona-core/percona-xtradb-cluster/5.6/.bzr/repository/packs/007b5f9b22468a35e9725368d439f270.pack: [Errno 110] Connection timed out"

now, is this an issue with launchpad or related to this?

Revision history for this message
Roel Van de Paar (roel11) said : 		#8

Hi William,

In regards the RAM, I think this is a bug of bzr I've seen before. Try this simple command:

$ bzr branch lp:percona-server

And I think you will observe a large memory issue if you monitor process memory usage of the /usr/bin/python /usr/bin/bzr process. Offcourse yours is sever side while this is client side, but I expect the issue to be similar.

All we would do from our end is either branch or pull.

If you confirm the same, can this bug be fixed please?

Can you help with this problem?

Provide an answer of your own, or ask Alexey Bychko for more information if necessary.

To post a message you must log in.