Fuel upgrade from 5.1 to 5.1.1 fails

Asked by John

I am in the process of upgrading my fuel master from 5.1 to 5.1.1, however, everytime I run the ./upgrade.sh script to upgrade the upgrade fails with below error. I have seen other similar problems but have found no solution to them. Can someone please help?

Steps I took to upgrade
-Downloaded upgrade script tar from mirantis openstack page
-CD to /var/tmp (plenty of space)
- executed ./upgrade --password *****

current Environment:
Fuel 5.1 installed in VM
54 Node HA cluster which includes 3 controller nodes
my current fuel VM is on proxmox cluster which is part of the 54 node cluster; the rest of the nodes form my openstack environment.

the error below is an excerpt from /var/opt/fuel_upgrade.log. Message is trimmed to save space.

...
2015-11-04 17:28:55 DEBUG 19596 (utils) Symlinking "/etc/supervisord.d/5.1" -> "/etc/supervisord.d/current" [overwrite=1]
2015-11-04 17:28:55 DEBUG 19596 (utils) Removing "/etc/supervisord.d/current"
2015-11-04 17:28:55 INFO 19596 (supervisor_client) Stop all services
2015-11-04 17:29:57 DEBUG 19596 (docker_engine) Stop container: {u'Status': u'Up 3 minutes', u'Created': 1446658009, u'Image': u'fuel/postgres_5.1.1:latest', u'Ports': [{u'IP': u'127.0.0.1', u'Type': u'tcp', u'PublicPort': 5432, u'PrivatePort': 5432}, {u'IP': u'10.20.0.2', u'Type': u'tcp', u'PublicPort': 5432, u'PrivatePort': 5432}], u'Command': u'/usr/local/bin/start.sh', u'Names': [u'/fuel-core-5.1.1-postgres'], u'Id': u'c3db4c880d0eadc36a86ac990c7885a3f59d5b625468525fd754b34f90617462'}
2015-11-04 17:29:57 DEBUG 19596 (docker_engine) Stop container: c3db4c880d0eadc36a86ac990c7885a3f59d5b625468525fd754b34f90617462
2015-11-04 17:30:18 DEBUG 19596 (docker_engine) Stop container: {u'Status': u'Up 3 minutes', u'Created': 1446657968, u'Image': u'fuel/mcollective_5.1.1:latest', u'Ports': [], u'Command': u'/usr/local/bin/start.sh', u'Names': [u'/fuel-core-5.1.1-mcollective'], u'Id': u'd737bd8c161044385f7e68ba2f67c2ce5dfc606d95dd5d03cf40425785f4da5b'}
2015-11-04 17:30:18 DEBUG 19596 (docker_engine) Stop container: d737bd8c161044385f7e68ba2f67c2ce5dfc606d95dd5d03cf40425785f4da5b
2015-11-04 17:30:38 DEBUG 19596 (docker_engine) Stop container: {u'Status': u'Up 4 minutes', u'Created': 1446657941, u'Image': u'fuel/cobbler_5.1.1:latest', u'Ports': [{u'IP': u'0.0.0.0', u'Type': u'tcp', u'PublicPort': 443, u'PrivatePort': 443}, {u'IP': u'127.0.0.1', u'Type': u'udp', u'PublicPort': 53, u'PrivatePort': 53}, {u'IP': u'10.20.0.2', u'Type': u'udp', u'PublicPort': 53, u'PrivatePort': 53}, {u'IP': u'127.0.0.1', u'Type': u'udp', u'PublicPort': 69, u'PrivatePort': 69}, {u'IP': u'10.20.0.2', u'Type': u'udp', u'PublicPort': 69, u'PrivatePort': 69}, {u'IP': u'0.0.0.0', u'Type': u'tcp', u'PublicPort': 80, u'PrivatePort': 80}, {u'Type': u'tcp', u'PublicPort': 53}, {u'Type': u'tcp', u'PublicPort': 67}, {u'Type': u'tcp', u'PublicPort': 69}, {u'Type': u'udp', u'PublicPort': 67}], u'Command': u'/bin/sh -c /usr/local/bin/start.sh', u'Names': [u'/fuel-core-5.1.1-cobbler'], u'Id': u'9ae70c37bb73e079432ffd5228ab54b2f18900196b463b12ab130fb046363250'}
2015-11-04 17:30:38 DEBUG 19596 (docker_engine) Stop container: 9ae70c37bb73e079432ffd5228ab54b2f18900196b463b12ab130fb046363250
2015-11-04 17:30:58 INFO 19596 (supervisor_client) Restart supervisor
2015-11-04 17:30:59 DEBUG 19596 (supervisor_client) List of supervisor processes [{'now': 1446658259, 'group': 'dhcrelay_monitor', 'description': '', 'pid': 26243, 'stderr_logfile': '/var/log/dhcrelay-check-err.log', 'stop': 0, 'statename': 'STARTING', 'start': 1446658258, 'state': 10, 'stdout_logfile': '/var/log/dhcrelay-check.log', 'logfile': '/var/log/dhcrelay-check.log', 'exitstatus': 0, 'spawnerr': '', 'name': 'dhcrelay_monitor'}, {'now': 1446658259, 'group': 'docker-astute', 'description': '', 'pid': 26226, 'stderr_logfile': '/var/log/docker-astuteerr.log', 'stop': 0, 'statename': 'STARTING', 'start': 1446658258, 'state': 10, 'stdout_logfile': '/var/log/docker-astute.log', 'logfile': '/var/log/docker-astute.log', 'exitstatus': 0, 'spawnerr': '', 'name': 'docker-astute'}, {'now': 1446658259, 'group': 'docker-cobbler', 'description': '', 'pid': 26218, 'stderr_logfile': '/var/log/docker-cobbler-err.log', 'stop': 0, 'statename': 'STARTING', 'start': 1446658258, 'state': 10, 'stdout_logfile': '/var/log/docker-cobbler.log', 'logfile': '/var/log/docker-cobbler.log', 'exitstatus': 0, 'spawnerr': '', 'name': 'docker-cobbler'}, {'now': 1446658259, 'group': 'docker-keystone', 'description': '', 'pid': 26224, 'stderr_logfile': '/var/log/docker-keystone-err.log', 'stop': 0, 'statename': 'STARTING', 'start': 1446658258, 'state': 10, 'stdout_logfile': '/var/log/docker-keystone.log', 'logfile': '/var/log/docker-keystone.log', 'exitstatus': 0, 'spawnerr': '', 'name': 'docker-keystone'}, {'now': 1446658259, 'group': 'docker-mcollective', 'description': '', 'pid': 26230, 'stderr_logfile': '/var/log/docker-mcollective-err.log', 'stop': 0, 'statename': 'STARTING', 'start': 1446658258, 'state': 10, 'stdout_logfile': '/var/log/docker-mcollective.log', 'logfile': '/var/log/docker-mcollective.log', 'exitstatus': 0, 'spawnerr': '', 'name': 'docker-mcollective'}, {'now': 1446658259, 'group': 'docker-nailgun', 'description': '', 'pid': 26229, 'stderr_logfile': '/var/log/docker-nailgun-err.log', 'stop': 0, 'statename': 'STARTING', 'start': 1446658258, 'state': 10, 'stdout_logfile': '/var/log/docker-nailgun.log', 'logfile': '/var/log/docker-nailgun.log', 'exitstatus': 0, 'spawnerr': '', 'name': 'docker-nailgun'}, {'now': 1446658259, 'group': 'docker-nginx', 'description': '', 'pid': 26233, 'stderr_logfile': '/var/log/docker-nginx-err.log', 'stop': 0, 'statename': 'STARTING', 'start': 1446658258, 'state': 10, 'stdout_logfile': '/var/log/docker-nginx.log', 'logfile': '/var/log/docker-nginx.log', 'exitstatus': 0, 'spawnerr': '', 'name': 'docker-nginx'}, {'now': 1446658259, 'group': 'docker-ostf', 'description': '', 'pid': 26232, 'stderr_logfile': '/var/log/docker-ostf-err.log', 'stop': 0, 'statename': 'STARTING', 'start': 1446658258, 'state': 10, 'stdout_logfile': '/var/log/docker-ostf.log', 'logfile': '/var/log/docker-ostf.log', 'exitstatus': 0, 'spawnerr': '', 'name': 'docker-ostf'}, {'now': 1446658259, 'group': 'docker-postgres', 'description': '', 'pid': 26219, 'stderr_logfile': '/var/log/docker-postgres-err.log', 'stop': 0, 'statename': 'STARTING', 'start': 1446658258, 'state': 10, 'stdout_logfile': '/var/log/docker-postgres.log', 'logfile': '/var/log/docker-postgres.log', 'exitstatus': 0, 'spawnerr': '', 'name': 'docker-postgres'}, {'now': 1446658259, 'group': 'docker-rabbitmq', 'description': '', 'pid': 26220, 'stderr_logfile': '/var/log/docker-rabbitmq-err.log', 'stop': 0, 'statename': 'STARTING', 'start': 1446658258, 'state': 10, 'stdout_logfile': '/var/log/docker-rabbitmq.log', 'logfile': '/var/log/docker-rabbitmq.log', 'exitstatus': 0, 'spawnerr': '', 'name': 'docker-rabbitmq'}, {'now': 1446658259, 'group': 'docker-rsync', 'description': '', 'pid': 26221, 'stderr_logfile': '/var/log/docker-rsync-err.log', 'stop': 0, 'statename': 'STARTING', 'start': 1446658258, 'state': 10, 'stdout_logfile': '/var/log/docker-rsync.log', 'logfile': '/var/log/docker-rsync.log', 'exitstatus': 0, 'spawnerr': '', 'name': 'docker-rsync'}, {'now': 1446658259, 'group': 'docker-rsyslog', 'description': '', 'pid': 26238, 'stderr_logfile': '/var/log/docker-rsyslog-err.log', 'stop': 0, 'statename': 'STARTING', 'start': 1446658258, 'state': 10, 'stdout_logfile': '/var/log/docker-rsyslog.log', 'logfile': '/var/log/docker-rsyslog.log', 'exitstatus': 0, 'spawnerr': '', 'name': 'docker-rsyslog'}]
2015-11-04 17:30:59 DEBUG 19596 (upgrade) BootstrapUpgrader: rollbacking...
2015-11-04 17:30:59 INFO 19596 (bootstrap) bootstrap upgrader: rollbacking...
2015-11-04 17:30:59 DEBUG 19596 (utils) Removing "/var/www/nailgun/5.1.1_bootstrap"
2015-11-04 17:30:59 DEBUG 19596 (utils) Copying "/var/www/nailgun/5.1_bootstrap" -> "/var/www/nailgun/bootstrap" [overwrite=1 symlinks=1]
2015-11-04 17:30:59 DEBUG 19596 (utils) Removing "/var/www/nailgun/bootstrap"
2015-11-04 17:31:02 INFO 19596 (bootstrap) bootstrap upgrader: rollbacked
2015-11-04 17:31:02 DEBUG 19596 (upgrade) HostSystemUpgrader: rollbacking...
2015-11-04 17:31:02 DEBUG 19596 (utils) Remove file "/etc/yum.repos.d/5.1.1_nailgun.repo"
2015-11-04 17:31:02 ERROR 19596 (upgrade) *** UPGRADE FAILED
2015-11-04 17:31:02 ERROR 19596 (cli) Shell command executed with "125" exit code: lxc-attach --name c3db4c880d0eadc36a86ac990c7885a3f59d5b625468525fd754b34f90617462 -- su postgres -c "psql -f /tmp/upgrade/pg_dump_all.sql postgres"
Traceback (most recent call last):
  File "/var/tmp/upgrade/site-packages/fuel_upgrade/cli.py", line 172, in main
    run_upgrade(parse_args(sys.argv[1:]))
  File "/var/tmp/upgrade/site-packages/fuel_upgrade/cli.py", line 164, in run_upgrade
    upgrade_manager.run()
  File "/var/tmp/upgrade/site-packages/fuel_upgrade/upgrade.py", line 51, in run
    upgrader.upgrade()
  File "/var/tmp/upgrade/site-packages/fuel_upgrade/engines/docker_engine.py", line 94, in upgrade
    self.create_and_start_new_containers()
  File "/var/tmp/upgrade/site-packages/fuel_upgrade/engines/docker_engine.py", line 338, in create_and_start_new_containers
    self.run_after_container_creation_command(container)
  File "/var/tmp/upgrade/site-packages/fuel_upgrade/engines/docker_engine.py", line 358, in run_after_container_creation_command
    '', retries=30, interval=4)
  File "/var/tmp/upgrade/site-packages/fuel_upgrade/engines/docker_engine.py", line 413, in exec_with_retries
    return func()
  File "/var/tmp/upgrade/site-packages/fuel_upgrade/engines/docker_engine.py", line 354, in execute
    self.exec_cmd_in_container(container['container_name'], command)
  File "/var/tmp/upgrade/site-packages/fuel_upgrade/engines/docker_engine.py", line 373, in exec_cmd_in_container
    db_container_id, cmd))
  File "/var/tmp/upgrade/site-packages/fuel_upgrade/utils.py", line 57, in exec_cmd
    _wait_and_check_exit_code(cmd, child)
  File "/var/tmp/upgrade/site-packages/fuel_upgrade/utils.py", line 108, in _wait_and_check_exit_code
    'exit code: {1} '.format(exit_code, cmd))
ExecutedErrorNonZeroExitCode: Shell command executed with "125" exit code: lxc-attach --name c3db4c880d0eadc36a86ac990c7885a3f59d5b625468525fd754b34f90617462 -- su postgres -c "psql -f /tmp/upgrade/pg_dump_all.sql postgres" ******end of file*******

Question information

Language:
English Edit question
Status:
Expired
For:
Fuel for OpenStack Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Fabrizio Soppelsa (fsoppelsa) said :
#1

Hi John,

it seems that the script is failing to operate inside the Postgres container, can you check its status and try to dump the DB with:

`dockerctl check postgres`
`dockerctl shell postgres su postgres -c 'pg_dumpall --clean' > /root/postgres_backup_test.sql`

If postgres is up and running well, I guess we need to file a bug. So please attach the full log.

Best regards,
Fabrizio
Mirantis Team

Revision history for this message
John (john-aguilar) said :
#2

Thanks for the help Fabrizio,

I am cc'ing our developer(Alan)so he is also included on this email history.

I am copying Alan response to your solution and what he have already done and tried. Hoping we
Can move further along on the upgrade process. See below

*********
Regarding the "User 'postgres' does not exist" su error, we found that it was a race condition between the upgrade script running on the host and the puppet module running in the container. The upgrade script was trying to do the postgres dump before puppet had finished installing postgres. We worked around this by manually creating a new container based on the supplied postgres image (which was still loaded in Docker due to the Docker daemon dying during the rollback). We created the postgres container as specified in fuel_upgrade/config.py (including the proper docker volumes, mounted to the 5.1.1 version files on the host, and the correct start command: /usr/local/bin/start.sh). We allowed the container to take its time to fully install postgres, create the necessary roles, etc, then stopped postgres gracefully, and docker "committed", "tagged", and "saved" this image to take the place of the supplied postgres.tar. Our rationale was that this would allow it to finish starting postgres in time for the upgrade script.to continue. This worked.

We noticed lots of calls timing out, so we increased all timeouts in config.py about ten-fold, just to be sure, since Fuel has always run quite slow on our 16 core, 32GB ram fuel-master VM (presumably because of slow disk I/O). All of this allowed the upgrade script to successfully install and configure all 5.1.1 containers.

Our problem now is that the integration "checkers" are repeatedly failing, reporting "Connection refused" while trying to get a token from keystone. We tried manually starting the configured postgres and keystone containers, and were able to successfully get a token from this container via keystone client on fuel master, without issue.

-John
Cc: Alan

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Fabrizio Soppelsa
Sent: Thursday, November 05, 2015 9:09 AM
To: Aguilar, John <email address hidden>
Subject: Re: [Question #273738]: Fuel upgrade from 5.1 to 5.1.1 fails

Your question #273738 on Fuel for OpenStack changed:
https://answers.launchpad.net/fuel/+question/273738

    Status: Open => Answered

Fabrizio Soppelsa proposed the following answer:
Hi John,

it seems that the script is failing to operate inside the Postgres container, can you check its status and try to dump the DB with:

`dockerctl check postgres`
`dockerctl shell postgres su postgres -c 'pg_dumpall --clean' > /root/postgres_backup_test.sql`

If postgres is up and running well, I guess we need to file a bug. So please attach the full log.

Best regards,
Fabrizio
Mirantis Team

--
If this answers your question, please go to the following page to let us know that it is solved:
https://answers.launchpad.net/fuel/+question/273738/+confirm?answer_id=0

If you still need help, you can reply to this email or go to the following page to enter your feedback:
https://answers.launchpad.net/fuel/+question/273738

You received this question notification because you asked the question.

Revision history for this message
Fabrizio Soppelsa (fsoppelsa) said :
#3

Thank you Alan for the report! Well done with the tweaks.

Can you now please clarify what do you mean by "integration checkers"? Are these? https://docs.mirantis.com/fuel/fuel-5.1/user-guide.html#post-deployment-check

Thanks,
Fabrizio

Revision history for this message
Alan Smith (a+smith) said :
#4

Thanks, and It's a different set of health checkers, which are run automatically as part of the upgrade script. Here are the relevant lines from the log at this step, which happens after successful rendering of the supervisor.conf templates for all containers:

2015-10-30 18:40:17 DEBUG 11275 (health_checker) Start ostf checker
2015-10-30 18:40:17 DEBUG 11275 (health_checker) Start rabbitmq checker
2015-10-30 18:40:17 DEBUG 11275 (health_checker) Start cobbler checker
2015-10-30 18:40:17 DEBUG 11275 (health_checker) Start postgres checker
2015-10-30 18:40:17 DEBUG 11275 (health_checker) Start rsync checker
2015-10-30 18:40:17 DEBUG 11275 (health_checker) Start rsyslog checker
2015-10-30 18:40:17 DEBUG 11275 (health_checker) Start mcollective checker
2015-10-30 18:40:17 DEBUG 11275 (health_checker) Start keystone checker
2015-10-30 18:40:17 DEBUG 11275 (health_checker) Start nginx checker
2015-10-30 18:40:17 DEBUG 11275 (health_checker) Start integration_ostf_keystone checker
2015-10-30 18:40:17 DEBUG 11275 (keystone_client) Cannot authenticate in keystone: HTTPConnectionPool(host='10.20.0.2', port=5000): Max retries exceeded with url: /v2.0/tokens (Caused b
y <class 'socket.error'>: [Errno 111] Connection refused)
2015-10-30 18:40:17 DEBUG 11275 (health_checker) Start integration_nginx_nailgun checker
2015-10-30 18:40:17 DEBUG 11275 (health_checker) Start integration_postgres_nailgun_nginx checker
2015-10-30 18:40:17 DEBUG 11275 (keystone_client) Cannot authenticate in keystone: HTTPConnectionPool(host='10.20.0.2', port=5000): Max retries exceeded with url: /v2.0/tokens (Caused by <class 'socket.error'>: [Errno 111] Connection refused)
2015-10-30 18:40:17 DEBUG 11275 (health_checker) Start integration_rabbitmq_astute_nailgun checker
2015-10-30 18:40:17 INFO 11275 (health_checker) Failed checkers: ['integration_postgres_nailgun_nginx', 'integration_ostf_keystone', 'integration_nginx_nailgun', 'ostf', 'nginx', 'keystone', 'integration_rabbitmq_astute_nailgun']

-----

The above messages were repeated every 3 seconds for 15 minutes, until it finally gave up and tried to rollback (at which time the Docker daemon died, but that's a separate issue):

2015-10-30 18:56:18 ERROR 11275 (upgrade) DockerUpgrader: failed to upgrade: "Failed to run services ['integration_postgres_nailgun_nginx', 'integration_ostf_keystone', 'keystone', 'ost
f', 'integration_nginx_nailgun']"
Traceback (most recent call last):
  File "/var/opt/upgrade/site-packages/fuel_upgrade/upgrade.py", line 51, in run
    upgrader.upgrade()
  File "/var/opt/upgrade/site-packages/fuel_upgrade/engines/docker_engine.py", line 100, in upgrade
    self.upgrade_verifier.verify()
  File "/var/opt/upgrade/site-packages/fuel_upgrade/health_checker.py", line 403, in verify
    self._get_non_running_services()))
UpgradeVerificationError: Failed to run services ['integration_postgres_nailgun_nginx', 'integration_ostf_keystone', 'keystone', 'ostf', 'integration_nginx_nailgun']
2015-10-30 18:56:19 DEBUG 11275 (upgrade) Run rollback
2015-10-30 18:56:19 DEBUG 11275 (upgrade) DockerUpgrader: rollbacking...
2015-10-30 18:56:19 INFO 11275 (version_file) Switch current version file to previous version
2015-10-30 18:56:19 DEBUG 11275 (utils) Symlinking "/etc/fuel/5.1/version.yaml" -> "/etc/fuel/version.yaml" [overwrite=1]
2015-10-30 18:56:19 DEBUG 11275 (utils) Removing "/etc/fuel/version.yaml"
2015-10-30 18:56:19 DEBUG 11275 (utils) Symlinking "/etc/supervisord.d/5.1" -> "/etc/supervisord.d/current" [overwrite=1]
2015-10-30 18:56:19 DEBUG 11275 (utils) Removing "/etc/supervisord.d/current"
2015-10-30 18:56:19 INFO 11275 (supervisor_client) Stop all services
2015-10-30 18:57:37 DEBUG 11275 (docker_engine) Stop container: {u'Status': u'Up 17 minutes', u'Created': 1446230328, u'Image': u'fuel/nginx_5.1.1:latest', u'Ports': [{u'IP': u'0.0.0.0', u'Type': u'tcp', u'PublicPort': 8000, u'PrivatePort': 8000}, {u'IP': u'0.0.0.0', u'Type': u'tcp', u'PublicPort': 8080, u'PrivatePort': 8080}], u'Command': u'/usr/local/bin/start.sh', u'Names': [u'/fuel-core-5.1.1-nginx'], u'Id': u'2e2b83570284d0559a425da0759d814e3535fdbd8a7755f302332b81fa244649'}
2015-10-30 18:57:37 DEBUG 11275 (docker_engine) Stop container: 2e2b83570284d0559a425da0759d814e3535fdbd8a7755f302332b81fa244649
2015-10-30 19:00:58 DEBUG 11275 (docker_engine) Stop container: {u'Status': u'Up 19 minutes', u'Created': 1446230258, u'Image': u'fuel/ostf_5.1.1:latest', u'Ports': [{u'IP': u'127.0.0.1', u'Type': u'tcp', u'PublicPort': 8777, u'PrivatePort': 8777}, {u'IP': u'10.20.0.2', u'Type': u'tcp', u'PublicPort': 8777, u'PrivatePort': 8777}], u'Command': u'/bin/sh -c /usr/local/bin/start.sh', u'Names': [u'/fuel-core-5.1.1-nginx/ostf', u'/fuel-core-5.1.1-ostf'], u'Id': u'cc29aa13191b8dc2dc422f8873b34705070ed23b1b5234b6f96fa9c3ab089021'}
2015-10-30 19:00:58 DEBUG 11275 (docker_engine) Stop container: cc29aa13191b8dc2dc422f8873b34705070ed23b1b5234b6f96fa9c3ab089021
2015-10-30 19:04:33 ERROR 11275 (cli) UnixHTTPConnectionPool(host='localhost', port=None): Max retries exceeded with url: /run/docker.sock/v1.10/containers/cc29aa13191b8dc2dc422f8873b34705070ed23b1b5234b6f96fa9c3ab089021/stop?t=200 (Caused by <class 'httplib.BadStatusLine'>: )
Traceback (most recent call last):
  File "/var/opt/upgrade/site-packages/fuel_upgrade/cli.py", line 172, in main
    run_upgrade(parse_args(sys.argv[1:]))
  File "/var/opt/upgrade/site-packages/fuel_upgrade/cli.py", line 164, in run_upgrade
    upgrade_manager.run()
  File "/var/opt/upgrade/site-packages/fuel_upgrade/upgrade.py", line 58, in run
    self.rollback()
  File "/var/opt/upgrade/site-packages/fuel_upgrade/upgrade.py", line 90, in rollback
    upgrader.rollback()
  File "/var/opt/upgrade/site-packages/fuel_upgrade/engines/docker_engine.py", line 108, in rollback
    self.stop_fuel_containers()
  File "/var/opt/upgrade/site-packages/fuel_upgrade/engines/docker_engine.py", line 540, in stop_fuel_containers
    self.stop_container(container['Id'])
  File "/var/opt/upgrade/site-packages/fuel_upgrade/engines/docker_engine.py", line 611, in stop_container
    container_id, self.config.docker['stop_container_timeout'])
  File "/var/opt/upgrade/site-packages/docker/client.py", line 749, in stop
    timeout=max(timeout, self._timeout))
  File "/var/opt/upgrade/site-packages/docker/client.py", line 66, in _post
    return self.post(url, **self._set_request_timeout(kwargs))
  File "/var/opt/upgrade/site-packages/requests/sessions.py", line 425, in post
    return self.request('POST', url, data=data, **kwargs)
  File "/var/opt/upgrade/site-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/var/opt/upgrade/site-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/var/opt/upgrade/site-packages/requests/adapters.py", line 378, in send
    raise ConnectionError(e)
ConnectionError: UnixHTTPConnectionPool(host='localhost', port=None): Max retries exceeded with url: /run/docker.sock/v1.10/containers/cc29aa13191b8dc2dc422f8873b34705070ed23b1b5234b6f96fa9c3ab089021/stop?t=200 (Caused by <class 'httplib.BadStatusLine'>: )

Revision history for this message
John (john-aguilar) said :
#5

Thanks, and It's a different set of health checkers, which are run automatically as part of the upgrade script. Here are the relevant lines from the log at this step, which happens after successful rendering of the supervisor.conf templates for all containers:

2015-10-30 18:40:17 DEBUG 11275 (health_checker) Start ostf checker
 2015-10-30 18:40:17 DEBUG 11275 (health_checker) Start rabbitmq checker
 2015-10-30 18:40:17 DEBUG 11275 (health_checker) Start cobbler checker
 2015-10-30 18:40:17 DEBUG 11275 (health_checker) Start postgres checker
 2015-10-30 18:40:17 DEBUG 11275 (health_checker) Start rsync checker
 2015-10-30 18:40:17 DEBUG 11275 (health_checker) Start rsyslog checker
 2015-10-30 18:40:17 DEBUG 11275 (health_checker) Start mcollective checker
 2015-10-30 18:40:17 DEBUG 11275 (health_checker) Start keystone checker
 2015-10-30 18:40:17 DEBUG 11275 (health_checker) Start nginx checker
 2015-10-30 18:40:17 DEBUG 11275 (health_checker) Start integration_ostf_keystone checker
 2015-10-30 18:40:17 DEBUG 11275 (keystone_client) Cannot authenticate in keystone: HTTPConnectionPool(host='10.20.0.2', port=5000): Max retries exceeded with url: /v2.0/tokens (Caused b
 y <class 'socket.error'>: [Errno 111] Connection refused)
 2015-10-30 18:40:17 DEBUG 11275 (health_checker) Start integration_nginx_nailgun checker
 2015-10-30 18:40:17 DEBUG 11275 (health_checker) Start integration_postgres_nailgun_nginx checker
 2015-10-30 18:40:17 DEBUG 11275 (keystone_client) Cannot authenticate in keystone: HTTPConnectionPool(host='10.20.0.2', port=5000): Max retries exceeded with url: /v2.0/tokens (Caused by <class 'socket.error'>: [Errno 111] Connection refused)
 2015-10-30 18:40:17 DEBUG 11275 (health_checker) Start integration_rabbitmq_astute_nailgun checker
 2015-10-30 18:40:17 INFO 11275 (health_checker) Failed checkers: ['integration_postgres_nailgun_nginx', 'integration_ostf_keystone', 'integration_nginx_nailgun', 'ostf', 'nginx', 'keystone', 'integration_rabbitmq_astute_nailgun']

-----

The above messages were repeated every 3 seconds for 15 minutes, until it finally gave up and tried to rollback (at which time the Docker daemon died, but that's a separate issue):

2015-10-30 18:56:18 ERROR 11275 (upgrade) DockerUpgrader: failed to upgrade: "Failed to run services ['integration_postgres_nailgun_nginx', 'integration_ostf_keystone', 'keystone', 'ost
 f', 'integration_nginx_nailgun']"
 Traceback (most recent call last):
   File "/var/opt/upgrade/site-packages/fuel_upgrade/upgrade.py", line 51, in run
     upgrader.upgrade()
   File "/var/opt/upgrade/site-packages/fuel_upgrade/engines/docker_engine.py", line 100, in upgrade
     self.upgrade_verifier.verify()
   File "/var/opt/upgrade/site-packages/fuel_upgrade/health_checker.py", line 403, in verify
     self._get_non_running_services()))
 UpgradeVerificationError: Failed to run services ['integration_postgres_nailgun_nginx', 'integration_ostf_keystone', 'keystone', 'ostf', 'integration_nginx_nailgun']
 2015-10-30 18:56:19 DEBUG 11275 (upgrade) Run rollback
 2015-10-30 18:56:19 DEBUG 11275 (upgrade) DockerUpgrader: rollbacking...
 2015-10-30 18:56:19 INFO 11275 (version_file) Switch current version file to previous version
 2015-10-30 18:56:19 DEBUG 11275 (utils) Symlinking "/etc/fuel/5.1/version.yaml" -> "/etc/fuel/version.yaml" [overwrite=1]
 2015-10-30 18:56:19 DEBUG 11275 (utils) Removing "/etc/fuel/version.yaml"
 2015-10-30 18:56:19 DEBUG 11275 (utils) Symlinking "/etc/supervisord.d/5.1" -> "/etc/supervisord.d/current" [overwrite=1]
 2015-10-30 18:56:19 DEBUG 11275 (utils) Removing "/etc/supervisord.d/current"
 2015-10-30 18:56:19 INFO 11275 (supervisor_client) Stop all services
 2015-10-30 18:57:37 DEBUG 11275 (docker_engine) Stop container: {u'Status': u'Up 17 minutes', u'Created': 1446230328, u'Image': u'fuel/nginx_5.1.1:latest', u'Ports': [{u'IP': u'0.0.0.0', u'Type': u'tcp', u'PublicPort': 8000, u'PrivatePort': 8000}, {u'IP': u'0.0.0.0', u'Type': u'tcp', u'PublicPort': 8080, u'PrivatePort': 8080}], u'Command': u'/usr/local/bin/start.sh', u'Names': [u'/fuel-core-5.1.1-nginx'], u'Id': u'2e2b83570284d0559a425da0759d814e3535fdbd8a7755f302332b81fa244649'}
 2015-10-30 18:57:37 DEBUG 11275 (docker_engine) Stop container: 2e2b83570284d0559a425da0759d814e3535fdbd8a7755f302332b81fa244649
 2015-10-30 19:00:58 DEBUG 11275 (docker_engine) Stop container: {u'Status': u'Up 19 minutes', u'Created': 1446230258, u'Image': u'fuel/ostf_5.1.1:latest', u'Ports': [{u'IP': u'127.0.0.1', u'Type': u'tcp', u'PublicPort': 8777, u'PrivatePort': 8777}, {u'IP': u'10.20.0.2', u'Type': u'tcp', u'PublicPort': 8777, u'PrivatePort': 8777}], u'Command': u'/bin/sh -c /usr/local/bin/start.sh', u'Names': [u'/fuel-core-5.1.1-nginx/ostf', u'/fuel-core-5.1.1-ostf'], u'Id': u'cc29aa13191b8dc2dc422f8873b34705070ed23b1b5234b6f96fa9c3ab089021'}
 2015-10-30 19:00:58 DEBUG 11275 (docker_engine) Stop container: cc29aa13191b8dc2dc422f8873b34705070ed23b1b5234b6f96fa9c3ab089021
 2015-10-30 19:04:33 ERROR 11275 (cli) UnixHTTPConnectionPool(host='localhost', port=None): Max retries exceeded with url: /run/docker.sock/v1.10/containers/cc29aa13191b8dc2dc422f8873b34705070ed23b1b5234b6f96fa9c3ab089021/stop?t=200 (Caused by <class 'httplib.BadStatusLine'>: )
 Traceback (most recent call last):
   File "/var/opt/upgrade/site-packages/fuel_upgrade/cli.py", line 172, in main
     run_upgrade(parse_args(sys.argv[1:]))
   File "/var/opt/upgrade/site-packages/fuel_upgrade/cli.py", line 164, in run_upgrade
     upgrade_manager.run()
   File "/var/opt/upgrade/site-packages/fuel_upgrade/upgrade.py", line 58, in run
     self.rollback()
   File "/var/opt/upgrade/site-packages/fuel_upgrade/upgrade.py", line 90, in rollback
     upgrader.rollback()
   File "/var/opt/upgrade/site-packages/fuel_upgrade/engines/docker_engine.py", line 108, in rollback
     self.stop_fuel_containers()
   File "/var/opt/upgrade/site-packages/fuel_upgrade/engines/docker_engine.py", line 540, in stop_fuel_containers
     self.stop_container(container['Id'])
   File "/var/opt/upgrade/site-packages/fuel_upgrade/engines/docker_engine.py", line 611, in stop_container
     container_id, self.config.docker['stop_container_timeout'])
   File "/var/opt/upgrade/site-packages/docker/client.py", line 749, in stop
     timeout=max(timeout, self._timeout))
   File "/var/opt/upgrade/site-packages/docker/client.py", line 66, in _post
     return self.post(url, **self._set_request_timeout(kwargs))
   File "/var/opt/upgrade/site-packages/requests/sessions.py", line 425, in post
     return self.request('POST', url, data=data, **kwargs)
   File "/var/opt/upgrade/site-packages/requests/sessions.py", line 383, in request
     resp = self.send(prep, **send_kwargs)
   File "/var/opt/upgrade/site-packages/requests/sessions.py", line 486, in send
     r = adapter.send(request, **kwargs)
   File "/var/opt/upgrade/site-packages/requests/adapters.py", line 378, in send
     raise ConnectionError(e)
 ConnectionError: UnixHTTPConnectionPool(host='localhost', port=None): Max retries exceeded with url: /run/docker.sock/v1.10/containers/cc29aa13191b8dc2dc422f8873b34705070ed23b1b5234b6f96fa9c3ab089021/stop?t=200 (Caused by <class 'httplib.BadStatusLine'>: )

Revision history for this message
Launchpad Janitor (janitor) said :
#6

This question was expired because it remained in the 'Open' state without activity for the last 15 days.