I am in the process of upgrading my fuel master from 5.1 to 5.1.1, however, everytime I run the ./upgrade.sh script to upgrade the upgrade fails with below error. I have seen other similar problems but have found no solution to them. Can someone please help?
Steps I took to upgrade
-Downloaded upgrade script tar from mirantis openstack page
-CD to /var/tmp (plenty of space)
- executed ./upgrade --password *****
current Environment:
Fuel 5.1 installed in VM
54 Node HA cluster which includes 3 controller nodes
my current fuel VM is on proxmox cluster which is part of the 54 node cluster; the rest of the nodes form my openstack environment.
the error below is an excerpt from /var/opt/fuel_upgrade.log. Message is trimmed to save space.
...
2015-11-04 17:28:55 DEBUG 19596 (utils) Symlinking "/etc/supervisord.d/5.1" -> "/etc/supervisord.d/current" [overwrite=1]
2015-11-04 17:28:55 DEBUG 19596 (utils) Removing "/etc/supervisord.d/current"
2015-11-04 17:28:55 INFO 19596 (supervisor_client) Stop all services
2015-11-04 17:29:57 DEBUG 19596 (docker_engine) Stop container: {u'Status': u'Up 3 minutes', u'Created': 1446658009, u'Image': u'fuel/postgres_5.1.1:latest', u'Ports': [{u'IP': u'127.0.0.1', u'Type': u'tcp', u'PublicPort': 5432, u'PrivatePort': 5432}, {u'IP': u'10.20.0.2', u'Type': u'tcp', u'PublicPort': 5432, u'PrivatePort': 5432}], u'Command': u'/usr/local/bin/start.sh', u'Names': [u'/fuel-core-5.1.1-postgres'], u'Id': u'c3db4c880d0eadc36a86ac990c7885a3f59d5b625468525fd754b34f90617462'}
2015-11-04 17:29:57 DEBUG 19596 (docker_engine) Stop container: c3db4c880d0eadc36a86ac990c7885a3f59d5b625468525fd754b34f90617462
2015-11-04 17:30:18 DEBUG 19596 (docker_engine) Stop container: {u'Status': u'Up 3 minutes', u'Created': 1446657968, u'Image': u'fuel/mcollective_5.1.1:latest', u'Ports': [], u'Command': u'/usr/local/bin/start.sh', u'Names': [u'/fuel-core-5.1.1-mcollective'], u'Id': u'd737bd8c161044385f7e68ba2f67c2ce5dfc606d95dd5d03cf40425785f4da5b'}
2015-11-04 17:30:18 DEBUG 19596 (docker_engine) Stop container: d737bd8c161044385f7e68ba2f67c2ce5dfc606d95dd5d03cf40425785f4da5b
2015-11-04 17:30:38 DEBUG 19596 (docker_engine) Stop container: {u'Status': u'Up 4 minutes', u'Created': 1446657941, u'Image': u'fuel/cobbler_5.1.1:latest', u'Ports': [{u'IP': u'0.0.0.0', u'Type': u'tcp', u'PublicPort': 443, u'PrivatePort': 443}, {u'IP': u'127.0.0.1', u'Type': u'udp', u'PublicPort': 53, u'PrivatePort': 53}, {u'IP': u'10.20.0.2', u'Type': u'udp', u'PublicPort': 53, u'PrivatePort': 53}, {u'IP': u'127.0.0.1', u'Type': u'udp', u'PublicPort': 69, u'PrivatePort': 69}, {u'IP': u'10.20.0.2', u'Type': u'udp', u'PublicPort': 69, u'PrivatePort': 69}, {u'IP': u'0.0.0.0', u'Type': u'tcp', u'PublicPort': 80, u'PrivatePort': 80}, {u'Type': u'tcp', u'PublicPort': 53}, {u'Type': u'tcp', u'PublicPort': 67}, {u'Type': u'tcp', u'PublicPort': 69}, {u'Type': u'udp', u'PublicPort': 67}], u'Command': u'/bin/sh -c /usr/local/bin/start.sh', u'Names': [u'/fuel-core-5.1.1-cobbler'], u'Id': u'9ae70c37bb73e079432ffd5228ab54b2f18900196b463b12ab130fb046363250'}
2015-11-04 17:30:38 DEBUG 19596 (docker_engine) Stop container: 9ae70c37bb73e079432ffd5228ab54b2f18900196b463b12ab130fb046363250
2015-11-04 17:30:58 INFO 19596 (supervisor_client) Restart supervisor
2015-11-04 17:30:59 DEBUG 19596 (supervisor_client) List of supervisor processes [{'now': 1446658259, 'group': 'dhcrelay_monitor', 'description': '', 'pid': 26243, 'stderr_logfile': '/var/log/dhcrelay-check-err.log', 'stop': 0, 'statename': 'STARTING', 'start': 1446658258, 'state': 10, 'stdout_logfile': '/var/log/dhcrelay-check.log', 'logfile': '/var/log/dhcrelay-check.log', 'exitstatus': 0, 'spawnerr': '', 'name': 'dhcrelay_monitor'}, {'now': 1446658259, 'group': 'docker-astute', 'description': '', 'pid': 26226, 'stderr_logfile': '/var/log/docker-astuteerr.log', 'stop': 0, 'statename': 'STARTING', 'start': 1446658258, 'state': 10, 'stdout_logfile': '/var/log/docker-astute.log', 'logfile': '/var/log/docker-astute.log', 'exitstatus': 0, 'spawnerr': '', 'name': 'docker-astute'}, {'now': 1446658259, 'group': 'docker-cobbler', 'description': '', 'pid': 26218, 'stderr_logfile': '/var/log/docker-cobbler-err.log', 'stop': 0, 'statename': 'STARTING', 'start': 1446658258, 'state': 10, 'stdout_logfile': '/var/log/docker-cobbler.log', 'logfile': '/var/log/docker-cobbler.log', 'exitstatus': 0, 'spawnerr': '', 'name': 'docker-cobbler'}, {'now': 1446658259, 'group': 'docker-keystone', 'description': '', 'pid': 26224, 'stderr_logfile': '/var/log/docker-keystone-err.log', 'stop': 0, 'statename': 'STARTING', 'start': 1446658258, 'state': 10, 'stdout_logfile': '/var/log/docker-keystone.log', 'logfile': '/var/log/docker-keystone.log', 'exitstatus': 0, 'spawnerr': '', 'name': 'docker-keystone'}, {'now': 1446658259, 'group': 'docker-mcollective', 'description': '', 'pid': 26230, 'stderr_logfile': '/var/log/docker-mcollective-err.log', 'stop': 0, 'statename': 'STARTING', 'start': 1446658258, 'state': 10, 'stdout_logfile': '/var/log/docker-mcollective.log', 'logfile': '/var/log/docker-mcollective.log', 'exitstatus': 0, 'spawnerr': '', 'name': 'docker-mcollective'}, {'now': 1446658259, 'group': 'docker-nailgun', 'description': '', 'pid': 26229, 'stderr_logfile': '/var/log/docker-nailgun-err.log', 'stop': 0, 'statename': 'STARTING', 'start': 1446658258, 'state': 10, 'stdout_logfile': '/var/log/docker-nailgun.log', 'logfile': '/var/log/docker-nailgun.log', 'exitstatus': 0, 'spawnerr': '', 'name': 'docker-nailgun'}, {'now': 1446658259, 'group': 'docker-nginx', 'description': '', 'pid': 26233, 'stderr_logfile': '/var/log/docker-nginx-err.log', 'stop': 0, 'statename': 'STARTING', 'start': 1446658258, 'state': 10, 'stdout_logfile': '/var/log/docker-nginx.log', 'logfile': '/var/log/docker-nginx.log', 'exitstatus': 0, 'spawnerr': '', 'name': 'docker-nginx'}, {'now': 1446658259, 'group': 'docker-ostf', 'description': '', 'pid': 26232, 'stderr_logfile': '/var/log/docker-ostf-err.log', 'stop': 0, 'statename': 'STARTING', 'start': 1446658258, 'state': 10, 'stdout_logfile': '/var/log/docker-ostf.log', 'logfile': '/var/log/docker-ostf.log', 'exitstatus': 0, 'spawnerr': '', 'name': 'docker-ostf'}, {'now': 1446658259, 'group': 'docker-postgres', 'description': '', 'pid': 26219, 'stderr_logfile': '/var/log/docker-postgres-err.log', 'stop': 0, 'statename': 'STARTING', 'start': 1446658258, 'state': 10, 'stdout_logfile': '/var/log/docker-postgres.log', 'logfile': '/var/log/docker-postgres.log', 'exitstatus': 0, 'spawnerr': '', 'name': 'docker-postgres'}, {'now': 1446658259, 'group': 'docker-rabbitmq', 'description': '', 'pid': 26220, 'stderr_logfile': '/var/log/docker-rabbitmq-err.log', 'stop': 0, 'statename': 'STARTING', 'start': 1446658258, 'state': 10, 'stdout_logfile': '/var/log/docker-rabbitmq.log', 'logfile': '/var/log/docker-rabbitmq.log', 'exitstatus': 0, 'spawnerr': '', 'name': 'docker-rabbitmq'}, {'now': 1446658259, 'group': 'docker-rsync', 'description': '', 'pid': 26221, 'stderr_logfile': '/var/log/docker-rsync-err.log', 'stop': 0, 'statename': 'STARTING', 'start': 1446658258, 'state': 10, 'stdout_logfile': '/var/log/docker-rsync.log', 'logfile': '/var/log/docker-rsync.log', 'exitstatus': 0, 'spawnerr': '', 'name': 'docker-rsync'}, {'now': 1446658259, 'group': 'docker-rsyslog', 'description': '', 'pid': 26238, 'stderr_logfile': '/var/log/docker-rsyslog-err.log', 'stop': 0, 'statename': 'STARTING', 'start': 1446658258, 'state': 10, 'stdout_logfile': '/var/log/docker-rsyslog.log', 'logfile': '/var/log/docker-rsyslog.log', 'exitstatus': 0, 'spawnerr': '', 'name': 'docker-rsyslog'}]
2015-11-04 17:30:59 DEBUG 19596 (upgrade) BootstrapUpgrader: rollbacking...
2015-11-04 17:30:59 INFO 19596 (bootstrap) bootstrap upgrader: rollbacking...
2015-11-04 17:30:59 DEBUG 19596 (utils) Removing "/var/www/nailgun/5.1.1_bootstrap"
2015-11-04 17:30:59 DEBUG 19596 (utils) Copying "/var/www/nailgun/5.1_bootstrap" -> "/var/www/nailgun/bootstrap" [overwrite=1 symlinks=1]
2015-11-04 17:30:59 DEBUG 19596 (utils) Removing "/var/www/nailgun/bootstrap"
2015-11-04 17:31:02 INFO 19596 (bootstrap) bootstrap upgrader: rollbacked
2015-11-04 17:31:02 DEBUG 19596 (upgrade) HostSystemUpgrader: rollbacking...
2015-11-04 17:31:02 DEBUG 19596 (utils) Remove file "/etc/yum.repos.d/5.1.1_nailgun.repo"
2015-11-04 17:31:02 ERROR 19596 (upgrade) *** UPGRADE FAILED
2015-11-04 17:31:02 ERROR 19596 (cli) Shell command executed with "125" exit code: lxc-attach --name c3db4c880d0eadc36a86ac990c7885a3f59d5b625468525fd754b34f90617462 -- su postgres -c "psql -f /tmp/upgrade/pg_dump_all.sql postgres"
Traceback (most recent call last):
File "/var/tmp/upgrade/site-packages/fuel_upgrade/cli.py", line 172, in main
run_upgrade(parse_args(sys.argv[1:]))
File "/var/tmp/upgrade/site-packages/fuel_upgrade/cli.py", line 164, in run_upgrade
upgrade_manager.run()
File "/var/tmp/upgrade/site-packages/fuel_upgrade/upgrade.py", line 51, in run
upgrader.upgrade()
File "/var/tmp/upgrade/site-packages/fuel_upgrade/engines/docker_engine.py", line 94, in upgrade
self.create_and_start_new_containers()
File "/var/tmp/upgrade/site-packages/fuel_upgrade/engines/docker_engine.py", line 338, in create_and_start_new_containers
self.run_after_container_creation_command(container)
File "/var/tmp/upgrade/site-packages/fuel_upgrade/engines/docker_engine.py", line 358, in run_after_container_creation_command
'', retries=30, interval=4)
File "/var/tmp/upgrade/site-packages/fuel_upgrade/engines/docker_engine.py", line 413, in exec_with_retries
return func()
File "/var/tmp/upgrade/site-packages/fuel_upgrade/engines/docker_engine.py", line 354, in execute
self.exec_cmd_in_container(container['container_name'], command)
File "/var/tmp/upgrade/site-packages/fuel_upgrade/engines/docker_engine.py", line 373, in exec_cmd_in_container
db_container_id, cmd))
File "/var/tmp/upgrade/site-packages/fuel_upgrade/utils.py", line 57, in exec_cmd
_wait_and_check_exit_code(cmd, child)
File "/var/tmp/upgrade/site-packages/fuel_upgrade/utils.py", line 108, in _wait_and_check_exit_code
'exit code: {1} '.format(exit_code, cmd))
ExecutedErrorNonZeroExitCode: Shell command executed with "125" exit code: lxc-attach --name c3db4c880d0eadc36a86ac990c7885a3f59d5b625468525fd754b34f90617462 -- su postgres -c "psql -f /tmp/upgrade/pg_dump_all.sql postgres" ******end of file*******