glance image-download fails 'silently'

Asked by Jake Hill on 2019-06-14

I'm running cloud:bionic-rocky. I've run into a wall trying to diagnose an issue with glance which is blocking the creation of new instances.

I can replicate the problem from the command line;

routergod@juju:~/openstack-base$ glance image-download --file cirros.image 2302424f-51b5-4075-8f2a-84bc2d0071bf
Error finding address for http://192.168.200.201:9292/v2/images/2302424f-51b5-4075-8f2a-84bc2d0071bf/file: Unable to establish connection to http://192.168.200.201:9292/v2/images/2302424f-51b5-4075-8f2a-84bc2d0071bf/file: ('Connection aborted.', BadStatusLine("''",))

The operation unexpectedly only produces a single line in the normal glance-api.log;

2019-06-14 10:04:55.651 27832 INFO eventlet.wsgi.server [-] 192.168.200.201 - - [14/Jun/2019 10:04:55] "GET /versions HTTP/1.1" 200 1025 0.002114

The debug logs are pretty brief too (see below).

If I log in to the glance server, I can download the image from rbd/ceph;

root@juju-533538-0-lxd-2:~# rbd --id glance -p glance export 2302424f-51b5-4075-8f2a-84bc2d0071bf cirros.test.img
Exporting image: 100% complete...done.
root@juju-533538-0-lxd-2:~# ls -l cirros.test.img
-rw-r--r-- 1 root root 12716032 Jun 14 10:11 cirros.test.img

All very confusing, I'm struggling to find an actual error or exception anywhere. Any clues please? Many thanks in advance!

/.J

Debug log:

2019-06-14 10:04:55.648 27832 DEBUG eventlet.wsgi.server [-] (27832) accepted ('192.168.200.201', 38290) server /usr/lib/python3/dist-packages/eventlet/wsgi.py:883
2019-06-14 10:04:55.650 27832 DEBUG glance.api.middleware.version_negotiation [-] Determining version of request: GET /versions Accept: */* process_request /usr/lib/python3/dist-packages/glance/api/middleware/version_negotiation.py:45
2019-06-14 10:04:55.651 27832 INFO eventlet.wsgi.server [-] 192.168.200.201 - - [14/Jun/2019 10:04:55] "GET /versions HTTP/1.1" 200 1025 0.002114
2019-06-14 10:04:56.229 27832 DEBUG eventlet.wsgi.server [-] (27832) accepted ('192.168.200.201', 38302) server /usr/lib/python3/dist-packages/eventlet/wsgi.py:883
2019-06-14 10:04:56.231 27832 DEBUG glance.api.middleware.version_negotiation [-] Determining version of request: GET /v2/images/2302424f-51b5-4075-8f2a-84bc2d0071bf/file Accept: */* process_request /usr/lib/python3/dist-packages/glance/api/middleware/version_negotiation.py:45
2019-06-14 10:04:56.231 27832 DEBUG glance.api.middleware.version_negotiation [-] Using url versioning process_request /usr/lib/python3/dist-packages/glance/api/middleware/version_negotiation.py:57
2019-06-14 10:04:56.231 27832 DEBUG glance.api.middleware.version_negotiation [-] Matched version: v2 process_request /usr/lib/python3/dist-packages/glance/api/middleware/version_negotiation.py:69
2019-06-14 10:04:56.232 27832 DEBUG glance.api.middleware.version_negotiation [-] new path /v2/images/2302424f-51b5-4075-8f2a-84bc2d0071bf/file process_request /usr/lib/python3/dist-packages/glance/api/middleware/version_negotiation.py:70
2019-06-14 10:04:56.530 27832 DEBUG oslo_policy._cache_handler [req-d6a7ad6c-8f7a-40e8-bc4e-8b08c8ce5998 41f5b059d3bf4bebad08ffcabd0899bf d25b3d096e5c47a3b1701b3fff2c2823 - f7f19dbcac14459c99e4dc1aad7e0df0 f7f19dbcac14459c99e4dc1aad7e0df0] Reloading cached file /etc/glance/policy.json read_cached_file /usr/lib/python3/dist-packages/oslo_policy/_cache_handler.py:40
2019-06-14 10:04:56.531 27832 DEBUG oslo_policy.policy [req-d6a7ad6c-8f7a-40e8-bc4e-8b08c8ce5998 41f5b059d3bf4bebad08ffcabd0899bf d25b3d096e5c47a3b1701b3fff2c2823 - f7f19dbcac14459c99e4dc1aad7e0df0 f7f19dbcac14459c99e4dc1aad7e0df0] Reloaded policy file: /etc/glance/policy.json _load_policy_file /usr/lib/python3/dist-packages/oslo_policy/policy.py:770
2019-06-14 10:04:56.536 27832 DEBUG glance_store.backend [req-d6a7ad6c-8f7a-40e8-bc4e-8b08c8ce5998 41f5b059d3bf4bebad08ffcabd0899bf d25b3d096e5c47a3b1701b3fff2c2823 - f7f19dbcac14459c99e4dc1aad7e0df0 f7f19dbcac14459c99e4dc1aad7e0df0] Attempting to import store glance.store.rbd.Store _load_store /usr/lib/python3/dist-packages/glance_store/backend.py:267
2019-06-14 10:04:56.538 27832 DEBUG glance_store.capabilities [req-d6a7ad6c-8f7a-40e8-bc4e-8b08c8ce5998 41f5b059d3bf4bebad08ffcabd0899bf d25b3d096e5c47a3b1701b3fff2c2823 - f7f19dbcac14459c99e4dc1aad7e0df0 f7f19dbcac14459c99e4dc1aad7e0df0] Store glance_store._drivers.rbd.Store doesn't support updating dynamic storage capabilities. Please overwrite 'update_capabilities' method of the store to implement updating logics if needed. update_capabilities /usr/lib/python3/dist-packages/glance_store/capabilities.py:97
2019-06-14 10:04:56.539 27832 DEBUG glance_store.driver [req-d6a7ad6c-8f7a-40e8-bc4e-8b08c8ce5998 41f5b059d3bf4bebad08ffcabd0899bf d25b3d096e5c47a3b1701b3fff2c2823 - f7f19dbcac14459c99e4dc1aad7e0df0 f7f19dbcac14459c99e4dc1aad7e0df0] Late loading location class glance_store._drivers.rbd.StoreLocation get_store_location_class /usr/lib/python3/dist-packages/glance_store/driver.py:112
2019-06-14 10:04:56.539 27832 DEBUG glance_store.location [req-d6a7ad6c-8f7a-40e8-bc4e-8b08c8ce5998 41f5b059d3bf4bebad08ffcabd0899bf d25b3d096e5c47a3b1701b3fff2c2823 - f7f19dbcac14459c99e4dc1aad7e0df0 f7f19dbcac14459c99e4dc1aad7e0df0] Registering scheme rbd with {'store': <glance_store._drivers.rbd.Store object at 0x7fe699a4bbe0>, 'location_class': <class 'glance_store._drivers.rbd.StoreLocation'>, 'store_entry': 'glance.store.rbd.Store'} register_scheme_map /usr/lib/python3/dist-packages/glance_store/location.py:142

Question information

Language:
English Edit question
Status:
Solved
For:
Glance Edit question
Assignee:
No assignee Edit question
Solved by:
Jake Hill
Solved:
2019-06-14
Last query:
2019-06-14
Last reply:
Jake Hill (routergod) said : #1

Turns out this was due to ceph being full/broken.

root@juju-533538-0-lxd-2:~# ceph --id glance health
HEALTH_ERR 1 full osd(s); 27 pool(s) full; Degraded data redundancy: 96/30336 objects degraded (0.316%), 11 pgs degraded; Degraded data redundancy (low space): 11 pgs recovery_toofull

I added some more ceph-osd nodes and it works again.