Failed to upload image data due to HTTP error: webob.exc.HTTPBadRequest: invalid literal for int() with base 16: b''

Asked by Rajiv Mucheli on 2020-05-15

Hi,

While trying to create an image of 230Gb from sles.vmdk file i get the below error :

2020-05-15 04:26:18,536.536 38 ERROR glance.api.v2.image_data [req-f14b4663-4e5f-48af-bdea-00cdc012e574 c28ee5fe28d0f1f6ba0c9e1a17097f20db7664972584fdcb8ed24b7a7ccb820d 4d0477a019414ee08ad0fd11b777eee2 - ec213443e8834473b579f7bea9e8c194 ec213443e8834473b579f7bea9e8c194] Failed to upload image data due to HTTP error: webob.exc.HTTPBadRequest: invalid literal for int() with base 16: b''

2020-05-15 04:25:54,819.819 38 ERROR glance_store._drivers.swift.store [req-f14b4663-4e5f-48af-bdea-00cdc012e574 c28ee5fe28d0f1f6ba0c9e1a17097f20db7664972584fdcb8ed24b7a7ccb820d 4d0477a019414ee08ad0fd11b777eee2 - ec213443e8834473b579f7bea9e8c194 ec213443e8834473b579f7bea9e8c194] Error during chunked upload to backend, deleting stale chunks.: eventlet.wsgi.ChunkReadError: invalid literal for int() with base 16: b''

Please assist.

Question information

Language:
English Edit question
Status:
Answered
For:
Glance Edit question
Assignee:
No assignee Edit question
Last query:
23 hours ago
Last reply:
22 hours ago

You need to provide some more context from the logs. Does this happen immediately? How far into the upload does the error occur? The swift driver has to break the image into segments, there should be messages as each segment is uploaded. Are there any log messages from swift? Also, what version of openstack are you using, and how are you doing the upload (horizon, openstackclient, glanceclient, direct Images API call)?

Finally, 230GB??? That's pretty enormous.

Hi Brian,

Thanks for assisting once again.

1. It happens after 120 written chunks
2. Swift generates :

Failed to add object to Swift.
Got error from Swift: put_object('glance_950b4338-7fc0-425a-b0f1-2dad781128b7', '950b4338-7fc0-425a-b0f1-2dad781128b7-00791', ...) failure and no ability to reset contents for reupload..

Caught error: Failed to add object to Swift.
Got error from Swift: put_object('glance_950b4338-7fc0-425a-b0f1-2dad781128b7', '950b4338-7fc0-425a-b0f1-2dad781128b7-00791', ...) failure and no ability to reset contents for reupload..

'swiftclient.exceptions.ClientException: put_object('glance_950b4338-7fc0-425a-b0f1-2dad781128b7', '950b4338-7fc0-425a-b0f1-2dad781128b7-00791', ...) failure and no ability to reset contents for reupload.'

3. Is there any code difference between openstackclient 5.2.0 and glance 3.1.1 for creating images on python3 environment ? sometimes openstack client takes a long time in calculating hashes, hence i switch to glanceclient.

Image client initialized using OpenStack SDK: <openstack.image.v2._proxy.Proxy object at 0x7fd6aca01438>
Calculating hashes for cceeimage.vmdk

4. Yes, its 230 Gb, the current chunk size is 204Mb, would the upload speed increase if chunk size is doubled or increased ?

Since glance needs stable and open connection during image creation, it becomes difficult to while creating a image from 230 Gb vmdk, running this as a background job helps. Are there better ways to manage image creation for big vmdks ?

changing status

Hi Rajiv, this is just off the top of my head -- you should ask in #openstack-glance on IRC for verification because some of my glance knowledge may be outdated.

1, 2: is it always after the same chunk? If so, have you tried a different image (maybe there's something weird about that image).

3: the openstackclient and glanceclient have completely different code.

4: it wouldn't hurt to try increasing the swift_store_large_object_chunk_size. Try 512 MB. You might want to check with some swift people for suggestions.

Another thing you could try is swift_buffer_on_upload (but read the help text for that option in glance-api.conf first). I'm not sure it will help for your specific issue, but it's worth knowing about if you're dealing with a lot of images of enormous size. Another thing: I'm not sure how widely used the BufferedReader is, so it may have some bugs.

As far as "better ways" -- if you are a software developer, you could try modifying the swift_store driver to do parallel segment uploads. I'm pretty sure that right now it only does serial uploads. You'd definitely want to check with some swift experts first, because there are all sorts of problems you could run into (for example, swift can throttle writes to a single container).

Your best bet would be to send something to the openstack-discuss mailing list with [ops][glance][swift] in the subject line and ask if anyone has experience uploading 200GB images to glance using the swift store, and what they've done to get it to work well.

Can you help with this problem?

Provide an answer of your own, or ask Rajiv Mucheli for more information if necessary.

To post a message you must log in.