Failed to upload image data due to HTTP error: webob.exc.HTTPBadRequest: invalid literal for int() with base 16: b''

You need to provide some more context from the logs. Does this happen immediately? How far into the upload does the error occur? The swift driver has to break the image into segments, there should be messages as each segment is uploaded. Are there any log messages from swift? Also, what version of openstack are you using, and how are you doing the upload (horizon, openstackclient, glanceclient, direct Images API call)?

Finally, 230GB??? That's pretty enormous.

Hi Brian,

Thanks for assisting once again.

1. It happens after 120 written chunks
2. Swift generates :

Failed to add object to Swift.
Got error from Swift: put_object('glance_950b4338-7fc0-425a-b0f1-2dad781128b7', '950b4338-7fc0-425a-b0f1-2dad781128b7-00791', ...) failure and no ability to reset contents for reupload..

Caught error: Failed to add object to Swift.
Got error from Swift: put_object('glance_950b4338-7fc0-425a-b0f1-2dad781128b7', '950b4338-7fc0-425a-b0f1-2dad781128b7-00791', ...) failure and no ability to reset contents for reupload..

'swiftclient.exceptions.ClientException: put_object('glance_950b4338-7fc0-425a-b0f1-2dad781128b7', '950b4338-7fc0-425a-b0f1-2dad781128b7-00791', ...) failure and no ability to reset contents for reupload.'

3. Is there any code difference between openstackclient 5.2.0 and glance 3.1.1 for creating images on python3 environment ? sometimes openstack client takes a long time in calculating hashes, hence i switch to glanceclient.

Image client initialized using OpenStack SDK: <openstack.image.v2._proxy.Proxy object at 0x7fd6aca01438>
Calculating hashes for cceeimage.vmdk

4. Yes, its 230 Gb, the current chunk size is 204Mb, would the upload speed increase if chunk size is doubled or increased ?

Since glance needs stable and open connection during image creation, it becomes difficult to while creating a image from 230 Gb vmdk, running this as a background job helps. Are there better ways to manage image creation for big vmdks ?

changing status

Hi Rajiv, this is just off the top of my head -- you should ask in #openstack-glance on IRC for verification because some of my glance knowledge may be outdated.

1, 2: is it always after the same chunk? If so, have you tried a different image (maybe there's something weird about that image).

3: the openstackclient and glanceclient have completely different code.

4: it wouldn't hurt to try increasing the swift_store_large_object_chunk_size. Try 512 MB. You might want to check with some swift people for suggestions.

Another thing you could try is swift_buffer_on_upload (but read the help text for that option in glance-api.conf first). I'm not sure it will help for your specific issue, but it's worth knowing about if you're dealing with a lot of images of enormous size. Another thing: I'm not sure how widely used the BufferedReader is, so it may have some bugs.

As far as "better ways" -- if you are a software developer, you could try modifying the swift_store driver to do parallel segment uploads. I'm pretty sure that right now it only does serial uploads. You'd definitely want to check with some swift experts first, because there are all sorts of problems you could run into (for example, swift can throttle writes to a single container).

Your best bet would be to send something to the openstack-discuss mailing list with [ops][glance][swift] in the subject line and ask if anyone has experience uploading 200GB images to glance using the swift store, and what they've done to get it to work well.