Nova + Glance + Swift: large (>5G) file upload error again

Asked by Alexander Paschenko on 2011-11-12

Hi all,

Here's my story. I have a small (single-node in fact) cluster which I decided to pump up to the max (utilize Swift at storage engine and obtain images from Nova via Glance with that Swift backend just to get experience in tuning all the stuff to work together).
I have OpenSUSE 11.4 and I've installed all the components from sources (all stable 2011-3 Diablo releases from 2011-09-22) - I've already had some experience with OpenStack on Ubuntu and RHEL/CentOS, so I had the understanding of each component's role. Therefore I first installed Swift, then Glance and tied them together. After that I tested that they work - I successfully uploaded a raw disk image of 6 gigabytes in size by plainly using 'glance-upload' utility - everything went just perfect, and I saw that image successfully uploaded into Swift storage (by using 'swift' command) as well as added into Glance registry (checked by invoking 'glance index' command). Later, when I installed Nova and invoked 'euca-describe-images', I saw that image in the list as 'active' - that's one more proof that everything worked smooth until that point.
The problem is that when I try to upload the same image via Nova object store by following that common sequence of
euca-bundle-image -> euca-upload-bundle -> euca-register
I run into the following problem. The image does not get uploaded properly, it ends up with state of 'failed_upload' when I invoke 'euca-describe-images', and it appears to me that when Nova object store tries to push that image to Glance and when Glance pushes that image to Swift somewhere between those services the file size isn't counted (or isn't properly reported in metadata). I of course do know about bug #827660, but it's apparently fixed, and it was a large surprise to me to see that this problem still pops up accidentally. So I think that after all the problem hides somewhere in Nova object store Glance client because when I use Glance directly by invoking aforementioned 'glance-upload' utility, everything works fine.

This is what I've found in glance-api.log about this issue:

2011-11-12 19:01:05 ERROR [glance.store.swift] Failed to add object to Swift.
Got error from Swift: Object PUT failed: https://192.168.9.20:8080/v1/AUTH_system/glance/8 413 Request Entity Too Large
2011-11-12 19:01:05 ERROR [glance.api.v1.images] Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/glance-2011.3-py2.7.egg/glance/api/v1/images.py", line 369, in _upload
    image_size)
  File "/usr/local/lib/python2.7/site-packages/glance-2011.3-py2.7.egg/glance/store/swift.py", line 410, in add
    raise glance.store.BackendException(msg)
BackendException: Failed to add object to Swift.
Got error from Swift: Object PUT failed: https://192.168.9.20:8080/v1/AUTH_system/glance/8 413 Request Entity Too Large

Thank you very much,
Alex

Question information

Language:
English Edit question
Status:
Solved
For:
Glance Edit question
Assignee:
No assignee Edit question
Solved by:
Alexander Paschenko
Solved:
2011-11-13
Last query:
2011-11-13
Last reply:

Ah, nevermind that - I've read about bug #845788 that appears to be connected with that problem and is apparently fixed... However, it's strange that the fix didn't make it to Diablo 2011-3 release that's been published on September 22 although the fix had been released earlier - probably it needs to be clarified somehow.
This question should be closed and removed, anyway, so admins please do so.