There is no Glance quota check for instance snapshot

Asked by Rajiv Mucheli on 2019-07-02

If the user creates a snapshot for Compute instances and there is not sufficient quota available (e.g. for object store), the snapshot will fail with a non obvious error for the end user. The error message should point to the missing quota.

When user uploads an image and the swift quota is full, glance just resets the connection without notifying about the actual problem

I found https://bugs.launchpad.net/horizon/+bug/1493122

https://blueprints.launchpad.net/glance/+spec/glance-quota-enhancements

 but the bp doesnt seem to be implemented. Any suggestions how to proceed?

Actual glance log inside the container contains this:
2018-06-18 08:09:57,023.023 43 ERROR glance.common.wsgi [req-12ddd27e-8135-449e-8f42-daa6f7cd2c52 5060093745e6c6868ab934cbcd8e857c2e8876f62c75139464ef77928b0de77e e7149006df644d54828d07b18a574777 - 7ad3da798ed141bebba80f763e21c711 7ad3da798ed141bebba80f763e21c711] Caught error: Failed to add object to Swift.
Got error from Swift: Object PUT failed: https://objectstore-3.eu-de-2.cloud.sap:443/v1/AUTH_e7149006df644d54828d07b18a574777/glance_489ad498-abf4-42f6-b75b-d80e3b5712ed/489ad498-abf4-42f6-b75b-d80e3b5712ed-00001 413 Request Entity Too Large Upload exceeds quota..: BackendException: Failed to add object to Swift.
Got error from Swift: Object PUT failed: https://objectstore-3.eu-de-2.cloud.sap:443/v1/AUTH_e7149006df644d54828d07b18a574777/glance_489ad498-abf4-42f6-b75b-d80e3b5712ed/489ad498-abf4-42f6-b75b-d80e3b5712ed-00001 413 Request Entity Too Large Upload exceeds quota..

Regards,
Rajiv

Question information

Language:
English Edit question
Status:
Answered
For:
Glance Edit question
Assignee:
No assignee Edit question
Last query:
2019-07-03
Last reply:
2019-07-11

This question was reopened

Could you provide some more information?

(1) How are you making the call (API, glanceclient, openstackclient, horizon)?

(2) What exactly is the error message you are receiving?

Also, from the log extract you posted, glance quotas won't help this problem ... the quota is being exceeded on the swift side when glance tries to store the image. (But I agree that you should be receiving a more helpful error message.)

Short term, if you are using the swift store in its default configuration, there's a glance user who owns *all* the images. Thus this user needs to have a high storage quota in swift. You should increase this user's quota.

Rajiv Mucheli (rajiv.mucheli) said : #2

1) openstack client

2) When user uploads an image and the swift quota is full, glance just resets the connection without notifying about the actual problem:

$ openstack image create bio-windows-2016-amd64-vmware-BIOComp-new3 --disk-format vmdk --container-format bare --property hypervisor_type="vmware" --property vmware_adaptertype="lsiLogicsas" --property vmware_disktype=streamOptimized --property hw_vif_model="vmxnet3" --property vmware_ostype="windows8Server64Guest" --property hw_disk_bus=scsi --file bio-windows-2016-amd64-vmware-BIOComp-new3.vmdk
Error finding address for https://*****/v2/images/e1900699-2146-41b9-b4c3-f27cd0e8e0c8/file: [Errno 104] Connection reset by peer

Could you share more details on which user it is?

Rajiv Mucheli (rajiv.mucheli) said : #3

changing status to need answer

Hi Rajiv,

Before you fix anything, can you see what happens when you use the glanceclient to create the image? Use 'glance image-create' instead of 'openstack image create'. I think all the options can be the same, except you'll need to add '--name' before the image name. (I'm curious whether the glanceclient handles this better.)

To determine the glance swift user, look in your glance-api.conf and see if 'swift_store_multi_tenant' is True (the default is False). If the option is commented out, then glance is using the default (False). If it's set to True (unlikely), then each project's images are being stored in the swift account owned by that project, and you'll have to change the quota in swift on a user-by-user basis.

Otherwise, check to see if there's a value for 'swift_store_config_file'. If there's not, then the glance user is the value of 'swift_store_user'. It will be in the format <project_name>:<user_name>

 If there *is* a swift store config file, look in it for the value of the 'user' (format will be the same as above).

Rajiv Mucheli (rajiv.mucheli) said : #6

Hi Brian,

Could you please share an update ?

Regards,
Rajiv

Hi Rajiv,

The helm templates you point to will generate a glance configuration, but I can't tell from the templates what your configuration is. You need to look at the config that's generated (or your helm Values) to see whether you are using swift in multi-tenant or single-tenant mode.

If .Values.swift.multi_tenant is True, then you need to adjust your swift quota on a per-project basis.
Otherwise, you need to check .Values.global.glance_service_user to get the name of the glance service user (who owns all the images in single-tenant mode), and make sure that user's swift quota is increased sufficiently to be able to store all images.

Can you help with this problem?

Provide an answer of your own, or ask Rajiv Mucheli for more information if necessary.

To post a message you must log in.