Glance

upstream timed out during image_upload

Asked by Rajiv Mucheli

Hi,

I see the below error during image upload :

2020/10/09 12:04:17 [error] 19#19: *3044066 upstream timed out (110: Operation timed out) while sending request to upstream, client: XXX, server: objectstore-3, request: "PUT /v1/AUTH_53d8861350d946f8a83e4f84f8c951c8/glance_bac06799-0f50-4b53-a8c1-f0ce95769cf8/bac06799-0f50-4b53-a8c1-f0ce95769cf8-00098 HTTP/1.1", upstream: "http://127.0.0.1:8080/v1/AUTH_53d8861350d946f8a83e4f84f8c951c8/glance_bac06799-0f50-4b53-a8c1-f0ce95769cf8/bac06799-0f50-4b53-a8c1-f0ce95769cf8-00098", host: "objectstore-3"

Openstack Glance is on Train release
Openstack Glance backend is Swift (Ussuri Release)

This error is generated after 7mins timeout set in swift, this occurs at different chunk# and images, since this error occurs sporadically i am unable to capture tcpdump to determine why glance cannot send data for 7mins or if there is another issue.

Please let me know if further data / information is required.

Regards,
Rajiv

Question information

Language:
English Edit question
Status:
Expired
For:
Glance Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Rajiv Mucheli (rajiv.mucheli) said : 		#1

2020/10/09 12:04:17 [error] 19#19: *3044066 upstream timed out (110: Operation timed out) while sending request to upstream, client: XXX , server: objectstore-3, request: "PUT /v1/AUTH_53d8861350d946f8a83e4f84f8c951c8/glance_bac06799-0f50-4b53-a8c1-f0ce95769cf8/bac06799-0f50-4b53-a8c1-f0ce95769cf8-00098 HTTP/1.1", upstream: "http://127.0.0.1:8080/v1/AUTH_53d8861350d946f8a83e4f84f8c951c8/glance_bac06799-0f50-4b53-a8c1-f0ce95769cf8/bac06799-0f50-4b53-a8c1-f0ce95769cf8-00098", host: "objectstore-3"

Got error from Swift: put_object('glance_bac06799-0f50-4b53-a8c1-f0ce95769cf8', 'bac06799-0f50-4b53-a8c1-f0ce95769cf8-00098', ...) failure and no ability to reset contents for reupload..: glance_store.exceptions.BackendException: Failed to add object to Swift.

Got error from Swift: put_object('glance_bac06799-0f50-4b53-a8c1-f0ce95769cf8', 'bac06799-0f50-4b53-a8c1-f0ce95769cf8-00098', ...) failure and no ability to reset contents for reupload..

2020-10-09 12:04:38,772.772 44 ERROR glance.common.wsgi swiftclient.exceptions.ClientException: put_object('glance_bac06799-0f50-4b53-a8c1-f0ce95769cf8', 'bac06799-0f50-4b53-a8c1-f0ce95769cf8-00098', ...) failure and no ability to reset contents for reupload.

2020-10-09 12:04:38,772.772 44 ERROR glance.common.wsgi Got error from Swift: put_object('glance_bac06799-0f50-4b53-a8c1-f0ce95769cf8', 'bac06799-0f50-4b53-a8c1-f0ce95769cf8-00098', ...) failure and no ability to reset contents for reupload..

Got error from Swift: put_object('glance_bac06799-0f50-4b53-a8c1-f0ce95769cf8', 'bac06799-0f50-4b53-a8c1-f0ce95769cf8-00098', ...) failure and no ability to reset contents for reupload..: swiftclient.exceptions.ClientException: put_object('glance_bac06799-0f50-4b53-a8c1-f0ce95769cf8', 'bac06799-0f50-4b53-a8c1-f0ce95769cf8-00098', ...) failure and no ability to reset contents for reupload.

2020-10-09 12:04:23,464.464 44 ERROR glance_store._drivers.swift.store [req-4a8ab54a-0826-48d3-8a4d-54c71af100de 16fbf18dd4ee5d415412c19df7d1f3ba6942d6a7f2f58b411815eae206f4a1ca 53d8861350d946f8a83e4f84f8c951c8 - cc7f99d95b14466a84faab103a7b8a52 cc7f99d95b14466a84faab103a7b8a52] Error during chunked upload to backend, deleting stale chunks.: swiftclient.exceptions.ClientException: put_object('glance_bac06799-0f50-4b53-a8c1-f0ce95769cf8', 'bac06799-0f50-4b53-a8c1-f0ce95769cf8-00098', ...) failure and no ability to reset contents for reupload.

Revision history for this message
Launchpad Janitor (janitor) said : 		#2

This question was expired because it remained in the 'Open' state without activity for the last 15 days.