Is there glance image upload limit ?

Asked by Rajiv Mucheli on 2020-05-11

Hi,

I need to move several images approx 300gb from a different cloud vendor to my cloud (my glance backend is swift), i am currently testing this upload scenario in batches, all image upload work well until 8 but all fail with the below error when i try to upload 30 images in parallel. I see the below types of error in sentry :

1. Image creation failed: Unable to establish connection to https://####/v2/images/304ae96b-3763-488a-aed9-bdcd101173f6/file: ('Connection aborted.', OSError("(32, 'EPIPE')",))

2. DBDeadlock
(pymysql.err.InternalError) (1205, 'Lock wait timeout exceeded; try restarting transaction')
[SQL: DELETE FROM image_properties WHERE image_properties.id in (SELECT T1.id FROM (SELECT image_properties.id
FROM image_properties
WHERE image_properties.deleted_at < %(deleted_at_1)s ORDER BY image_properties.deleted_at
 LIMIT %(param_1)s) as T1)]
[parameters: {'deleted_at_1': datetime.datetime(2020, 4, 27, 15, 24, 48, 945401), 'param_1': 50}]

3. Image status transition from saving to saving is not allowed

4. Failed to upload image data due to HTTP error

5. Caught error: Failed to add object to Swift.
Got error from Swift: put_object('glance_1107ee83-5a0a-4f1a-8a9d-d62c0a5c2a9f', '1107ee83-5a0a-4f1a-8a9d-d62c0a5c2a9f-00012', ...) failure and no ability to reset contents for reupload..

Notes:
1. Glance keystone token is for 8hrs
2. I have 10TB of object based storage
3. i have 3 glance-api pods on k8s

Question information

Language:
English Edit question
Status:
Expired
For:
Glance Edit question
Assignee:
No assignee Edit question
Last query:
2020-05-11
Last reply:
4 hours ago
Rajiv Mucheli (rajiv.mucheli) said : #2

more debug logs :

https://####:443 "GET /v2/images/2d4ab8aa-a1bc-45b6-89cb-50da76dd4f31/file HTTP/1.1" 200 147964164096
Unexpected exception for https://####/v2/images/2d4ab8aa-a1bc-45b6-89cb-50da76dd4f31/file: ('Connection broken: OSError("(104, \'ECONNRESET\')",)', OSError("(104, 'ECONNRESET')",))
Traceback (most recent call last):
  File "/home/ccloud/.local/lib/python3.5/site-packages/urllib3/contrib/pyopenssl.py", line 313, in recv_into
    return self.connection.recv_into(*args, **kwargs)
  File "/home/ccloud/.local/lib/python3.5/site-packages/OpenSSL/SSL.py", line 1840, in recv_into
    self._raise_ssl_error(self._ssl, result)
  File "/home/ccloud/.local/lib/python3.5/site-packages/OpenSSL/SSL.py", line 1663, in _raise_ssl_error
    raise SysCallError(errno, errorcode.get(errno))
OpenSSL.SSL.SysCallError: (104, 'ECONNRESET')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/ccloud/.local/lib/python3.5/site-packages/urllib3/response.py", line 437, in _error_catcher
    yield
  File "/home/ccloud/.local/lib/python3.5/site-packages/urllib3/response.py", line 519, in read
    data = self._fp.read(amt) if not fp_closed else b""
  File "/usr/lib/python3.5/http/client.py", line 458, in read
    n = self.readinto(b)
  File "/usr/lib/python3.5/http/client.py", line 498, in readinto
    n = self.fp.readinto(b)
  File "/usr/lib/python3.5/socket.py", line 575, in readinto
    return self._sock.recv_into(b)
  File "/home/ccloud/.local/lib/python3.5/site-packages/urllib3/contrib/pyopenssl.py", line 318, in recv_into
    raise SocketError(str(e))
OSError: (104, 'ECONNRESET')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/ccloud/.local/lib/python3.5/site-packages/requests/models.py", line 751, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/home/ccloud/.local/lib/python3.5/site-packages/urllib3/response.py", line 576, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/home/ccloud/.local/lib/python3.5/site-packages/urllib3/response.py", line 541, in read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/ccloud/.local/lib/python3.5/site-packages/urllib3/response.py", line 455, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: OSError("(104, \'ECONNRESET\')",)', OSError("(104, 'ECONNRESET')",))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/ccloud/.local/lib/python3.5/site-packages/keystoneauth1/session.py", line 1004, in _send_request
    resp = self.session.request(method, url, **kwargs)
  File "/home/ccloud/.local/lib/python3.5/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/ccloud/.local/lib/python3.5/site-packages/requests/sessions.py", line 683, in send
    r.content
  File "/home/ccloud/.local/lib/python3.5/site-packages/requests/models.py", line 829, in content
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
  File "/home/ccloud/.local/lib/python3.5/site-packages/requests/models.py", line 754, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: OSError("(104, \'ECONNRESET\')",)', OSError("(104, 'ECONNRESET')",))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/ccloud/.local/lib/python3.5/site-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/home/ccloud/.local/lib/python3.5/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/home/ccloud/.local/lib/python3.5/site-packages/cliff/command.py", line 187, in run
    return_code = self.take_action(parsed_args) or 0
  File "/home/ccloud/.local/lib/python3.5/site-packages/openstackclient/image/v2/image.py", line 789, in take_action
    image_client.download_image(image.id, output=parsed_args.file)
  File "/home/ccloud/.local/lib/python3.5/site-packages/openstack/image/v2/_proxy.py", line 371, in download_image
    self, stream=stream, output=output, chunk_size=chunk_size)
  File "/home/ccloud/.local/lib/python3.5/site-packages/openstack/image/_download.py", line 35, in download
    resp = session.get(url, stream=stream)
  File "/home/ccloud/.local/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 386, in get
    return self.request(url, 'GET', **kwargs)
  File "/home/ccloud/.local/lib/python3.5/site-packages/openstack/proxy.py", line 98, in request
    **kwargs)
  File "/home/ccloud/.local/lib/python3.5/site-packages/keystoneauth1/adapter.py", line 248, in request
    return self.session.request(url, method, **kwargs)
  File "/home/ccloud/.local/lib/python3.5/site-packages/keystoneauth1/session.py", line 913, in request
    resp = send(**kwargs)
  File "/home/ccloud/.local/lib/python3.5/site-packages/keystoneauth1/session.py", line 1024, in _send_request
    raise exceptions.UnknownConnectionError(msg, e)
keystoneauth1.exceptions.connection.UnknownConnectionError: Unexpected exception for https://####/v2/images/2d4ab8aa-a1bc-45b6-89cb-50da76dd4f31/file: ('Connection broken: OSError("(104, \'ECONNRESET\')",)', OSError("(104, 'ECONNRESET')",))
clean_up SaveImage: Unexpected exception for https://####/v2/images/2d4ab8aa-a1bc-45b6-89cb-50da76dd4f31/file: ('Connection broken: OSError("(104, \'ECONNRESET\')",)', OSError("(104, 'ECONNRESET')",))
END return value: 1

Launchpad Janitor (janitor) said : #3

This question was expired because it remained in the 'Open' state without activity for the last 15 days.