OpenStack Object Storage (swift)

Is max object size 5GB+2byte?

Asked by Keisuke Yamamoto

Hello,

Could anyone please answer my question on the maximum size of a single object on Swift?
"Swift 1.7.6-dev documentation >> Large Object Support > Overview" says the default maximum object size is "5GB", however I have successfully uploaded a 5GB+2byte(5368709122 byte) object.
It also seems that the source says max file size is 5GB+2byte. (see below) What is this "+ 2" for?

Thanks,
Keisuke Yamamoto

Source code:
swift-1.7.4/swift/common/constraints.py
https://github.com/openstack/swift/blob/master/swift/common/constraints.py

------------------------swift/common/constraints.py #L35-37---------------------------------------

#: Max file size allowed for objects
MAX_FILE_SIZE = constraints_conf_int('max_file_size',
                                     5368709122) # 5 * 1024 * 1024 * 1024 + 2
                                                                                                                ^^^
--------------------------------------------------------------------------------------------------

--------------------------------upload example----------------------------------------------------
# ls -l 1_2.txt
-rw-r--r-- 1 root root 5368709122 Sep 10 18:48 1_2.txt

# curl -v -X PUT -H "X-Auth-Token: AUTH_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" http://xxx.xxx.xxx.xxx/v1.0/AUTH_ACCOUNT0/bucket0/object_1_2 -T 1_2.txt
* About to connect() to xxx.xxx.xxx.xxx port 8080
* Trying xxx.xxx.xxx.xxx... connected
* Connected to xxx.xxx.xxx.xxx (xxx.xxx.xxx.xxx) port 8080
> PUT /v1.0/AUTH_ACCOUNT0/bucket0/object_1_2 HTTP/1.1
> User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
> Host: xxx.xxx.xxx.xxx:8080
> Accept: */*
> X-Auth-Token: AUTH_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> Content-Length: 5368709122
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
HTTP/1.1 201 Created
< Content-Length: 118
< Content-Type: text/html; charset=UTF-8
< Etag: 255a618b796bd4ee3f21d6f432357d81
< Last-Modified: Mon, 10 Sep 2012 10:37:06 GMT
< X-Trans-Id: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
< Date: Mon, 10 Sep 2012 10:59:20 GMT
<html>
 <head>
  <title>201 Created</title>
 </head>
 <body>
  <h1>201 Created</h1>
  <br /><br />

 </body>
Connection #0 to host xxx.xxx.xxx.xxx left intact
* Closing connection #0
</html>
-------------------------------------------------------------------------------------------------

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Object Storage (swift) Edit question
Assignee:
No assignee Edit question
Solved by:
John Dickinson
Solved:
Last query:
Last reply:
Revision history for this message
Best John Dickinson (notmyname) said : 		#1

Historical reasons, and somewhat of an inside joke.

AWS launched S3 with a 5GB limit. Rackspace Cloud Files (pre-swift) launched with 5GB+1 bytes (because that's better, right?). Swift (developed internally at Rackspace to replace the existing Cloud Files storage engine) was set at 5GB+2 bytes. Swift was then open-sourced as part of OpenStack, and the default persists to this day.

Revision history for this message
Keisuke Yamamoto (keisuk-yamamoto) said : 		#3

Thanks John Dickinson, that solved my question.