OpenStack Object Storage (swift)

ETag of manifest file

Asked by Matt Mao

I find the following description from http://docs.openstack.org/developer/swift/overview_large_objects.htm
“The response’s ETag for a GET or HEAD on the manifest file will be the MD5 sum of the concatenated string of ETags for each of the segments in the <container>/<prefix> listing, dynamically.”

So I upload two segment files and a manifest file, then I want to use the ETag of segment file to calculate the ETag of manifest file.
a) segment file1:
          ETag: c4ca4238a0b923820dcc509a6f75849b
b) segment file2:
          ETag: c81e728d9d4c2f636f067f89cc14862c
c) So I think the ETag of manifest file should be 02e26c67c011b6b0914cfd3f5cac506f：
      #cat manifestETag
          c4ca4238a0b923820dcc509a6f75849bc81e728d9d4c2f636f067f89cc14862c
          #md5sum manifestETag
          02e26c67c011b6b0914cfd3f5cac506f manifestETag

But in face the ETag of manifest is 302cbafc0dfbc97f30d576a6f394dad3.

So I think the ETag of manifest should be “02e26c67c011b6b0914cfd3f5cac506f”,
But in fact it is 302cbafc0dfbc97f30d576a6f394dad3.

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Object Storage (swift) Edit question
Assignee:
No assignee Edit question
Solved by:
David Goetz
Solved:
Last query:
Last reply:
Revision history for this message
Best David Goetz (david-goetz) said : 		#1

I think you just made a mistake somewhere- I'm getting this:

In [1]: from hashlib import md5

In [2]: a = 'c4ca4238a0b923820dcc509a6f75849b'

In [3]: b = 'c81e728d9d4c2f636f067f89cc14862c'

In [4]: md5(a + b).hexdigest()
Out[4]: '302cbafc0dfbc97f30d576a6f394dad3'

Revision history for this message
Matt Mao (maoliping455) said : 		#2

Thanks David Goetz, that solved my question.

Revision history for this message
simo90 (s-decrescente) said : 		#3

Hi,
i want to know if checking the integrity of an object with the MD5 hash file with the auditor process is doing by Swift also for objects smaller than 5GB or only for large objects. Excuse me for my English but i'm italian.

Revision history for this message
Christian Schwede (cschwede) said : 		#4

Yes, auditors will check all objects, no matter their size.

Revision history for this message
simo90 (s-decrescente) said : 		#5

Thanks for the reply!excuse me but i have another question.
I tried to put an object(a file text) and then i corrupted its data by hand changing some letters in it. When the object-auditor checks the file's integrity,it doesn't check by using the MD5 hash file but by the different dimension of the new corrupted file. Can you tell me why??

Revision history for this message
Christian Schwede (cschwede) said : 		#6

You mean you saw something like this in your logfiles?

 Apr 20 08:37:55 saio object-6020: ERROR Object /srv/node2/sdb2/objects/742/813/b9be8d1ce215338587e9bd48c089b813 failed audit and was quarantined: metadata content-length 6 does not match actual object size 8

That happens if the file size changed; it's much "cheaper" to compare the size first, and quarantine the object if there is a mismatch. If the size didn't change, a md5 comparison will be done afterwards, and you should see a line similar to this if the content is different:

 Apr 20 08:40:10 saio object-6020: Quarantined object /srv/node2/sdb2/objects/742/813/b9be8d1ce215338587e9bd48c089b813/1461141581.50282.data: ETag b1946ac92492d2347c6235b4d2611184 and file's md5 aee97cb3ad288ef0add6c6b5b5fae48a do not match

Revision history for this message
simo90 (s-decrescente) said : 		#7

Yes, i see that.
 But i have changed only a letter. For example in my text at the beginning there is 'blablb' .
Then ,with an istruction of echo, i replaced 'blabla' . Why is the file size changed?