Are glance images immutable?

Asked by Michael Still on 2012-01-01

Hi. I have a base image (an image in $state_path/instances/_base) whose checksum fails compared with the checksum for the image stored in glance. There is an instance using this image at the moment. As best as I can tell, you can't update an image in glance once its uploaded -- you'd instead upload a new image (and get a new ID). Is that correct?

I am trying to decide if I am looking at genuine corruption, or perhaps someone updated the image since the _base directory was populated?

Thanks,
Mikal

Question information

Language:
English Edit question
Status:
Solved
For:
Glance Edit question
Assignee:
No assignee Edit question
Solved by:
Michael Still
Solved:
2012-01-02
Last query:
2012-01-02
Last reply:
2012-01-01
Jay Pipes (jaypipes) said : #1

Well hello there, Mikal :)

You are absolutely correct about images in Glance. Once an image is uploaded and its checksum generated, it may not be modified -- at least not through any public interface. One would have to manually go into the backend storage and replace the data and/or modify the registry database by hand to change the checksum stored for an image. I believe that the more likely scenario may be that the image in the _base directory was overwritten/modified or corrupted. I see in /nova/libvirt/connection.py that there are multiple checks to ensure that an image placed in the $instances_path/_base/ directory (the local image cache IIRC) are not overwritten if the file already exists, so I think Vish would think what you are describing is a very unlikely scenario (HDD corruption perhaps?)

Lemme know if you find any other clues as to what happened. I'm curious to know if there might be a hole somewhere in Glance that allows a changes like this (I'm skeptical, but always curious ;)

Cheers!
-jay

Michael Still (mikal) said : #2

Hey Jay. I hope 2012 has been good so far...

So, is the checksum in glance the checksum for the expanded image file on disk? Perhaps I've missed a step when verifying the checksums (although some other image files pass). All I've done is a MD5 of the image file in _base, and compared that with the glance checksum. I have more than one image failing, across more than one machine, which makes me think there is something systemic happening here.

Perhaps as a next step I should fetch the image from glance and see if perhaps the checksum differs from what glance has in its datastore?

Thanks,
Mikal

Michael Still (mikal) said : #3

Oh, I think I understand now. Nova compute sometimes converts the format of the images it downloads from glance, and this changes their sizes in the _base directory. For example I wrote a simple python script to fetch raw images from glance (I couldn't immediately find an existing tool to do it), and it returns this for the image in question:

$ ls -lrt
-rw-r--r-- 1 mikal warthogs 233701376 2012-01-02 11:24 291

Which is the right size compare with glance. However, when I convert the image like nova does, it gets a lot bigger:

$ qemu-img convert -O raw 291 291.converted
$ ls -lrt
-rw-r--r-- 1 mikal mikal 233701376 2012-01-02 22:41 291
-rw-r--r-- 1 mikal mikal 2147483648 2012-01-02 23:01 291.converted

This explains the size and checksum mismatches. I think this is going to make the blueprint I am working on a little bit harder, as image verification is going to be a bit difficult now.

Cheers,
Mikal

Vish Ishaya (vishvananda) said : #4

This is correct. There is a potential security issue with using a qcow image as a backing file, so nova will convert the qcow2 raw before using it as a backing file for new images.

ustcdylan (ustcdylan) said : #5

Hi Vish, what's the security issue you speak of ?