OpenStack Compute (nova)

VM image caching on nova-compute issue

Asked by Graham Hemingway

Recently I was bitten by a Glance issue/bug (see https://bugs.launchpad.net/nova/+bug/773412) that caused me to not think I had any images. So, I cleaned out Glance's DB and removed all of the image files and buckets. I then republished (uec-publish-tarball) several of the images and tried to get instances running. These all failed during calls to qemu-nbd or tune2fs during key injection. Much head scratching.

I finally looked and saw the image files that the nova-compute process was pulling from were much older than the newly uploaded images (i.e. nova-compute wasn't made aware that I had cleared out Glance). I think that nova-compute sees a request for an instance with image #00000001, sees that it has a cached version of #00000001 (in /var/lib/nova/instance/_base) and tries to run it even though this image might be very out of date.

I got things working by deleting all of the files in /var/lib/nova/instances/_base. Then things started working better. I still have some key injection failures, but I at least get running instances.

So, do I have this caching thing right? Is this an issue worth addressing?

Thanks,
   Graham

Question information

Language:
English Edit question
Status:
Expired
For:
OpenStack Compute (nova) Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Jay Pipes (jaypipes) said : 		#1

Hey Graham,

So, it's not actually a Glance bug -- it's a Nova bug in the S3ImageService we think. Kevin Bringard, Vish and I took a look at this while we were at the design summit last week and we'll probably have a chat about it tomorrow. Sorry, was away Sunday and today. Promise tomorrow we will finalize where we think the problem is and I will post an FAQ about the relationship between nova-objectstore and Glance, which I totally agree is not well documented.

Cheers,
jay

Revision history for this message
Launchpad Janitor (janitor) said : 		#2

This question was expired because it remained in the 'Open' state without activity for the last 15 days.