Volume says it is attached, but doesn't show up in guest
Cactus nova. Can create volumes. Doing euca-attach-volume appears to work; reports that the volume is in-use etc. But when I log into the guest, there's no vdz in /dev, and sudo fdisk -l doesn't show it either. Suggestions for what to look for in logs appreciated :-) I've been trawling them but don't see anything obviously wrong to me yet. Rebooted the guest, didn't help. Thanks...
// the first 3 below are the bad ones from my last question, dunno how to get rid of them.
root@cc-vm-3:~# euca-describe-
VOLUME vol-00000001 1 nova creating (default-project, None, None, None) 2011-07-
VOLUME vol-00000002 1 nova creating (default-project, None, None, None) 2011-07-
VOLUME vol-00000003 1 nova creating (default-project, None, None, None) 2011-07-
VOLUME vol-00000004 1 nova available (default-project, cc-vm-3, None, None) 2011-07-
VOLUME vol-00000005 1 nova in-use (default-project, cc-vm-3, i-00000016[
Question information
- Language:
- English Edit question
- Status:
- Answered
- Assignee:
- No assignee Edit question
- Last query:
- Last reply:
Revision history for this message
|
#1 |
Possibly relevant log extract from the compute host:
2011-07-22 17:34:35,396 DEBUG nova.rpc [-] received {u'_context_
z', u'volume_id': 5}, u'_context_
ject', u'_context_
2011-07-22 17:34:35,397 DEBUG nova.rpc [-] unpacked context: {'timestamp': u'2011-
u'admin', 'request_id': u'MFXGNIG4RLVB9
2011-07-22 17:34:35,397 INFO nova.compute.
2011-07-22 17:34:35,398 INFO nova.compute.
nova.context.
2011-07-22 17:34:35,398 DEBUG nova.compute.
/compute/
2011-07-22 17:34:35,455 INFO nova.compute.
2011-07-22 17:34:35,455 INFO nova.compute.
2011-07-22 17:34:35,455 INFO nova.compute.
2011-07-22 17:34:35,498 AUDIT nova.compute.
2011-07-22 17:34:35,509 WARNING nova.volume.driver [-] ISCSI provider_location not stored, using discovery
2011-07-22 17:34:35,510 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m discovery -t sendtargets -p cc-vm-3 from (pid=27378) execute /usr/lib/
2011-07-22 17:34:35,544 DEBUG nova.volume.driver [-] ISCSI Discovery: Found 10.4.0.24:3260,1 iqn.2010-
2011-07-22 17:34:35,544 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-
2011-07-22 17:34:36,068 DEBUG nova.volume.driver [-] iscsiadm --login: stdout=Logging in to [iface: default, target: iqn.2010-
Login to [iface: default, target: iqn.2010-
stderr= from (pid=27378) _run_iscsiadm /usr/lib/
2011-07-22 17:34:36,069 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-
2011-07-22 17:34:36,080 DEBUG nova.volume.driver [-] iscsiadm ('--op', 'update', '-n', 'node.startup', '-v', 'automatic'):
BEGIN RECORD 2.0-871
node.name = iqn.2010-
node.tpgt = 1
node.startup = manual
iface.hwaddress = <empty>
iface.ipaddress = <empty>
iface.iscsi_
iface.net_ifacename = <empty>
iface.transport
iface.initiatorname = <empty>
node.discovery_
node.discovery_port = 3260
node.discovery_type = send_targets
node.session.
node.session.
node.session.
node.session.
node.session.
node.session.
node.session.
node.session.
node.session.
node.session.
node.session.
node.session.
node.session.
node.session.
node.session.
node.session.
node.session.
node.session.
node.session.
node.session.
node.session.
node.session.
node.session.
node.session.
node.conn[
node.conn[0].port = 3260
node.conn[
node.conn[
node.conn[
node.conn[
node.conn[
node.conn[
node.conn[
node.conn[
node.conn[
node.conn[
node.conn[
node.conn[
node.conn[
# END RECORD
stderr= from (pid=27378) _run_iscsiadm /usr/lib/
2011-07-22 17:34:36,094 WARNING nova.volume.driver [-] ISCSI volume not yet found at: /dev/disk/
& retry. Try number: 0
2011-07-22 17:34:36,094 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2010-
/usr/lib/
2011-07-22 17:34:36,110 DEBUG nova.volume.driver [-] iscsiadm --rescan: stdout=Rescanning session [sid: 3, target: iqn.2010-
stderr= from (pid=27378) _run_iscsiadm /usr/lib/
2011-07-22 17:34:37,111 DEBUG nova.volume.driver [-] Found iSCSI node /dev/disk/
7378) discover_volume /usr/lib/
2011-07-22 17:41:19,832 DEBUG nova.rpc [-] received {u'_msg_id': u'66d56ec141c94
s': {u'instance_id': 22}, u'_context_
2011-07-22 17:41:19,833 DEBUG nova.rpc [-] unpacked context: {'timestamp': u'2011-
Revision history for this message
|
#2 |
root@cnode-
10.4.0.24:3260,1 iqn.2010-
10.4.0.24:3260,1 iqn.2010-
so they are visible to the compute host it seems, so I'm guessing there's something about getting those mapped up and into the guest that isn't working & that I don't yet understand.
Revision history for this message
|
#3 |
(p.s. I experimented with 2 volumes. Neither showed up. The first one I could detach. The second one throws errors when detaching, and never shows up as detached. Furthermore, it appears to prevent the instance from being correctly terminated, so the instance won't fully go away. The error is File "/usr/lib/
Revision history for this message
|
#4 |
(p.p.s. this is with kvm if that matters.)
Revision history for this message
|
#5 |
Fyi, the volume will not be at /dev/vdz. That is the attach point from the hypervisors view, but the guest will pick its own name. it will show up in the next available slot (usually /dev/vdc)
On Jul 22, 2011, at 2:26 PM, Jon Slenk wrote:
> Question #165695 on OpenStack Compute (nova) changed:
> https:/
>
> Jon Slenk gave more information on the question:
> (p.p.s. this is with kvm if that matters.)
>
> --
> You received this question notification because you are a member of Nova
> Core, which is an answer contact for OpenStack Compute (nova).
Revision history for this message
|
#6 |
Huh. I'm pretty surprised and confused! Sorry if I've totally misunderstood things.
The man page for euca-attach-volume says "-d local device name (inside the guest VM) to use." guest, not host. The docs http://
So I just retried, and the controller could send out the attach, and things showed up in the host logs, but the host does not show any new /dev/vdq i called it this time. There are no /dev/vd* anythings. Looking at syslog, I think if anything it showed up as /dev/sdd or maybe /dev/sdf on the host. BTW, the vg wasn't ever formatted & given a fs, if that is something that can confuse iscsi/nova-volume?
At any rate, all in all now I'm even more befuddled about the expected names of things.
(Looking at the Cactus code, it appears to me (but obviously I don't know what I'm doing) that compute/
Revision history for this message
|
#7 |
Sorry for the confusion.
Unfortunately this is a KVM limitation. You can specify a device name to the KVM hypervisor, but the actual means of attach to the guest is over a virtual pci bus. When the guest sees a new device on the pci bus, it picks the next available name (which in most cases is /dev/vdc) and the disk will show up there. If you then detach this disk and attach another one it will show up as /dev/vdd, etc.. The device name is required on the hypervisor side so it can be detached later, but the device name that the hypervisor reports is completely different from the device name in the guest.
Remember that euca is designed to work with AWS which is using xen. Xen will more reliably attach a device to a specific name, so in that case the names are consistent. I've considered a possible workaround in the guest of a udev rule that could query metadata to find the requested attach point and make a symlink to the guest device, but this has not been created yet.
Perhaps this issue needs to be clarified in the docs.
Vish
On Jul 22, 2011, at 3:15 PM, Jon Slenk wrote:
> Question #165695 on OpenStack Compute (nova) changed:
> https:/
>
> Status: Answered => Open
>
> Jon Slenk is still having a problem:
> Huh. I'm pretty surprised and confused! Sorry if I've totally
> misunderstood things.
>
> The man page for euca-attach-volume says "-d local device name (inside
> the guest VM) to use." guest, not host. The docs
> http://
> compute/
> the same device name in the guest. If it is truly true that the guest
> sees some effectively-
> device name, it would be great to call that out in the docs. :-} (To
> reiterate, we're on Cactus, in case something about that behaviour has
> changed since then.)
>
> So I just retried, and the controller could send out the attach, and
> things showed up in the host logs, but the host does not show any new
> /dev/vdq i called it this time. There are no /dev/vd* anythings. Looking
> at syslog, I think if anything it showed up as /dev/sdd or maybe
> /dev/sdf on the host. BTW, the vg wasn't ever formatted & given a fs, if
> that is something that can confuse iscsi/nova-volume?
>
> At any rate, all in all now I'm even more befuddled about the expected
> names of things.
>
> (Looking at the Cactus code, it appears to me (but obviously I don't
> know what I'm doing) that compute/
> device name as supplied by the euca-attach-volume command in the RPC
> cast, and looking at virt/libvirt_
> getting the same value "/dev/vdq" for example per what I see in the
> host's nova-compute.log. But I've never seen it show up on either the
> host or guest as that yet.)
>
> --
> You received this question notification because you are a member of Nova
> Core, which is an answer contact for OpenStack Compute (nova).
Can you help with this problem?
Provide an answer of your own, or ask Jon Slenk for more information if necessary.