Cinder

stay in attaching volume to instances

Asked by Oliver Mueller

Hi,

i have create a OpenStack Grizzly MultiNode Infrastrucutr.

Controlnode management network 192.168.0.1 / external 10.123.4.xxx
Computenode management network 192.168.0.2 / data 10.10.10.2
networknode management network 192.168.0.3 / external 10.123.4.xxx /data 10.10.10.1

I Followed this Install Guide https://github.com/mseknibilel/OpenStack-Grizzly-Install-Guide/blob/OVS_MultiNode/OpenStack_Grizzly_Install_Guide.rst

When i attach a Volume to a existing instaces, in will be staying in attaching since 12 hours. I have reboot the control and compute node. Where could be the problem? Who can help me?

cinder-api.log
2013-04-12 10:09:30 AUDIT [cinder.api.v1.volumes] vol={'volume_metadata': [], 'availability_zone': u'nova', 'terminated_at': None, 'updated_at': datetime.datetime(2013, 4, 11, 14, 48, 12), 'snapshot_id': None, 'ec2_id': None, 'mountpoint': None, 'deleted_at': None, 'id': u'0358222b-3d77-438b-af1a-237db6c22865', 'size': 40L, 'user_id': u'fdcf86f4e7b843a4a69907423d32d073', 'attach_time': None, 'display_description': u'', 'project_id': u'd2a556839afe4e7a9eda53cd0b4e54f0', 'launched_at': datetime.datetime(2013, 4, 11, 14, 44, 31), 'scheduled_at': datetime.datetime(2013, 4, 11, 14, 44, 30), 'status': u'attaching', 'volume_type_id': None, 'deleted': False, 'provider_location': u'10.123.4.101:3260,1 iqn.2010-10.org.openstack:volume-0358222b-3d77-438b-af1a-237db6c22865 0', 'volume_glance_metadata': [], 'host': u'controlnode', 'source_volid': None, 'provider_auth': u'CHAP K9y8DoFPAG8LCTnYHFhH diikBrj9DfTLai92DMP3', 'display_name': u'kvasy', 'instance_uuid': None, 'created_at': datetime.datetime(2013, 4, 11, 14, 44, 30), 'attach_status': u'detached', 'volume_type': None}
2013-04-12 10:09:30 INFO [cinder.api.openstack.wsgi] http://10.123.4.101:8776/v1/d2a556839afe4e7a9eda53cd0b4e54f0/volumes/0358222b-3d77-438b-af1a-237db6c22865 returned with HTTP 200
2013-04-12 10:09:34 INFO [cinder.api.openstack.wsgi] GET http://10.123.4.101:8776/v1/d2a556839afe4e7a9eda53cd0b4e54f0/os-quota-sets/d2a556839afe4e7a9eda53cd0b4e54f0
2013-04-12 10:09:34 INFO [cinder.api.openstack.wsgi] http://10.123.4.101:8776/v1/d2a556839afe4e7a9eda53cd0b4e54f0/os-quota-sets/d2a556839afe4e7a9eda53cd0b4e54f0 returned with HTTP 200
2013-04-12 10:09:34 INFO [cinder.api.openstack.wsgi] GET http://10.123.4.101:8776/v1/d2a556839afe4e7a9eda53cd0b4e54f0/volumes/detail
2013-04-12 10:09:34 AUDIT [cinder.api.v1.volumes] vol=<cinder.db.sqlalchemy.models.Volume object at 0x35b2750>
2013-04-12 10:09:34 INFO [cinder.api.openstack.wsgi] http://10.123.4.101:8776/v1/d2a556839afe4e7a9eda53cd0b4e54f0/volumes/detail returned with HTTP 200
2013-04-12 10:32:02 INFO [cinder.api.openstack.wsgi] GET http://10.123.4.101:8776/v1/d2a556839afe4e7a9eda53cd0b4e54f0/volumes/detail
2013-04-12 10:32:02 AUDIT [cinder.api.v1.volumes] vol=<cinder.db.sqlalchemy.models.Volume object at 0x37e3d90>
2013-04-12 10:32:02 INFO [cinder.api.openstack.wsgi] http://10.123.4.101:8776/v1/d2a556839afe4e7a9eda53cd0b4e54f0/volumes/detail returned with HTTP 200
2013-04-12 10:32:03 INFO [cinder.api.openstack.wsgi] GET http://10.123.4.101:8776/v1/d2a556839afe4e7a9eda53cd0b4e54f0/volumes/0358222b-3d77-438b-af1a-237db6c22865
2013-04-12 10:32:03 AUDIT [cinder.api.v1.volumes] vol={'volume_metadata': [], 'availability_zone': u'nova', 'terminated_at': None, 'updated_at': datetime.datetime(2013, 4, 11, 14, 48, 12), 'snapshot_id': None, 'ec2_id': None, 'mountpoint': None, 'deleted_at': None, 'id': u'0358222b-3d77-438b-af1a-237db6c22865', 'size': 40L, 'user_id': u'fdcf86f4e7b843a4a69907423d32d073', 'attach_time': None, 'display_description': u'', 'project_id': u'd2a556839afe4e7a9eda53cd0b4e54f0', 'launched_at': datetime.datetime(2013, 4, 11, 14, 44, 31), 'scheduled_at': datetime.datetime(2013, 4, 11, 14, 44, 30), 'status': u'attaching', 'volume_type_id': None, 'deleted': False, 'provider_location': u'10.123.4.101:3260,1 iqn.2010-10.org.openstack:volume-0358222b-3d77-438b-af1a-237db6c22865 0', 'volume_glance_metadata': [], 'host': u'controlnode', 'source_volid': None, 'provider_auth': u'CHAP K9y8DoFPAG8LCTnYHFhH diikBrj9DfTLai92DMP3', 'display_name': u'kvasy', 'instance_uuid': None, 'created_at': datetime.datetime(2013, 4, 11, 14, 44, 30), 'attach_status': u'detached', 'volume_type': None}
2013-04-12 10:32:03 INFO [cinder.api.openstack.wsgi] http://10.123.4.101:8776/v1/d2a556839afe4e7a9eda53cd0b4e54f0/volumes/0358222b-3d77-438b-af1a-237db6c22865 returned with HTTP 200
2013-04-12 10:32:05 INFO [cinder.api.openstack.wsgi] GET http://10.123.4.101:8776/v1/d2a556839afe4e7a9eda53cd0b4e54f0/volumes/0358222b-3d77-438b-af1a-237db6c22865
2013-04-12 10:32:05 AUDIT [cinder.api.v1.volumes] vol={'volume_metadata': [], 'availability_zone': u'nova', 'terminated_at': None, 'updated_at': datetime.datetime(2013, 4, 11, 14, 48, 12), 'snapshot_id': None, 'ec2_id': None, 'mountpoint': None, 'deleted_at': None, 'id': u'0358222b-3d77-438b-af1a-237db6c22865', 'size': 40L, 'user_id': u'fdcf86f4e7b843a4a69907423d32d073', 'attach_time': None, 'display_description': u'', 'project_id': u'd2a556839afe4e7a9eda53cd0b4e54f0', 'launched_at': datetime.datetime(2013, 4, 11, 14, 44, 31), 'scheduled_at': datetime.datetime(2013, 4, 11, 14, 44, 30), 'status': u'attaching', 'volume_type_id': None, 'deleted': False, 'provider_location': u'10.123.4.101:3260,1 iqn.2010-10.org.openstack:volume-0358222b-3d77-438b-af1a-237db6c22865 0', 'volume_glance_metadata': [], 'host': u'controlnode', 'source_volid': None, 'provider_auth': u'CHAP K9y8DoFPAG8LCTnYHFhH diikBrj9DfTLai92DMP3', 'display_name': u'kvasy', 'instance_uuid': None, 'created_at': datetime.datetime(2013, 4, 11, 14, 44, 30), 'attach_status': u'detached', 'volume_type': None}
2013-04-12 10:32:05 INFO [cinder.api.openstack.wsgi] http://10.123.4.101:8776/v1/d2a556839afe4e7a9eda53cd0b4e54f0/volumes/0358222b-3d77-438b-af1a-237db6c22865 returned with HTTP 200
2013-04-12 10:32:08 INFO [cinder.api.openstack.wsgi] GET http://10.123.4.101:8776/v1/d2a556839afe4e7a9eda53cd0b4e54f0/volumes/0358222b-3d77-438b-af1a-237db6c22865
2013-04-12 10:32:08 AUDIT [cinder.api.v1.volumes] vol={'volume_metadata': [], 'availability_zone': u'nova', 'terminated_at': None, 'updated_at': datetime.datetime(2013, 4, 11, 14, 48, 12), 'snapshot_id': None, 'ec2_id': None, 'mountpoint': None, 'deleted_at': None, 'id': u'0358222b-3d77-438b-af1a-237db6c22865', 'size': 40L, 'user_id': u'fdcf86f4e7b843a4a69907423d32d073', 'attach_time': None, 'display_description': u'', 'project_id': u'd2a556839afe4e7a9eda53cd0b4e54f0', 'launched_at': datetime.datetime(2013, 4, 11, 14, 44, 31), 'scheduled_at': datetime.datetime(2013, 4, 11, 14, 44, 30), 'status': u'attaching', 'volume_type_id': None, 'deleted': False, 'provider_location': u'10.123.4.101:3260,1 iqn.2010-10.org.openstack:volume-0358222b-3d77-438b-af1a-237db6c22865 0', 'volume_glance_metadata': [], 'host': u'controlnode', 'source_volid': None, 'provider_auth': u'CHAP K9y8DoFPAG8LCTnYHFhH diikBrj9DfTLai92DMP3', 'display_name': u'kvasy', 'instance_uuid': None, 'created_at': datetime.datetime(2013, 4, 11, 14, 44, 30), 'attach_status': u'detached', 'volume_type': None}
2013-04-12 10:32:08 INFO [cinder.api.openstack.wsgi] http://10.123.4.101:8776/v1/d2a556839afe4e7a9eda53cd0b4e54f0/volumes/0358222b-3d77-438b-af1a-237db6c22865 returned with HTTP 200
2013-04-12 10:32:13 INFO [cinder.api.openstack.wsgi] GET http://10.123.4.101:8776/v1/d2a556839afe4e7a9eda53cd0b4e54f0/volumes/0358222b-3d77-438b-af1a-237db6c22865
2013-04-12 10:32:13 AUDIT [cinder.api.v1.volumes] vol={'volume_metadata': [], 'availability_zone': u'nova', 'terminated_at': None, 'updated_at': datetime.datetime(2013, 4, 11, 14, 48, 12), 'snapshot_id': None, 'ec2_id': None, 'mountpoint': None, 'deleted_at': None, 'id': u'0358222b-3d77-438b-af1a-237db6c22865', 'size': 40L, 'user_id': u'fdcf86f4e7b843a4a69907423d32d073', 'attach_time': None, 'display_description': u'', 'project_id': u'd2a556839afe4e7a9eda53cd0b4e54f0', 'launched_at': datetime.datetime(2013, 4, 11, 14, 44, 31), 'scheduled_at': datetime.datetime(2013, 4, 11, 14, 44, 30), 'status': u'attaching', 'volume_type_id': None, 'deleted': False, 'provider_location': u'10.123.4.101:3260,1 iqn.2010-10.org.openstack:volume-0358222b-3d77-438b-af1a-237db6c22865 0', 'volume_glance_metadata': [], 'host': u'controlnode', 'source_volid': None, 'provider_auth': u'CHAP K9y8DoFPAG8LCTnYHFhH diikBrj9DfTLai92DMP3', 'display_name': u'kvasy', 'instance_uuid': None, 'created_at': datetime.datetime(2013, 4, 11, 14, 44, 30), 'attach_status': u'detached', 'volume_type': None}
2013-04-12 10:32:13 INFO [cinder.api.openstack.wsgi] http://10.123.4.101:8776/v1/d2a556839afe4e7a9eda53cd0b4e54f0/volumes/0358222b-3d77-438b-af1a-237db6c22865 returned with HTTP 200
2013-04-12 10:32:20 INFO [cinder.api.openstack.wsgi] GET http://10.123.4.101:8776/v1/d2a556839afe4e7a9eda53cd0b4e54f0/volumes/0358222b-3d77-438b-af1a-237db6c22865
2013-04-12 10:32:20 AUDIT [cinder.api.v1.volumes] vol={'volume_metadata': [], 'availability_zone': u'nova', 'terminated_at': None, 'updated_at': datetime.datetime(2013, 4, 11, 14, 48, 12), 'snapshot_id': None, 'ec2_id': None, 'mountpoint': None, 'deleted_at': None, 'id': u'0358222b-3d77-438b-af1a-237db6c22865', 'size': 40L, 'user_id': u'fdcf86f4e7b843a4a69907423d32d073', 'attach_time': None, 'display_description': u'', 'project_id': u'd2a556839afe4e7a9eda53cd0b4e54f0', 'launched_at': datetime.datetime(2013, 4, 11, 14, 44, 31), 'scheduled_at': datetime.datetime(2013, 4, 11, 14, 44, 30), 'status': u'attaching', 'volume_type_id': None, 'deleted': False, 'provider_location': u'10.123.4.101:3260,1 iqn.2010-10.org.openstack:volume-0358222b-3d77-438b-af1a-237db6c22865 0', 'volume_glance_metadata': [], 'host': u'controlnode', 'source_volid': None, 'provider_auth': u'CHAP K9y8DoFPAG8LCTnYHFhH diikBrj9DfTLai92DMP3', 'display_name': u'kvasy', 'instance_uuid': None, 'created_at': datetime.datetime(2013, 4, 11, 14, 44, 30), 'attach_status': u'detached', 'volume_type': None}
2013-04-12 10:32:20 INFO [cinder.api.openstack.wsgi] http://10.123.4.101:8776/v1/d2a556839afe4e7a9eda53cd0b4e54f0/volumes/0358222b-3d77-438b-af1a-237db6c22865 returned with HTTP 200
2013-04-12 10:32:30 INFO [cinder.api.openstack.wsgi] GET http://10.123.4.101:8776/v1/d2a556839afe4e7a9eda53cd0b4e54f0/volumes/0358222b-3d77-438b-af1a-237db6c22865
2013-04-12 10:32:30 AUDIT [cinder.api.v1.volumes] vol={'volume_metadata': [], 'availability_zone': u'nova', 'terminated_at': None, 'updated_at': datetime.datetime(2013, 4, 11, 14, 48, 12), 'snapshot_id': None, 'ec2_id': None, 'mountpoint': None, 'deleted_at': None, 'id': u'0358222b-3d77-438b-af1a-237db6c22865', 'size': 40L, 'user_id': u'fdcf86f4e7b843a4a69907423d32d073', 'attach_time': None, 'display_description': u'', 'project_id': u'd2a556839afe4e7a9eda53cd0b4e54f0', 'launched_at': datetime.datetime(2013, 4, 11, 14, 44, 31), 'scheduled_at': datetime.datetime(2013, 4, 11, 14, 44, 30), 'status': u'attaching', 'volume_type_id': None, 'deleted': False, 'provider_location': u'10.123.4.101:3260,1 iqn.2010-10.org.openstack:volume-0358222b-3d77-438b-af1a-237db6c22865 0', 'volume_glance_metadata': [], 'host': u'controlnode', 'source_volid': None, 'provider_auth': u'CHAP K9y8DoFPAG8LCTnYHFhH diikBrj9DfTLai92DMP3', 'display_name': u'kvasy', 'instance_uuid': None, 'created_at': datetime.datetime(2013, 4, 11, 14, 44, 30), 'attach_status': u'detached', 'volume_type': None}
2013-04-12 10:32:30 INFO [cinder.api.openstack.wsgi] http://10.123.4.101:8776/v1/d2a556839afe4e7a9eda53cd0b4e54f0/volumes/0358222b-3d77-438b-af1a-237db6c22865 returned with HTTP 200
2013-04-12 10:32:43 INFO [cinder.api.openstack.wsgi] GET http://10.123.4.101:8776/v1/d2a556839afe4e7a9eda53cd0b4e54f0/volumes/0358222b-3d77-438b-af1a-237db6c22865
2013-04-12 10:32:43 AUDIT [cinder.api.v1.volumes] vol={'volume_metadata': [], 'availability_zone': u'nova', 'terminated_at': None, 'updated_at': datetime.datetime(2013, 4, 11, 14, 48, 12), 'snapshot_id': None, 'ec2_id': None, 'mountpoint': None, 'deleted_at': None, 'id': u'0358222b-3d77-438b-af1a-237db6c22865', 'size': 40L, 'user_id': u'fdcf86f4e7b843a4a69907423d32d073', 'attach_time': None, 'display_description': u'', 'project_id': u'd2a556839afe4e7a9eda53cd0b4e54f0', 'launched_at': datetime.datetime(2013, 4, 11, 14, 44, 31), 'scheduled_at': datetime.datetime(2013, 4, 11, 14, 44, 30), 'status': u'attaching', 'volume_type_id': None, 'deleted': False, 'provider_location': u'10.123.4.101:3260,1 iqn.2010-10.org.openstack:volume-0358222b-3d77-438b-af1a-237db6c22865 0', 'volume_glance_metadata': [], 'host': u'controlnode', 'source_volid': None, 'provider_auth': u'CHAP K9y8DoFPAG8LCTnYHFhH diikBrj9DfTLai92DMP3', 'display_name': u'kvasy', 'instance_uuid': None, 'created_at': datetime.datetime(2013, 4, 11, 14, 44, 30), 'attach_status': u'detached', 'volume_type': None}
2013-04-12 10:32:43 INFO [cinder.api.openstack.wsgi] http://10.123.4.101:8776/v1/d2a556839afe4e7a9eda53cd0b4e54f0/volumes/0358222b-3d77-438b-af1a-237db6c22865 returned with HTTP 200
2013-04-12 10:32:58 INFO [cinder.api.openstack.wsgi] GET http://10.123.4.101:8776/v1/d2a556839afe4e7a9eda53cd0b4e54f0/volumes/0358222b-3d77-438b-af1a-237db6c22865
2013-04-12 10:32:58 AUDIT [cinder.api.v1.volumes] vol={'volume_metadata': [], 'availability_zone': u'nova', 'terminated_at': None, 'updated_at': datetime.datetime(2013, 4, 11, 14, 48, 12), 'snapshot_id': None, 'ec2_id': None, 'mountpoint': None, 'deleted_at': None, 'id': u'0358222b-3d77-438b-af1a-237db6c22865', 'size': 40L, 'user_id': u'fdcf86f4e7b843a4a69907423d32d073', 'attach_time': None, 'display_description': u'', 'project_id': u'd2a556839afe4e7a9eda53cd0b4e54f0', 'launched_at': datetime.datetime(2013, 4, 11, 14, 44, 31), 'scheduled_at': datetime.datetime(2013, 4, 11, 14, 44, 30), 'status': u'attaching', 'volume_type_id': None, 'deleted': False, 'provider_location': u'10.123.4.101:3260,1 iqn.2010-10.org.openstack:volume-0358222b-3d77-438b-af1a-237db6c22865 0', 'volume_glance_metadata': [], 'host': u'controlnode', 'source_volid': None, 'provider_auth': u'CHAP K9y8DoFPAG8LCTnYHFhH diikBrj9DfTLai92DMP3', 'display_name': u'kvasy', 'instance_uuid': None, 'created_at': datetime.datetime(2013, 4, 11, 14, 44, 30), 'attach_status': u'detached', 'volume_type': None}
2013-04-12 10:32:58 INFO [cinder.api.openstack.wsgi] http://10.123.4.101:8776/v1/d2a556839afe4e7a9eda53cd0b4e54f0/volumes/0358222b-3d77-438b-af1a-237db6c22865 returned with HTTP 200

cinder-scheduler.log
2013-04-11 15:55:33 AUDIT [cinder.service] Starting cinder-scheduler node (version 2013.1)
2013-04-11 15:55:33 INFO [cinder.openstack.common.rpc.common] Connected to AMQP server on localhost:5672
2013-04-11 15:55:33 INFO [cinder.openstack.common.rpc.common] Connected to AMQP server on localhost:5672
2013-04-11 16:02:06 AUDIT [cinder.service] SIGTERM received
2013-04-11 16:02:06 AUDIT [cinder.service] Starting cinder-scheduler node (version 2013.1)
2013-04-11 16:02:06 INFO [cinder.openstack.common.rpc.common] Connected to AMQP server on localhost:5672
2013-04-11 16:02:06 INFO [cinder.openstack.common.rpc.common] Connected to AMQP server on localhost:5672
2013-04-12 08:29:12 AUDIT [cinder.service] SIGTERM received
2013-04-12 08:29:29 AUDIT [cinder.service] Starting cinder-scheduler node (version 2013.1)
2013-04-12 08:29:29 ERROR [cinder.openstack.common.rpc.common] AMQP server on localhost:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2013-04-12 08:32:05 INFO [cinder.openstack.common.rpc.common] Reconnecting to AMQP server on localhost:5672
2013-04-12 08:32:05 INFO [cinder.openstack.common.rpc.common] Connected to AMQP server on localhost:5672
2013-04-12 08:32:05 INFO [cinder.openstack.common.rpc.common] Connected to AMQP server on localhost:5672

cinder-volume.log
2013-04-12 10:15:48 INFO [cinder.volume.manager] Updating volume status
2013-04-12 10:16:48 INFO [cinder.volume.manager] Updating volume status
2013-04-12 10:17:48 INFO [cinder.volume.manager] Updating volume status
2013-04-12 10:18:48 INFO [cinder.volume.manager] Updating volume status
2013-04-12 10:19:48 INFO [cinder.volume.manager] Updating volume status
2013-04-12 10:20:48 INFO [cinder.volume.manager] Updating volume status
2013-04-12 10:21:48 INFO [cinder.volume.manager] Updating volume status
2013-04-12 10:22:48 INFO [cinder.volume.manager] Updating volume status
2013-04-12 10:23:48 INFO [cinder.volume.manager] Updating volume status
2013-04-12 10:24:49 INFO [cinder.volume.manager] Updating volume status
2013-04-12 10:25:49 INFO [cinder.volume.manager] Updating volume status
2013-04-12 10:26:49 INFO [cinder.volume.manager] Updating volume status
2013-04-12 10:27:49 INFO [cinder.volume.manager] Updating volume status
2013-04-12 10:28:49 INFO [cinder.volume.manager] Updating volume status
2013-04-12 10:29:49 INFO [cinder.volume.manager] Updating volume status
2013-04-12 10:30:49 INFO [cinder.volume.manager] Updating volume status
2013-04-12 10:31:49 INFO [cinder.volume.manager] Updating volume status
2013-04-12 10:32:49 INFO [cinder.volume.manager] Updating volume status

cinder.conf
[DEFAULT]
rootwrap_config = /etc/cinder/rootwrap.conf
api_paste_confg = /etc/cinder/api-paste.ini
iscsi_helper = ietadm
volume_name_template = volume-%s
volume_group = cinder-volumes
verbose = True
auth_strategy = keystone
state_path = /var/lib/cinder
lock_path = /var/lock/cinder
volumes_dir = /var/lib/cinder/volumes
sql_connection = mysql://cinder:password@192.168.0.1/cinder

Question information

Language:
English Edit question
Status:
Solved
For:
Cinder Edit question
Assignee:
No assignee Edit question
Solved by:
Oliver Mueller
Solved:
Last query:
Last reply:
Revision history for this message
Oliver Mueller (ollimaller) said : 		#1

Another problem that i have, i can't start instances with Volumes. The instances going directly into a error status after starting.

Revision history for this message
Oliver Mueller (ollimaller) said : 		#2

root@controlnode:~# cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
 +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| 45d6f5b6-e623-41d4-b8c3-994c176a6b93 | creating | admin_test | 45 | Performance | false | |
| a12d50c8-f7e5-46f8-9e9f-4d0a0fb2c50e | available | admin_test2 | 30 | backup | false | |
| b365a120-eccd-4b73-8b2e-cf95606addf1 | available | admin_test3 | 15 | Performance | false | |
 +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
root@controlnode:~# locate iqn
root@controlnode:~#

Revision history for this message
Oliver Mueller (ollimaller) said : 		#3

Adding this line to compute node's nova.conf helped resolve the issue.
cinder_catalog_info=volume:cinder:internalURL