Cinder

Unable to create volume using cinder-volume on a separate node without public url

Asked by Mohan on 2013-05-30

Hi All,
Greetings, i need your help and advice regarding an issue that i am facing with cinder

I am unable to create a volume using cinder . I have cinder-volume which is on another machine. The cinder-api and the cinder-scheduler are on the controller machine.

For setting up the controller and the nova-nodes , my reference has been https://github.com/mseknibilel/OpenStack-Grizzly-Install-Guide/blob/OVS_MultiNode/OpenStack_Grizzly_Install_Guide.rst
and for setting up the cinder-volume on another machine, my reference is http://www.binaryroyale.com/index.php/2013/04/openstack-grizzly-creating-a-cinder-only-block-storage-node-standalone/, which i have followed except the contents of the interface file

10.0.0.11 is the internal ip address of the contoller, on which i have cinder-api,cinder-scheduler, keystone and other services.

10.0.0.15 is the internal ip address of the cinder-volume machine, I also have nova running on this machine

The details for the cinder-volumes machine is as follows:
Contents of /etc/network/interfaces:
# The primary network interface
auto eth0
iface eth0 inet static
        address 10.0.0.15
        netmask 255.255.255.0
        network 10.0.0.0
        broadcast 10.0.0.255
        gateway 10.0.0.1
        # dns-* options are implemented by the resolvconf package, if installed
        dns-nameservers 10.0.0.1
        dns-search ubuntu
auto eth1
iface eth1 inet static
        address 11.0.0.6
        netmask 255.255.255.0
        network 11.0.0.0
        broadcast 10.0.0.255

Contents of /etc/cinder/cinder.conf:
[DEFAULT]
rootwrap_config = /etc/cinder/rootwrap.conf
sql_connection = mysql://cinderUser:cinderPass@10.0.0.11/cinder
api_paste_confg = /etc/cinder/api-paste.ini
iscsi_helper = tgtadm
volume_name_template = volume-%s
volume_group = cinder-volumes
verbose = True
auth_strategy = keystone
state_path = /var/lib/cinder
lock_path = /var/lock/cinder
volumes_dir = /var/lib/cinder/volumes
volume_driver = cinder.volume.drivers.lvm.LVMISCSIDriver

rabbit_host = 10.0.0.11
rabbit_password = guest
rabbit_port = 5672
rabbit_userid = guest
rabbit_virtual_host = /

Contents of /etc/cinder/api-paste.ini:
[filter:authtoken]
#paste.filter_factory = keystone.middleware.auth_token:filter_factory
paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory
service_protocol = http
#service_host = 127.0.0.1
service_host = 10.0.0.11
service_port = 5000
#auth_host = 127.0.0.1
auth_host = 10.0.0.11
auth_port = 35357
auth_protocol = http
admin_tenant_name = service
admin_user = cinder
admin_password = service_pass
signing_dir = /var/lib/cinder
#admin_tenant_name = %SERVICE_TENANT_NAME%
#admin_user = %SERVICE_USER%
#admin_password = %SERVICE_PASSWORD%

Info on Volume group
$vgdisplay
  --- Volume group ---
  VG Name cinder-volumes
  System ID
  Format lvm2
  Metadata Areas 1
  Metadata Sequence No 1
  VG Access read/write
  VG Status resizable
  MAX LV 0
  Cur LV 0
  Open LV 0
  Max PV 0
  Cur PV 1
  Act PV 1
  VG Size 1.82 TiB
  PE Size 4.00 MiB
  Total PE 476671
  Alloc PE / Size 0 / 0
  Free PE / Size 476671 / 1.82 TiB
  VG UUID A0LGS0-4DP6-GF63-rpaH-V188-hTVH-Say2bI

On looking at /var/log/cinder/cinder-volume.log, the last section i see
root@compute-2:/var/log/cinder# cat cinder-volume.log.1 | tail
2013-05-30 00:48:36 27950 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 147, in __init__
2013-05-30 00:48:36 27950 TRACE cinder self.manager = manager_class(host=self.host, *args, **kwargs)
2013-05-30 00:48:36 27950 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 81, in __init__
2013-05-30 00:48:36 27950 TRACE cinder self.driver = importutils.import_object(volume_driver)
2013-05-30 00:48:36 27950 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/openstack/common/importutils.py", line 40, in import_object
2013-05-30 00:48:36 27950 TRACE cinder return import_class(import_str)(*args, **kwargs)
2013-05-30 00:48:36 27950 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/openstack/common/importutils.py", line 30, in import_class
2013-05-30 00:48:36 27950 TRACE cinder __import__(mod_str)
2013-05-30 00:48:36 27950 TRACE cinder ImportError: No module named drivers.lvm
2013-05-30 00:48:36 27950 TRACE cinder

I could not understand the reason for the above ImportError, what could be the reason, and what is the impact of the above error?
This is not my main issue, but a reason to worry nonetheless

On the Controller machine:
Contents of /etc/network/interfaces:
auto lo
iface lo inet loopback

auto lo:0
iface lo:0 inet loopback
address 115.119.76.176

# The primary network interface
auto eth1
iface eth1 inet static
address 10.0.0.11
netmask 255.255.255.0
gateway 10.0.0.1
dns-nameservers 8.8.8.8

Keystone entries are as follows:

For cinder-volume, with port number 8776, i tried creating an endpoint without the publicurl, but the database got corrupted, so i used 10.0.0.15(the cinder-volume machine), now for $cinder list, i get a HTTP 400 error
So i gave the public ip address of http://115.119.76.176, and there is not way i am able to send apis to the cinder-volume. How do i solve this problem?

Please help.
Regards
Mohan

Question information

Language:: English Edit question

Status:: Solved

For:: Cinder Edit question

Assignee:: No assignee Edit question

Solved by:: Mohan

Solved:: 2013-06-25

Last query:: 2013-06-25

Last reply:: 2013-06-25

Link existing bug

Revision history for this message

Shanthakumar K (shantha-kumar) said on 2013-05-31:

Have a look at this link https://answers.launchpad.net/cinder/+question/226003,

1. whether your nodes are in sync ?
2. whether you able to see the cinder host from the cinder-manage host list command.

Please do share cinder-scheduler log for more info

Revision history for this message

Mohan (v-mohan) said on 2013-06-09:

Hi Shausy,
Thank you for answering my query, apologies for the late reply.

I had a look at the link you reffered, i restarted ntp(i am assuming that resync can be done with this) and there is no change in the behaviour.

Just to bring some context to this reply. I briefly explain my set-up and issue.
1. I have a controller running on 10.0.0.11, which has cinder-api, and cinder-scheduler running as well. I also have cinder-volume, but i do not want to use this. I have the other required services running on this.
2. I have another node running nova-compute and cinder-volume. This is on 10.0.0.15

I have modified the end-point for the cinder-volume according in the keystone DB, the output of $keystone endpoint-list is below

Id 6d9f7a9c515740c4ab629677e11de6aa refers to the cinder-volume.

As per your advice, when i check the services running on the cinder DB, i get the below data, this was done on the controller, and not on the cinder-volume machine. I do not have mysql running on the cinder-volume machine

+---------------------+---------------------+------------+---------+----+------------+------------------+------------------+--------------+----------+-------------------+
| created_at | updated_at | deleted_at | deleted | id | host | binary | topic | report_count | disabled | availability_zone |
+---------------------+---------------------+------------+---------+----+------------+------------------+------------------+--------------+----------+-------------------+
| 2013-04-12 13:11:18 | 2013-04-27 13:24:43 | NULL | 0 | 1 | ubuntu | cinder-volume | cinder-volume | 103569 | 0 | nova |
| 2013-04-15 04:45:04 | 2013-05-09 04:30:18 | NULL | 0 | 2 | ubuntu | cinder-scheduler | cinder-scheduler | 126541 | 0 | nova |
| 2013-05-09 04:31:16 | 2013-06-09 07:21:39 | NULL | 0 | 3 | controller | cinder-scheduler | cinder-scheduler | 267363 | 0 | nova |
+---------------------+---------------------+------------+---------+----+------------+------------------+------------------+--------------+----------+-------------------+

I do not see an entry for compute-2 machine, on which cinder-volume is runnning. The entry for host ubuntu is invalid now, since i do not use the cinder-volume. Machine ubuntu is not renamed controller.
What should i do for the cinder-volume entry to appear here? Do i have to purge the unwanted entry in the above table?

If i run $cinder-manage host list command, i get the below data, both on the controller and the cinder-volume nodes.I do not see an entry for cinder. What is wrong?

host zone
ubuntu nova
controller nova

If i try to give the below command - $cinder create --display-name myvolume1 1
I get ERROR: n/a (HTTP 400) error message, and none of the log files(cinder-api.log, cinder-scheduler.log on controller and cinder-volume.log on cinder-volume) are updated.

If i send $cinder --debug list, i get the below data
connect: (115.119.76.176, 5000)
send: 'POST /v2.0/tokens HTTP/1.1\r\nHost: 115.119.76.176:5000\r\nContent-Length: 105\r\ncontent-type: application/json\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: python-cinderclient\r\n\r\n{"auth": {"tenantName": "admin", "passwordCredentials": {"username": "admin", "password": "admin_pass"}}}'
reply: 'HTTP/1.1 200 OK\r\n'
header: Vary: X-Auth-Token
header: Content-Type: application/json
header: Date: Sun, 09 Jun 2013 11:38:57 GMT
header: Transfer-Encoding: chunked
connect: (10.0.0.15, 8776)
connect fail: (u'10.0.0.15', 8776)
DEBUG (shell:473) n/a (HTTP 400)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cinderclient/shell.py", line 470, in main OpenStackCinderShell().main(sys.argv[1:])
  File "/usr/lib/python2.7/dist-packages/cinderclient/shell.py", line 421, in main args.func(self.cs, args)
  File "/usr/lib/python2.7/dist-packages/cinderclient/v1/shell.py", line 139, in do_list volumes = cs.volumes.list(search_opts=search_opts)
  File "/usr/lib/python2.7/dist-packages/cinderclient/v1/volumes.py", line 171, in list "volumes")
  File "/usr/lib/python2.7/dist-packages/cinderclient/base.py", line 63, in _list resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 138, in get return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 126, in _cs_request **kwargs)
  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 109, in request raise exceptions.from_response(resp, body)
BadRequest: n/a (HTTP 400)
ERROR: n/a (HTTP 400)

Please help with suggestions and advice. I have hit a dead end!
If you need more information, please let me know.

I see another problem on the cinder-volume side, will open another question, so as not to mix things up.

Regards
Mohan

Hi Shausy,
Thank you for answering my query, apologies for the late reply.

I had a look at the link you reffered, i restarted ntp(i am assuming that resync can be done with this) and there is no change in the behaviour.

Just to bring some context to this reply. I briefly explain my set-up and issue.
1. I have a controller running on 10.0.0.11, which has cinder-api, and cinder-scheduler running as well. I also have cinder-volume, but i do not want to use this. I have the other required services running on this. 
2. I have another node running nova-compute and cinder-volume. This is on 10.0.0.15

I have modified the end-point for the cinder-volume according in the keystone DB, the output of $keystone endpoint-list is below

Id 6d9f7a9c515740c4ab629677e11de6aa  refers to the cinder-volume.

+---------------------+---------------------+------------+---------+----+------------+------------------+------------------+--------------+----------+-------------------+
| created_at          | updated_at          | deleted_at | deleted | id | host       | binary           | topic            | report_count | disabled | availability_zone |
+---------------------+---------------------+------------+---------+----+------------+------------------+------------------+--------------+----------+-------------------+
| 2013-04-12 13:11:18 | 2013-04-27 13:24:43 | NULL       |       0 |  1 | ubuntu     | cinder-volume    | cinder-volume    |       103569 |        0 | nova              |
| 2013-04-15 04:45:04 | 2013-05-09 04:30:18 | NULL       |       0 |  2 | ubuntu     | cinder-scheduler | cinder-scheduler |       126541 |        0 | nova              |
| 2013-05-09 04:31:16 | 2013-06-09 07:21:39 | NULL       |       0 |  3 | controller | cinder-scheduler | cinder-scheduler |       267363 |        0 | nova              |
+---------------------+---------------------+------------+---------+----+------------+------------------+------------------+--------------+----------+-------------------+

I do not see an entry for compute-2 machine, on which cinder-volume is runnning. The entry for host ubuntu is invalid now, since i do not use the cinder-volume. Machine ubuntu is not renamed controller. 
What should i do for the cinder-volume entry to appear here? Do i have to purge the unwanted entry in the above table?

If i run $cinder-manage host list command, i get the below data, both on the controller and the cinder-volume nodes.I do not see an entry for cinder. What is wrong?

host                            zone
ubuntu                          nova
controller                      nova

Please help with suggestions and advice. I have hit a dead end! 
If you need more information, please let me know.

I see another problem on the cinder-volume side, will open another question, so as not to mix things up.

Regards
Mohan

Revision history for this message

Launchpad Janitor (janitor) said on 2013-06-25:

This question was expired because it remained in the 'Open' state without activity for the last 15 days.

Revision history for this message

Mohan (v-mohan) said on 2013-06-25:

I could solve this problem by installing the cinder-api and the cinder-scehduler packages on the cinder node. I stopped these services on the controller.

Revision history for this message

Mohan (v-mohan) said on 2013-06-25:

Since the state is expired, i wanted to click the "I still need an answer", provide the answer, and mark this as answered. I hope this will help someone.

Revision history for this message

Mohan (v-mohan) said on 2013-06-25:

I could solve this problem by installing the cinder-api and the cinder-scehduler packages on the cinder node. I stopped these services on the controller.

To post a message you must log in.

Ask a question

Edit question

Cinder

Unable to create volume using cinder-volume on a separate node without public url

Question information

Related bugs

Related FAQ:

Subscribers