multi node instance failure
Hi,
I've 6 machines I want to run OpenStack on. I'm planning on one controller, and the rest nodes.
I've configured the controller to run instances locally, using ubuntu1010-
I then add a compute node. I'm manually installing parts to get a feel for what's happening. Here's what I do:-
$ sudo apt-get install python-
$ sudo add-apt-repository ppa:nova-core/trunk
$ sudo apt-get update
$ sudo apt-get install -y nova-common nova-compute python-mysqldb python-suds vlan
I'll then scp nova.conf from the controller to the node, and restart all services on all machines, after nova-manage db sync on the controller. I don't edit network/interfaces for br100 as I'm running VlanManager, this could be a mistake. The node will then appear in the services table.
When I attempt to run instances, and when they're scheduled for on the additional nodes the instances always fail. Here are 2 log files, from 2 nodes I've tried to configure:-
CLOUD NODE 1
2011-03-30 17:07:34,326 WARNING nova.virt.
le running command.
Command: sudo tune2fs -c 0 -i 0 /dev/nbd15
Exit code: 1
Stdout: 'tune2fs 1.41.12 (17-May-2010)\n'
Stderr: "tune2fs: Invalid argument while trying to open /dev/nbd15\
2011-03-30 17:08:05,516 ERROR nova.exception [-] Uncaught exception
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File "/usr/lib/
(nova.exception): TRACE: return f(*args, **kw)
(nova.exception): TRACE: File "/usr/lib/
(nova.exception): TRACE: self._conn.
(nova.exception): TRACE: File "/usr/lib/
(nova.exception): TRACE: if ret is None:raise libvirtError(
(nova.exception): TRACE: libvirtError: internal error process exited while connecting to monitor: char device redirected to /dev/pts/1
(nova.exception): TRACE: qemu: could not load kernel '/var/lib/
(nova.exception): TRACE:
(nova.exception): TRACE:
2011-03-30 17:08:05,518 ERROR nova.compute.
IOS?
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
CLOUD NODE 2
2011-03-30 17:29:12,937 WARNING nova.virt.
le running command.
Command: sudo tune2fs -c 0 -i 0 /dev/nbd15
Exit code: 1
Stdout: 'tune2fs 1.41.12 (17-May-2010)\n'
Stderr: "tune2fs: Invalid argument while trying to open /dev/nbd15\
2011-03-30 17:29:15,123 ERROR nova.exception [-] Uncaught exception
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File "/usr/lib/
(nova.exception): TRACE: return f(*args, **kw)
(nova.exception): TRACE: File "/usr/lib/
(nova.exception): TRACE: self._conn.
(nova.exception): TRACE: File "/usr/lib/
(nova.exception): TRACE: if ret is None:raise libvirtError(
(nova.exception): TRACE: libvirtError: operation failed: failed to retrieve chardev info in qemu with 'info chardev'
(nova.exception): TRACE:
2011-03-30 17:29:15,125 ERROR nova.compute.
IOS?
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
2011-03-30 17:29:19,186 INFO nova.compute.
2011-03-30 17:29:19,187 INFO nova.compute.
There are error messages about virtualization not being enabled in the BIOS, but this is not the case, it is enabled.
Any ideas?
Question information
- Language:
- English Edit question
- Status:
- Solved
- Assignee:
- No assignee Edit question
- Solved by:
- nimda
- Solved:
- 2011-04-06
- Last query:
- 2011-04-06
- Last reply:
- 2011-03-31
Vish Ishaya (vishvananda) said : | #1 |
There are probably a couple of issues here. One is that you will need to use glance so that that the other node can grab images. You also may have issues with kvm module and permissions:
sudo modprobe kvm
sudo chgroup kvm /dev/kvm
sudo chmod g+w /dev/kvm
Vish
On Mar 30, 2011, at 4:06 PM, nimda wrote:
> New question #151125 on OpenStack Compute (nova):
> https:/
>
> Hi,
>
> I've 6 machines I want to run OpenStack on. I'm planning on one controller, and the rest nodes.
>
> I've configured the controller to run instances locally, using ubuntu1010-
>
> I then add a compute node. I'm manually installing parts to get a feel for what's happening. Here's what I do:-
>
> $ sudo apt-get install python-
> $ sudo add-apt-repository ppa:nova-core/trunk
> $ sudo apt-get update
> $ sudo apt-get install -y nova-common nova-compute python-mysqldb python-suds vlan
>
> I'll then scp nova.conf from the controller to the node, and restart all services on all machines, after nova-manage db sync on the controller. I don't edit network/interfaces for br100 as I'm running VlanManager, this could be a mistake. The node will then appear in the services table.
>
> When I attempt to run instances, and when they're scheduled for on the additional nodes the instances always fail. Here are 2 log files, from 2 nodes I've tried to configure:-
>
> CLOUD NODE 1
>
> 2011-03-30 17:07:34,326 WARNING nova.virt.
> le running command.
> Command: sudo tune2fs -c 0 -i 0 /dev/nbd15
> Exit code: 1
> Stdout: 'tune2fs 1.41.12 (17-May-2010)\n'
> Stderr: "tune2fs: Invalid argument while trying to open /dev/nbd15\
> 2011-03-30 17:08:05,516 ERROR nova.exception [-] Uncaught exception
> (nova.exception): TRACE: Traceback (most recent call last):
> (nova.exception): TRACE: File "/usr/lib/
> (nova.exception): TRACE: return f(*args, **kw)
> (nova.exception): TRACE: File "/usr/lib/
> (nova.exception): TRACE: self._conn.
> (nova.exception): TRACE: File "/usr/lib/
> (nova.exception): TRACE: if ret is None:raise libvirtError(
> (nova.exception): TRACE: libvirtError: internal error process exited while connecting to monitor: char device redirected to /dev/pts/1
> (nova.exception): TRACE: qemu: could not load kernel '/var/lib/
> (nova.exception): TRACE:
> (nova.exception): TRACE:
> 2011-03-30 17:08:05,518 ERROR nova.compute.
> IOS?
> (nova.compute.
> (nova.compute.
> (nova.compute.
> (nova.compute.
> (nova.compute.
> (nova.compute.
> (nova.compute.
> (nova.compute.
> (nova.compute.
>
> CLOUD NODE 2
>
> 2011-03-30 17:29:12,937 WARNING nova.virt.
> le running command.
> Command: sudo tune2fs -c 0 -i 0 /dev/nbd15
> Exit code: 1
> Stdout: 'tune2fs 1.41.12 (17-May-2010)\n'
> Stderr: "tune2fs: Invalid argument while trying to open /dev/nbd15\
> 2011-03-30 17:29:15,123 ERROR nova.exception [-] Uncaught exception
> (nova.exception): TRACE: Traceback (most recent call last):
> (nova.exception): TRACE: File "/usr/lib/
> (nova.exception): TRACE: return f(*args, **kw)
> (nova.exception): TRACE: File "/usr/lib/
> (nova.exception): TRACE: self._conn.
> (nova.exception): TRACE: File "/usr/lib/
> (nova.exception): TRACE: if ret is None:raise libvirtError(
> (nova.exception): TRACE: libvirtError: operation failed: failed to retrieve chardev info in qemu with 'info chardev'
> (nova.exception): TRACE:
> 2011-03-30 17:29:15,125 ERROR nova.compute.
> IOS?
> (nova.compute.
> (nova.compute.
> (nova.compute.
> (nova.compute.
> (nova.compute.
> (nova.compute.
> (nova.compute.
> 2011-03-30 17:29:19,186 INFO nova.compute.
> 2011-03-30 17:29:19,187 INFO nova.compute.
>
> There are error messages about virtualization not being enabled in the BIOS, but this is not the case, it is enabled.
>
> Any ideas?
>
>
> --
> You received this question notification because you are a member of Nova
> Core, which is an answer contact for OpenStack Compute (nova).
nimda (jon-whiteheat) said : | #2 |
I've tested a qemu image on both nodes, and was able to execute a VM. I'm still receiving exactly the same errors in nova-compute log when trying to launch OpenStack instances though.
I'll look into setting up Glance, to see what effect that has on my experience. Are there any guides to set up Glance?
nimda (jon-whiteheat) said : | #3 |
I tried to install an additional 2 node, just to see what sort of results I got. I'm starting to see a pattern now:-
NODE 3
2011-03-31 17:56:35,236 WARNING nova.virt.
le running command.
Command: sudo tune2fs -c 0 -i 0 /dev/nbd15
Exit code: 1
Stdout: 'tune2fs 1.41.12 (17-May-2010)\n'
Stderr: "tune2fs: Invalid argument while trying to open /dev/nbd15\
2011-03-31 17:56:37,437 ERROR nova.exception [-] Uncaught exception
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File "/usr/lib/
(nova.exception): TRACE: return f(*args, **kw)
(nova.exception): TRACE: File "/usr/lib/
(nova.exception): TRACE: self._conn.
(nova.exception): TRACE: File "/usr/lib/
(nova.exception): TRACE: if ret is None:raise libvirtError(
(nova.exception): TRACE: libvirtError: operation failed: failed to retrieve chardev info in qemu with 'info chardev'
(nova.exception): TRACE:
2011-03-31 17:56:37,439 ERROR nova.compute.
IOS?
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
NODE 4
2011-03-31 17:56:38,218 WARNING nova.virt.
le running command.
Command: sudo tune2fs -c 0 -i 0 /dev/nbd15
Exit code: 1
Stdout: 'tune2fs 1.41.12 (17-May-2010)\n'
Stderr: "tune2fs: Invalid argument while trying to open /dev/nbd15\
2011-03-31 17:56:40,447 ERROR nova.exception [-] Uncaught exception
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File "/usr/lib/
(nova.exception): TRACE: return f(*args, **kw)
(nova.exception): TRACE: File "/usr/lib/
(nova.exception): TRACE: self._conn.
(nova.exception): TRACE: File "/usr/lib/
(nova.exception): TRACE: if ret is None:raise libvirtError(
(nova.exception): TRACE: libvirtError: operation failed: failed to retrieve chardev info in qemu with 'info chardev'
(nova.exception): TRACE:
2011-03-31 17:56:40,449 ERROR nova.compute.
IOS?
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
I'd still like to know if there are recommended setup guides for Glance.
Jay Pipes (jaypipes) said : | #4 |
On Thu, Mar 31, 2011 at 12:46 PM, nimda
<email address hidden> wrote:
> I'll look into setting up Glance, to see what effect that has on my
> experience. Are there any guides to set up Glance?
Sure thing.
http://
http://
Working on the configuration documentation...
Cheers,
jay
nimda (jon-whiteheat) said : | #5 |
I'd like to know if creating instances across multiple nodes requires setting up, past what's told in this guide:-
http://
I can create a single node, that's able to create instances, as mentioned above, but not onto additional nodes. Is the guide above missing anything I need to know to accomplish this?
I had started looking into Glance, as that's what similar questions have been resolved with. But, since reading about Glance, I'm now understanding I should be able to use nova's own object-store.
At the very least, is there a simple guide to create a multi-node cloud, that I can learn from?
nimda (jon-whiteheat) said : | #6 |
Okay, I've had some success. I'm able to create instances on all nodes, only I'm not able to ping them. I was able to do this by copying /var/lib/
nimda (jon-whiteheat) said : | #7 |
Additionally, I'm not able to ping any instance from the 2nd node.
nimda (jon-whiteheat) said : | #8 |
This evening I managed to get instances on on controller/node and a standalone node, and was able to communicate with them.
I've tweaked many different parts so I'm not entirely sure what combination finally worked. I have tried to install another node, and that failed, I think this is because of missing /var/lib/
So here's nova.conf
--dhcpbridge_
--dhcpbridge=
--logdir=
--state_
--lock_
--verbose
--sql_connectio
--s3_host=
--rabbit_
--cc_host=
--ec2_url=http://
--network_
--flat_
--fixed_
--network_size=32
--quota_
...and /etc/network/
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth0
iface eth0 inet dhcp
Other than that, I just followed this guide http://
sunhee Park (sunny790821) said : | #9 |
I got similar message. T.T
I use XCP(xenServer community version) as hypervisor.
Some has the answer?
2011-07-21 16:01:21,767 ERROR nova.compute.
S?
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.
(nova.compute.