[MultiZones] NoValidHost: No hosts were available

How long after starting up the compute service did you attempt to create the instance? Hosts report their status on a regular basis, defined by FLAGS.periodic_interval, which I believe defaults to 60 seconds. What that means is that for the first 60 seconds a host is running, the zone may not know about it.

Try again, but wait at least a minute before attempting to create the instance. It should work fine then; if not, post the error you receive.

Hi Ed,

I waited this time out. I check again and I steel have the problem.
I found something strange. I add some log in the source file '/nova/scheduler/host_filter.py ' to check which default host filter is used by the scheduler and it use the 'InstanceTypeFilter', instead I set flag 'default_host_filter' to 'nova.scheduler.host_filter.AllHostsFilter'. And we can see that in the first line of child or the parent logs 'u'num_instances': 1, u'filter': u'nova.scheduler.host_filter.InstanceTypeFilter', u'blob': None'

I've got other questions about Multizones:

1) How project is distributed between zones ? Is it possible to have some instances of project 1 in zone 1 and other instances of project 1 in zone 2 ?

2) How the VLAN network mode must be set between zones ? One nova-network daemon on each zone ? Same VLAN for each zone ?

> I waited this time out. I check again and I steel have the problem. I
> found something strange. I add some log in the source file
> '/nova/scheduler/host_filter.py ' to check which default host filter is
> used by the scheduler and it use the 'InstanceTypeFilter', instead I set
> flag 'default_host_filter' to
> 'nova.scheduler.host_filter.AllHostsFilter'. And we can see that in the
> first line of child or the parent logs 'u'num_instances': 1, u'filter':
> u'nova.scheduler.host_filter.InstanceTypeFilter', u'blob': None'

The InstanceTypeFilter is hard-coded into nova/compute/api.py. Setting the flag has no effect.

If you are still getting the 'no hosts' error, then your compute host does not have the capabilities required by the instance being created. To debug this, go into the 'filter_hosts()' method of nova/scheduler/host_filter.py, and inside the loop beginning with "for host, services in zone_manager.service_states.iteritems():", add logging to output the values of all the values that are created in that loop to determine if the host is capable of creating the instance.

> I've got other questions about Multizones:
>
> 1) How project is distributed between zones ? Is it possible to have
> some instances of project 1 in zone 1 and other instances of project 1
> in zone 2 ?

Right now it is possible, but you have to make sure that the project credentials are duplicated across zones. When Keystone is integrated, that will no longer be necessary.

> 2) How the VLAN network mode must be set between zones ? One
> nova-network daemon on each zone ? Same VLAN for each zone ?

Yes, there must be at least one network service running in each zone. I'm no expert on running VLAN mode, but I would suspect that since each zone can be configured independently, you don't need the VLAN settings to be the same in different zones, but you should ask someone who knows more about networking.

Why it's hard-coded ? It still under development ?

In fact, zone_manager.service_states.iteritems() return an empty dict. So the loop doesn't start.

For my question 2) I think, if a project can be distributed on different zones, the VLAN of the project must be the same in different zone. In this case, the layer 2 is distributed between zone.

> Why it's hard-coded ? It still under development ?

I don't know why - maybe Sandy Walsh can explain. But nova/compute/api.py in the _ask_scheduler_to_create_instance() method hard-codes that value.

Do you know why the 'zone_manager.service_states.iteritems()' method returns an empty dict ?

I've got two compute hosts in my zone with enough resources :

$ nova-manage service list
p-novamaster2 nova-vncproxy enabled :-) 2011-07-08 07:24:36
p-novamaster2 nova-volume enabled :-) 2011-07-08 07:24:36
p-novamaster2 nova-scheduler enabled :-) 2011-07-08 07:24:36
p-hs21-16 nova-compute enabled :-) 2011-07-08 07:24:36
p-hs21-17 nova-network enabled :-) 2011-07-08 07:24:36
p-hs21-17 nova-compute enabled :-) 2011-07-08 07:24:36

$ nova-manage service describe_resource p-hs21-16
HOST PROJECT cpu mem(mb) disk(gb)
p-hs21-16(total) 8 32240 36
p-hs21-16(used) 0 804 4

$ nova-manage service describe_resource p-hs21-17
HOST PROJECT cpu mem(mb) disk(gb)
p-hs21-17(total) 8 32240 36
p-hs21-17(used) 0 910 5

In fact, the method 'update_service_capabilities' in zone-manager.py is never call.
So the service_states is never updated and stills empty

> In fact, the method 'update_service_capabilities' in zone-manager.py is never call.
> So the service_states is never updated and stills empty

Do you have the 'rabbit_host' flag configured on your various nodes? All nodes in a zone need to have that set to the address of the machine running the rabbitmq-server service. If not, then the messages are not being sent to the proper queue, and would therefore never reach the zone manager.

Yes, I set this flag.
If i configure the zone in a classic mode (mono zone), it works nicely.

I made a test. I had code in a nova-manage command to force a call to 'update_service_capabilities' method:
        rpc.call(ctxt,
                 db.queue_get_for(ctxt, FLAGS.scheduler_topic, 'p-novamaster2'),
                 {"method": "update_service_capabilities"})

With, that, 'service_state' is filling but I get another error:

2011-07-08 14:53:05,303 ERROR nova [-] Exception during message handling
(nova): TRACE: Traceback (most recent call last):
(nova): TRACE: File "/usr/lib/pymodules/python2.7/nova/rpc.py", line 232, in _process_data
(nova): TRACE: rval = node_func(context=ctxt, **node_args)
(nova): TRACE: File "/usr/lib/pymodules/python2.7/nova/scheduler/manager.py", line 90, in _schedule
(nova): TRACE: **kwargs)
(nova): TRACE: File "/usr/lib/pymodules/python2.7/nova/scheduler/zone_aware_scheduler.py", line 225, in schedule_run_instance
(nova): TRACE: build_plan = self.select(context, request_spec)
(nova): TRACE: File "/usr/lib/pymodules/python2.7/nova/scheduler/zone_aware_scheduler.py", line 248, in select
(nova): TRACE: *args, **kwargs)
(nova): TRACE: File "/usr/lib/pymodules/python2.7/nova/scheduler/zone_aware_scheduler.py", line 283, in _schedule
(nova): TRACE: host_list = self.filter_hosts(topic, request_spec, host_list)
(nova): TRACE: File "/usr/lib/pymodules/python2.7/nova/scheduler/host_filter.py", line 348, in filter_hosts
(nova): TRACE: return host_filter.filter_hosts(self.zone_manager, query)
(nova): TRACE: File "/usr/lib/pymodules/python2.7/nova/scheduler/host_filter.py", line 127, in filter_hosts
(nova): TRACE: host_ram_mb = capabilities['host_memory_free']
(nova): TRACE: KeyError: 'host_memory_free'
(nova): TRACE:

It's hard coded because the only API methods for creating instances are based on InstanceTypes (aka Flavor). There are other groups that want more elaborate means for expressing the type of instance created, they would be the ones to "unstick" this setting.

As for the update_service_capabilities, we first need to ensure the periodic task is being called and that it has something to report. Network and Volume services currently don't have any capabilities to report so we'll never hear from them. It's only Compute so far, and even then it's only with XenServer. If you're using another hypervisor you need to add your own hook to extract the HV stats and report call update_service_capabilities.

Which virt layer are you using?

Hi Sandy,

I use libvirt/KVM hypervisor. So Multi-Cluster is only support with the XenServer hypervisor ?
Libvirt driver reports some host stats, like this :

$ nova-manage service list
p-novamaster2 nova-vncproxy enabled :-) 2011-07-08 07:24:36
p-novamaster2 nova-volume enabled :-) 2011-07-08 07:24:36
p-novamaster2 nova-scheduler enabled :-) 2011-07-08 07:24:36
p-hs21-16 nova-compute enabled :-) 2011-07-08 07:24:36
p-hs21-17 nova-network enabled :-) 2011-07-08 07:24:36
p-hs21-17 nova-compute enabled :-) 2011-07-08 07:24:36

$ nova-manage service describe_resource p-hs21-16
HOST PROJECT cpu mem(mb) disk(gb)
p-hs21-16(total) 8 32240 36
p-hs21-16(used) 0 804 4

It's not enough for the host_filter method ?

No, multi-cluster works with all HV's.

Look at nova.compute.manager.py periodic_tasks() ... you'll see that the _report_driver_status() method is called every 30 seconds or so. This call delegates to the virt layer to get the capabilities via the get_host_stats() call.

Currently, only the XenServer driver has this method implemented, KVM doesn't. But it should be pretty easy to add (and I'm sure it would be a very welcome addition). We didn't add it since we're a XenServer shop.

Hope it helps!

Ok, thanks for your help.
I don't know If I got enough experience in development to do that, but I will try.

Hi Sandy,

I tried to implement the code for get_host_stats() on the virt layer 'libvirt'.
I check what was done for XenServer and I don't understand something.

The method 'filter_host' in 'InstanceTypeFilter' class compares the capabilities of the host with specified resources of the instance. But the capabilities of the host are in bytes and the specified resources of the instance are in Mbyte (for memory) or Gbyte (for disk). Is it normal ?

Hey!

Yeah, we haven't really nailed down what the units of measure are for all that stuff yet ... or what the expected fields/names are. I think assume mb for ram and gb for disk for now. I'll file a bug to get it all on the same page.

Thanks for the reminder on that!

This question was expired because it remained in the 'Open' state without activity for the last 15 days.