Enable 1Gb interface on KVM with nova-compute

Asked by Mercadolibre CloudBuilders

Hi everyone, we are running Varnish Cache over kvm in our nova cluster.
The thing is, the performance is aufull, so ... we've realized that all instances booted with nova-compute on any node, are booted with 100Mb RTL interfaces, en we are seeing this line in libvirt.xml for any instance :

        <interface type='bridge'>
            <source bridge='br100'/>
            <mac address='02:16:3e:5a:ba:6f'/>
            <!-- <model type='virtio'/> CANT RUN virtio network right now --> <-- THIS LINE
            <filterref filter="nova-instance-instance-000000bf-02163e5aba6f">
                <parameter name="IP" value="172.16.144.24" />
                <parameter name="DHCPSERVER" value="172.16.144.1" />
            </filterref>
        </interface>

So, how to we enable virtio model or e1000 model for a performance boost on nova using kvm ??
We are really needing to enable this.

Thanks !!

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Compute (nova) Edit question
Assignee:
No assignee Edit question
Solved by:
Vish Ishaya
Solved:
Last query:
Last reply:

This question was reopened

Revision history for this message
Vish Ishaya (vishvananda) said :
#1

Remove the comments around the line. There are issues with nova-volume and
corrupt net and some older kernels. Specifically lucid and centos 5.5. If
you are not using nova-volume or those guests you should be able to
uncomment that line with no issues.
On Aug 7, 2011 4:51 PM, "Mercadolibre CloudBuilders" <
<email address hidden>> wrote:
> New question #167276 on OpenStack Compute (nova):
> https://answers.launchpad.net/nova/+question/167276
>
> Hi everyone, we are running Varnish Cache over kvm in our nova cluster.
> The thing is, the performance is aufull, so ... we've realized that all
instances booted with nova-compute on any node, are booted with 100Mb RTL
interfaces, en we are seeing this line in libvirt.xml for any instance :
>
> <interface type='bridge'>
> <source bridge='br100'/>
> <mac address='02:16:3e:5a:ba:6f'/>
> <!-- <model type='virtio'/> CANT RUN virtio network right now --> <-- THIS
LINE
> <filterref filter="nova-instance-instance-000000bf-02163e5aba6f">
> <parameter name="IP" value="172.16.144.24" />
> <parameter name="DHCPSERVER" value="172.16.144.1" />
> </filterref>
> </interface>
>
> So, how to we enable virtio model or e1000 model for a performance boost
on nova using kvm ??
> We are really needing to enable this.
>
> Thanks !!
>
> --
> You received this question notification because you are a member of Nova
> Core, which is an answer contact for OpenStack Compute (nova).

Revision history for this message
Mercadolibre CloudBuilders (cloudbuilders-n) said :
#2

Hi vish!!
We ve tried that, and ran euca-reboot-instances over that inatance with no result at all, we keep seeing the "-device rtl19........" and no virtio modification over the -net line.

The thing is, is enough to run euca-reboot-instances for those changes to take effect?
And the other thing is, yes, we are running lucid as a guest with kernel 2.6.32-32, should we DEFINITELLY go for maverick?

Thnk again vish

Revision history for this message
Brian Lamar (blamar) said :
#3

You're going to need to restart any and all nova-compute services after making the template change as currently templates are only read during service start-up. Does this still not work after changing the template, restarting nova-compute, and then calling run-instances?

Revision history for this message
Vish Ishaya (vishvananda) said :
#4

To change new instances, change the libvirt.xml template

To change running instances, change the libvirt.xml file in the instance's directory (/var/lib/nova/instances/instance-xxxx/), then:
  virsh destroy instance-xxx
  virsh undefine instance-xxx
  virsh define libvirt.xml
  virsh start instance-xxx

Lucid guests will be fine as long as you don't try to dynamically attach a volume to them. If you need attachable volumes you have to use maverick+

Vish

On Aug 7, 2011, at 11:01 PM, Mercadolibre CloudBuilders wrote:

> Question #167276 on OpenStack Compute (nova) changed:
> https://answers.launchpad.net/nova/+question/167276
>
> Status: Answered => Open
>
> Mercadolibre CloudBuilders is still having a problem:
> Hi vish!!
> We ve tried that, and ran euca-reboot-instances over that inatance with no result at all, we keep seeing the "-device rtl19........" and no virtio modification over the -net line.
>
> The thing is, is enough to run euca-reboot-instances for those changes to take effect?
> And the other thing is, yes, we are running lucid as a guest with kernel 2.6.32-32, should we DEFINITELLY go for maverick?
>
> Thnk again vish
>
> --
> You received this question notification because you are a member of Nova
> Core, which is an answer contact for OpenStack Compute (nova).

Revision history for this message
Mercadolibre CloudBuilders (cloudbuilders-n) said :
#5

Perfect Brian/Vish !!

I'll try that !! the thing is ... if i KNOW i will not be using nova-volume, can i touch the template from the nova-compute side, to make sure it allways create a new instance using virtio nic ?

Im not talking about the libvirt.xml inside each instance, but the template nova-compute uses for every new created instance.

Thanks for the quick response Vish !

Revision history for this message
Vish Ishaya (vishvananda) said :
#6

Absolutely, that is why it is a template file. It should be in virt/libvirt.xml.template

Vish

On Aug 8, 2011, at 9:51 AM, Mercadolibre CloudBuilders wrote:

> Question #167276 on OpenStack Compute (nova) changed:
> https://answers.launchpad.net/nova/+question/167276
>
> Status: Answered => Open
>
> Mercadolibre CloudBuilders is still having a problem:
> Perfect Brian/Vish !!
>
> I'll try that !! the thing is ... if i KNOW i will not be using nova-
> volume, can i touch the template from the nova-compute side, to make
> sure it allways create a new instance using virtio nic ?
>
> Im not talking about the libvirt.xml inside each instance, but the
> template nova-compute uses for every new created instance.
>
> Thanks for the quick response Vish !
>
> --
> You received this question notification because you are a member of Nova
> Core, which is an answer contact for OpenStack Compute (nova).

Revision history for this message
Mercadolibre CloudBuilders (cloudbuilders-n) said :
#7

Thanks Vish Ishaya, that solved my question.

Revision history for this message
Mercadolibre CloudBuilders (cloudbuilders-n) said :
#8

Vish ... one last thing.

What about if i want to try e1000 ethernet model and not virtio, by touching on the template <model type='virtio'/> for <model type='e1000'/> will do the work ?

Revision history for this message
Best Vish Ishaya (vishvananda) said :
#9

Yes that will work. I don't think you will see a significant difference in performance though.

Vish

On Aug 8, 2011, at 2:21 PM, Mercadolibre CloudBuilders wrote:

> Question #167276 on OpenStack Compute (nova) changed:
> https://answers.launchpad.net/nova/+question/167276
>
> Status: Solved => Open
>
> Mercadolibre CloudBuilders is still having a problem:
> Vish ... one last thing.
>
> What about if i want to try e1000 ethernet model and not virtio, by
> touching on the template <model type='virtio'/> for <model
> type='e1000'/> will do the work ?
>
> --
> You received this question notification because you are a member of Nova
> Core, which is an answer contact for OpenStack Compute (nova).

Revision history for this message
Mercadolibre CloudBuilders (cloudbuilders-n) said :
#10

Thanks Vish Ishaya, that solved my question.

Revision history for this message
Mercadolibre CloudBuilders (cloudbuilders-n) said :
#11

Just as a side note.

"virtio" drivers worked like a charm compared to "e1000".
We've reached the performance we wanted.

Maybe if there is any use, we can paste here the generic kernel sysctl tunning that worked very good for us.

Best regards

Revision history for this message
Vish Ishaya (vishvananda) said :
#12

That would be fantastic. What kind of performance are you getting? Did you tune both host and guest? Are you using 1G or 10G ethernet on the hosts?

Vish

On Aug 9, 2011, at 2:16 PM, Mercadolibre CloudBuilders wrote:

> Question #167276 on OpenStack Compute (nova) changed:
> https://answers.launchpad.net/nova/+question/167276
>
> Mercadolibre CloudBuilders posted a new comment:
> Just as a side note.
>
> "virtio" drivers worked like a charm compared to "e1000".
> We've reached the performance we wanted.
>
> Maybe if there is any use, we can paste here the generic kernel sysctl
> tunning that worked very good for us.
>
> Best regards
>
> --
> You received this question notification because you are a member of Nova
> Core, which is an answer contact for OpenStack Compute (nova).

Revision history for this message
Vish Ishaya (vishvananda) said :
#13

Promoted this to a FAQ
FAQ #1734: “Why doesn't nova use virtio networking by default?”.

Revision history for this message
Mercadolibre CloudBuilders (cloudbuilders-n) said :
#14

Hi Vish.
First of all, thanks for promoting this to FAQs, we are glad that this helped.

On the host side, we are using two 1Gb with bonding mode 4.
As we are using kvm instances in fully virtualized mode, we had to tune both the hosts a the newly created instances, the good news is that we aplied the exact kernel parameters to both, the host and guests.

One of the uses that we are giving to openstack instances, is to serve Varnish Cache and nginX as cache and load balancing services.

With "virtio" network drivers, and this kernel parameters on sysctl (lucid from the host side, and maverick from the guest side), we were able to reach ( with httperf ) 2000 concurrent requests per second with no problem, so we are very happy.

This are the kernel parameters, and some URLs were we get info from, hope it helps !!

############################################################
# http://bashshell.net/script-of-the-week/script-of-the-week-tuning-tcp-packets/
net.ipv4.tcp_max_syn_backlog = 30000
net.ipv4.tcp_max_tw_buckets = 2000000
net.core.netdev_max_backlog = 50000

# http://www.techrepublic.com/blog/opensource/tuning-the-linux-kernel-for-more-aggressive-network-throughput/62 - http://fasterdata.es.net/fasterdata/host-tuning/linux/
net.ipv4.tcp_window_scaling = 1
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_congestion_control = cubic
vm.swappiness = 0
net.ipv4.tcp_syncookies = 1
############################################################

Regards !

Revision history for this message
rick jones (perfgeek) said :
#15

I've never been terribly fond of those "make everything 16 MB" tuning guides, though perhaps I'm missing something about the settings that call for that - the bandwidth delay product which requires a 16 MB TCP window is quite large.

If the application(s) running on the system don't make explicit setsockopt() calls to set SO_[SND|RCV]BUF calls, the net.core.[rw]mem_max settings won't make any difference. Similarly, if the application(s) make explicit setsockopt() calls, the tcp_[rw]mem settings won't take effect.

When/if running something like netperf TCP_STREAM to measure throughput, the "classic" output will only show what the socket buffers were at the time the socket was opened. If you use the output selector functionality of 2.5.0 you can see what the socket sizes became by the end of the test. In the context of a TCP_STREAM test the sizes of interest will be the local socket send size and remote socket receive:

netperf -t TCP_STREAM -H <host> -- -o throughput,lsr_size_end,rsr_size_end