OpenStack Compute (nova)

windows metadata broken in grizzly

Asked by David Andrew

Layout: Network node (nova-api, nova-network) --> compute node --> windows guest instance
All openstack components are running Ubuntu 12.04

Description: Windows instance cannot pull metadata from 169.254.169.254 but works fine when going direct to network node api port (http://10.160.3.7:8775). Ubuntu cloud images work fine. The windows image was built with the device flags that kvm in grizzly uses. This same fashion was used for essex and folsom. The problem manifests itself from the guest instance as a connection timeout. Nova network is running in flat dhcp mode. I am running the latest windows virtio driver: virtio-win-0.1-59.iso

Network node IPs:
eth0: 10.160.3.7
br100: 10.160.20.12

Network node DNAT rule:
Chain nova-network-PREROUTING (1 references)
target prot opt source destination
DNAT tcp -- anywhere 169.254.169.254 tcp dpt:http to:10.160.3.7:8775

listening port for api on Network node:
tcp 0 0 0.0.0.0:8775 0.0.0.0:* LISTEN 9604/python

The following is the tcpdump from the network node followed by the compute host. network port on the compute host is the physical interface facing the instances ( eth1.360 ) and the interface on the compute host is the vnet facing the instance:

Network node:

11:35:20.456204 IP (tos 0x0, ttl 128, id 354, offset 0, flags [DF], proto TCP (6), length 52)
    10.160.20.13.49257 > 169.254.169.254.http: Flags [S], cksum 0x0d8b (correct), seq 3362866702, win 8192, options [mss 1372,nop,wscale 2,nop,nop,sackOK], length 0
11:35:20.456263 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52)
    169.254.169.254.http > 10.160.20.13.49257: Flags [S.], cksum 0x72d0 (incorrect -> 0xe2ff), seq 1526576664, ack 3362866703, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 6], length 0
11:35:21.652626 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52)
    169.254.169.254.http > 10.160.20.13.49257: Flags [S.], cksum 0x72d0 (incorrect -> 0xe2ff), seq 1526576664, ack 3362866703, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 6], length 0
11:35:23.652652 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52)
    169.254.169.254.http > 10.160.20.13.49257: Flags [S.], cksum 0x72d0 (incorrect -> 0xe2ff), seq 1526576664, ack 3362866703, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 6], length 0
11:35:27.852638 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52)
    169.254.169.254.http > 10.160.20.13.49257: Flags [S.], cksum 0x72d0 (incorrect -> 0xe2ff), seq 1526576664, ack 3362866703, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 6], length 0
11:35:35.852605 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52)
    169.254.169.254.http > 10.160.20.13.49257: Flags [S.], cksum 0x72d0 (incorrect -> 0xe2ff), seq 1526576664, ack 3362866703, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 6], length 0
11:35:51.852639 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52)
    169.254.169.254.http > 10.160.20.13.49257: Flags [S.], cksum 0x72d0 (incorrect -> 0xe2ff), seq 1526576664, ack 3362866703, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 6], length 0

Compute node:

11:35:19.965553 IP (tos 0x0, ttl 128, id 354, offset 0, flags [DF], proto TCP (6), length 52)
    10.160.20.13.49257 > 169.254.169.254.http: Flags [S], cksum 0x0d8b (correct), seq 3362866702, win 8192, options [mss 1372,nop,wscale 2,nop,nop,sackOK], length 0
11:35:19.965817 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52)
    169.254.169.254.http > 10.160.20.13.49257: Flags [S.], cksum 0xe2ff (correct), seq 1526576664, ack 3362866703, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 6], length 0
11:35:19.966204 IP (tos 0x0, ttl 128, id 355, offset 0, flags [DF], proto TCP (6), length 40)
    10.160.20.13.49257 > 169.254.169.254.http: Flags [.], cksum 0xdc38 (correct), seq 1, ack 1, win 32928, length 0
11:35:19.966391 IP (tos 0x0, ttl 253, id 53540, offset 0, flags [none], proto TCP (6), length 40)
    169.254.169.254.http > 10.160.20.13.49257: Flags [R.], cksum 0xdc34 (correct), seq 1, ack 1, win 32928, length 0
11:35:21.162117 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52)
    169.254.169.254.http > 10.160.20.13.49257: Flags [S.], cksum 0xe2ff (correct), seq 1526576664, ack 3362866703, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 6], length 0
11:35:22.973298 IP (tos 0x0, ttl 128, id 356, offset 0, flags [DF], proto TCP (6), length 52)
    10.160.20.13.49256 > 169.254.169.254.http: Flags [S], cksum 0x822b (correct), seq 4077758162, win 8192, options [mss 1372,nop,wscale 2,nop,nop,sackOK], length 0
11:35:23.162222 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52)
    169.254.169.254.http > 10.160.20.13.49257: Flags [S.], cksum 0xe2ff (correct), seq 1526576664, ack 3362866703, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 6], length 0
11:35:27.362209 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52)
    169.254.169.254.http > 10.160.20.13.49257: Flags [S.], cksum 0xe2ff (correct), seq 1526576664, ack 3362866703, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 6], length 0
11:35:28.979336 IP (tos 0x0, ttl 128, id 365, offset 0, flags [DF], proto TCP (6), length 48)
    10.160.20.13.49256 > 169.254.169.254.http: Flags [S], cksum 0x9634 (correct), seq 4077758162, win 8192, options [mss 1372,nop,nop,sackOK], length 0

Here is a network dump from the same ports as above but bypassing the DNAT rule by sending the windows guest to http://10.160.3.7:8775 and as above the network node is first and the compute node following:

Network node:

11:39:21.181761 IP (tos 0x0, ttl 128, id 401, offset 0, flags [DF], proto TCP (6), length 52)
    10.160.20.13.49258 > optnet01.os.staging.dmz.8775: Flags [S], cksum 0x4f79 (correct), seq 4242207764, win 8192, options [mss 1372,nop,wscale 2,nop,nop,sackOK], length 0
11:39:21.181839 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52)
    optnet01.os.staging.dmz.8775 > 10.160.20.13.49258: Flags [S.], cksum 0x2c7a (incorrect -> 0x93ec), seq 1924411235, ack 4242207765, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 6], length 0
11:39:21.182279 IP (tos 0x0, ttl 128, id 402, offset 0, flags [DF], proto TCP (6), length 40)
    10.160.20.13.49258 > optnet01.os.staging.dmz.8775: Flags [.], cksum 0x8d25 (correct), seq 1, ack 1, win 32928, length 0
11:39:21.182538 IP (tos 0x0, ttl 128, id 403, offset 0, flags [DF], proto TCP (6), length 317)
    10.160.20.13.49258 > optnet01.os.staging.dmz.8775: Flags [P.], cksum 0x002e (correct), seq 1:278, ack 1, win 32928, length 277
11:39:21.182556 IP (tos 0x0, ttl 64, id 25716, offset 0, flags [DF], proto TCP (6), length 40)
    optnet01.os.staging.dmz.8775 > 10.160.20.13.49258: Flags [.], cksum 0x2c6e (incorrect -> 0x0bbc), seq 1, ack 278, win 245, length 0
11:39:21.433617 IP (tos 0x0, ttl 64, id 25717, offset 0, flags [DF], proto TCP (6), length 211)
    optnet01.os.staging.dmz.8775 > 10.160.20.13.49258: Flags [P.], cksum 0x2d19 (incorrect -> 0x3345), seq 1:172, ack 278, win 245, length 171
11:39:21.636607 IP (tos 0x0, ttl 64, id 25718, offset 0, flags [DF], proto TCP (6), length 211)
    optnet01.os.staging.dmz.8775 > 10.160.20.13.49258: Flags [P.], cksum 0x2d19 (incorrect -> 0x3345), seq 1:172, ack 278, win 245, length 171
11:39:21.637104 IP (tos 0x0, ttl 128, id 404, offset 0, flags [DF], proto TCP (6), length 52)
    10.160.20.13.49258 > optnet01.os.staging.dmz.8775: Flags [.], cksum 0x109d (correct), seq 278, ack 172, win 32885, options [nop,nop,sack 1 {1:172}], length 0
11:40:21.533293 IP (tos 0x0, ttl 128, id 413, offset 0, flags [DF], proto TCP (6), length 40)
    10.160.20.13.49258 > optnet01.os.staging.dmz.8775: Flags [R.], cksum 0x0c02 (correct), seq 278, ack 172, win 0, length 0

Compute node:

11:39:20.690979 IP (tos 0x0, ttl 128, id 401, offset 0, flags [DF], proto TCP (6), length 52)
    10.160.20.13.49258 > optnet01.os.staging.dmz.8775: Flags [S], cksum 0x4f79 (correct), seq 4242207764, win 8192, options [mss 1372,nop,wscale 2,nop,nop,sackOK], length 0
11:39:20.691310 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52)
    optnet01.os.staging.dmz.8775 > 10.160.20.13.49258: Flags [S.], cksum 0x93ec (correct), seq 1924411235, ack 4242207765, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 6], length 0
11:39:20.691606 IP (tos 0x0, ttl 128, id 402, offset 0, flags [DF], proto TCP (6), length 40)
    10.160.20.13.49258 > optnet01.os.staging.dmz.8775: Flags [.], cksum 0x8d25 (correct), seq 1, ack 1, win 32928, length 0
11:39:20.691846 IP (tos 0x0, ttl 128, id 403, offset 0, flags [DF], proto TCP (6), length 317)
    10.160.20.13.49258 > optnet01.os.staging.dmz.8775: Flags [P.], cksum 0x002e (correct), seq 1:278, ack 1, win 32928, length 277
11:39:20.691995 IP (tos 0x0, ttl 64, id 25716, offset 0, flags [DF], proto TCP (6), length 40)
    optnet01.os.staging.dmz.8775 > 10.160.20.13.49258: Flags [.], cksum 0x0bbc (correct), seq 1, ack 278, win 245, length 0
11:39:20.943159 IP (tos 0x0, ttl 64, id 25717, offset 0, flags [DF], proto TCP (6), length 211)
    optnet01.os.staging.dmz.8775 > 10.160.20.13.49258: Flags [P.], cksum 0x3345 (correct), seq 1:172, ack 278, win 245, length 171
11:39:21.146110 IP (tos 0x0, ttl 64, id 25718, offset 0, flags [DF], proto TCP (6), length 211)
    optnet01.os.staging.dmz.8775 > 10.160.20.13.49258: Flags [P.], cksum 0x3345 (correct), seq 1:172, ack 278, win 245, length 171
11:39:21.146388 IP (tos 0x0, ttl 128, id 404, offset 0, flags [DF], proto TCP (6), length 52)
    10.160.20.13.49258 > optnet01.os.staging.dmz.8775: Flags [.], cksum 0x109d (correct), seq 278, ack 172, win 32885, options [nop,nop,sack 1 {1:172}], length 0
11:40:21.042512 IP (tos 0x0, ttl 128, id 413, offset 0, flags [DF], proto TCP (6), length 40)
    10.160.20.13.49258 > optnet01.os.staging.dmz.8775: Flags [R.], cksum 0x0c02 (correct), seq 278, ack 172, win 0, length 0

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Compute (nova) Edit question
Assignee:
No assignee Edit question
Solved by:
David Andrew
Solved:
Last query:
Last reply:
Revision history for this message
David Andrew (ev0ldave) said : 		#1

Well, problem solved by injecting static route for 169.254.169.254 to the gateway via dhcp option 121.

Revision history for this message
David Andrew (ev0ldave) said : 		#2

On a side note. This is unnecessary if you are running multihost = True with nova-api also on the compute host.