LBaaS not serving traffic with Floating IP (DVR)

Asked by Jean Duminy on 2018-05-12

Hi,

We having an ongoing issue with LBaaS not serving traffic. An existing LBaaS with a floating IP is fully functional is serving traffic, unaffected. When we build another LBaaS and use a Floating IP (same range), the IP is reachable but the we get a 503 error.
A few times I added 20 or so floating IP to a projects, thus removing them from the floating pool. I create the same LBaaS and attach a Floating IP and this workings. (See attached heat template).

In addition the floating IP of the instances, work each time. So I assume nova is releasing the floating IP as required. But the same is not true for LB (HAProxy LBaaS).

The bootstack managed cloud consists out of 14 servers. 3 Infra, 2 x Neutron, 9 x computes. (HA Cloud)
Its version pike, which was built a few weeks ago.
The networks are the traditional openstack network with floating provider and also provider (DVR). The networks are shared to specific projects by mean of RBAC.
All the projects are with in an new openstack domain.

The stack creates a private network adds and routers; builds a jumpbox and a web servers; created a Load balancer on same network connect to the web servers. I am manually add the floating IP to the LB.
When support restart the HAProxy the networking comes good.

I came across this bug which sort of touches on a few items, but I assume this would have already be fix is pike.
https://bugs.launchpad.net/neutron/+bug/1583694

"Distributed Virtual Routers are created on each Compute node dynamically on demand and removed when not required. Distributed Virtual Routers heavily depend on the port binding to identify the requirement of a DVR service on a particular node."

"This would create an issue because we will be seeing the same FloatingIP being advertised(GARP) from all nodes, and so the users on the external network will get confused on where the actual "ACTIVE" port is"

I dont have access to the servers so cannot get more details logs.

Where do I start to narrow down this problem?

Here is the stack web yaml where I can reproduce the error without exception.
#######

heat_template_version: 2016-10-14

description: >
  HOT template to create a private network, a router to the public
  network, and two servers in the new network. Floating IP addresses
  are assigned to each server so they are routable from the public network.

parameters:
  key_name:
    type: string
    default: phoenix
    description: Name of keypair to assign to servers
  image:
    type: string
    default: Ubuntu 16.04 LTS
    description: Name of image to use for servers
  flavor:
    type: string
    default: m1.medium
    description: Flavor to use for servers
  public_net:
    type: string
    default: floating_nonprod
    description: >
      ID or name of public network for which floating IP addresses will be allocated

resources:
  private_net:
    type: OS::Neutron::Net
    properties:
      name: web_private_net

  private_subnet:
    type: OS::Neutron::Subnet
    properties:
      network_id: { get_resource: private_net }
      cidr: 10.4.2.0/24
      allocation_pools:
        - start: 10.4.2.10
          end: 10.4.2.50

  router:
    type: OS::Neutron::Router
    properties:
      external_gateway_info:
        network: { get_param: public_net }

  private_router_interface:
    type: OS::Neutron::RouterInterface
    properties:
      router_id: { get_resource: router }
      subnet_id: { get_resource: private_subnet }

  allow_ssh:
    type: OS::Neutron::SecurityGroup
    properties:
      name: Allow SSH
      rules:
        - direction: ingress
          protocol: tcp
          port_range_max: 22
          port_range_min: 22
          remote_ip_prefix: 0.0.0.0/0

  allow_icmp:
    type: OS::Neutron::SecurityGroup
    properties:
      name: Allow ICMP
      rules:
        - protocol: icmp
          remote_ip_prefix: 0.0.0.0/0

  allow_web:
    type: OS::Neutron::SecurityGroup
    properties:
      name: Allow web
      rules:
        - direction: ingress
          protocol: tcp
          port_range_max: 80
          port_range_min: 80
          remote_ip_prefix: 0.0.0.0/0

  jumpbox:
    type: OS::Nova::Server
    properties:
      name: jumpbox
      image: { get_param: image }
      flavor: { get_param: flavor }
      key_name: { get_param: key_name }
      networks:
        - port: { get_resource: jump_box_port }

  jump_box_port:
    type: OS::Neutron::Port
    properties:
      network_id: { get_resource: private_net }
      security_groups:
        - { get_resource: allow_ssh }
        - { get_resource: allow_icmp }
      fixed_ips:
        - subnet_id: { get_resource: private_subnet }

  jump_box_floating_ip:
    type: OS::Neutron::FloatingIP
    properties:
      floating_network: { get_param: public_net }
      port_id: { get_resource: jump_box_port }

  server:
    type: OS::Nova::Server
    properties:
      name: Server
      image: { get_param: image }
      flavor: { get_param: flavor }
      key_name: { get_param: key_name }
      networks:
        - network: { get_resource: private_net }
          subnet: { get_resource: private_subnet }
      security_groups:
        - { get_resource: allow_ssh }
        - { get_resource: allow_icmp }
        - { get_resource: allow_web }
      user_data: |
        #!/bin/bash
        sudo http_proxy=http://swg.oc.removed.com:8080 apt update
        sudo http_proxy=http://swg.oc.removed.com:8080 apt install -y nginx

  lb:
    type: OS::Neutron::LBaaS::LoadBalancer
    properties:
      vip_subnet: { get_resource: private_subnet }

  listener:
    type: OS::Neutron::LBaaS::Listener
    properties:
      loadbalancer: {get_resource: lb}
      protocol: HTTP
      protocol_port: 80

  pool:
    type: OS::Neutron::LBaaS::Pool
    properties:
      listener: {get_resource: listener}
      lb_algorithm: ROUND_ROBIN
      protocol: HTTP

  pool_member:
    type: OS::Neutron::LBaaS::PoolMember
    properties:
      pool: { get_resource: pool }
      address: { get_attr: [server, first_address] }
      protocol_port: 80
      subnet: { get_resource: private_subnet }

  lb_port:
    type: OS::Neutron::Port
    properties:
      network_id: { get_resource: private_net }
      fixed_ips:
        - subnet_id: { get_resource: private_subnet }

outputs:
  server_private_ip:
    description: IP address of server in private network
    value:
      get_attr: [server, first_address]

  jumpbox_private_ip:
    description: IP address of jumpbox in private network
    value:
      get_attr: [jumpbox, first_address]

  jumpbox_public_ip:
    description: Floating IP address of jumpbox in public network
    value:
      get_attr: [jump_box_floating_ip, floating_ip_address]

Question information

Language:
English Edit question
Status:
Open
For:
Ubuntu neutron-lbaas Edit question
Assignee:
No assignee Edit question
Last query:
2018-05-12
Last reply:

Can you help with this problem?

Provide an answer of your own, or ask Jean Duminy for more information if necessary.

To post a message you must log in.