How can I verify my haproxy.cfg is correctly configured to load balance openstack services?

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux Openstack Platform

Issue

  • Various services in my openstack environment are configured for load balancing using haproxy. We have an haproxy.cfg deployed by the deployment tool. How can I verify if it's configured with the recommended options for these services, especially for MariaDB/Galera, redis and etc?
  • I am planning to use an external load balancer instead of haproxy to load balance openstack and required non-openstack services. Is there any guide to check how various services need to be configured?

Resolution

haproxy.cfg provisioned by default using our deployment tool have three sections.

Global Section


Global section would look like as below. 
global
  daemon  
  group  haproxy
  log  /dev/log local0
  maxconn  10000
  pidfile  /var/run/haproxy.pid
  user  haproxy

In this section maxconn 10000 need to be taken care of if you are creating haproxy.cfg manually. Default maxconn may be too low for all services together. If you are using a third party load balancer, an equivalent parameter need to be updated.

Defaults section

This section would look like as below.

defaults
  log  global
  mode  tcp
  retries  3
  timeout  http-request 10s
  timeout  queue 1m
  timeout  connect 10s
  timeout  client 1m
  timeout  server 1m
  timeout  check 10s

These parameters need to be explicitly configured to override the defaults values as they may be low for an openstack deployment. To get more details on default values for each parameter and what does it mean, refere Content from www.haproxy.org is not included.official haproxy.cfg documentation

Proxy Section

This section configures details of each proxy. This involves Openstack services and non-openstack services. All proxies use the same set of parameters except mysql and redis. It's recommended you use the same parameters or equivalent one if third party load balancer is used for each proxy.

All Proxies except mysql and redis
  • All proxy section except mysql and redis have below configuration.
listen name
  bind ip1:port 
  bind ip2:port 
  server overcloud-controller-0 ip:9696 check fall 5 inter 2000 rise 2
  server overcloud-controller-1 ip:9696 check fall 5 inter 2000 rise 2
  server overcloud-controller-2 ip:9696 check fall 5 inter 2000 rise 2
MariaDB/Galera Proxy
  • Proxy section for mysql need to be configured as below. mysql need different set of parameters to make sure that write requests are forwarded to only one of the nodes to avoid write deadlock for database.
listen mysql
  bind ip:3306 
  option httpchk
  stick on dst
  stick-table type ip size 1000
  timeout client 90m
  timeout server 90m
  server overcloud-controller-0 ip:3306 backup check fall 5 inter 2000 on-marked-down shutdown-sessions port 9200 rise 2
  server overcloud-controller-1 ip:3306 backup check fall 5 inter 2000 on-marked-down shutdown-sessions port 9200 rise 2
  server overcloud-controller-2 ip:3306 backup check fall 5 inter 2000 on-marked-down shutdown-sessions port 9200 rise 2

  • stick-table: This is used for mariadb/galera proxy because we want active/passive behavior. All database traffic should go to the same back end database server at any given time. Technically this only has to be for database writes, but we're currently not distinguishing between reads and writes. A stick table is just a table internal to haproxy. When a connection is made to the galera proxy, haproxy will look at the destination address which is always the VIP for database and look to see if there is an entry in the stick table with that same destination address. There almost always will be. That table will also tell haproxy to which back end database server it should send traffic. The result is that all database traffic destined to the same address will go to the same back end server until it fails. This is why we use stick on dst.

  • backup: With the above in mind, consider the case where the stick table is not yet populated (ie. at startup or when VIP failover occurs). If there are many concurrent db connections being made before the stick table is populated, there is a chance that we will get connections being sent to different back end database servers. The solution is to set all database servers as backup. It is fine to set all servers as backup. Haproxy will simply use the first one that is available.

It is common to think that setting all but one of the servers to backup is the right thing. While that will work in many cases, it is not desirable here since we want 1 )active/passive, 2) to avoid distributing traffic to >1 db server when the stick-table is not populated, and 3) to not failback if the failed db server recovers. We get #3 but using a stick-table and setting 'backup' everywhere**.

** When setting backup to all of 3 nodes, each node won't be used till all of other server is unavailable, also only the first operational backup server is used.

Content from www.haproxy.org is not included.official haproxy.cfg documentation

backup
  When "backup" is present on a server line, the server is only used in load
  balancing when all other non-backup servers are unavailable. Requests coming
  with a persistence cookie referencing the server will always be served
  though. By default, only the first operational backup server is used, unless
  the "allbackups" option is set in the backend. See also the "allbackups"
  option.

  Supported in default-server: No
  • on-marked-down shutdown sessions: When a back end database server is detected to be down, all connections to that server are immediately closed. The database connections are very long-lived and we do not want lingering connections to the database that might be reused when the node recovers. If the current, active db server fails, all connections are terminated and will need to be reestablished to the new db server.
Redis Proxy
  • Example proxy section for redis. 
listen redis
  bind ip:6379 
  balance first
  option tcp-check
  tcp-check send info\ replication\r\n
  tcp-check expect string role:master
  timeout client 90m
  timeout server 90m
  server overcloud-controller-0 ip:6379 check fall 5 inter 2000 rise 2
  server overcloud-controller-1 ip:6379 check fall 5 inter 2000 rise 2
  server overcloud-controller-2 ip:6379 check fall 5 inter 2000 rise 2

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.