# [Solved]HaProxy - slow new connections, performance issues

## sebekk23

Hi

I recently made 2 nodes with mariadb galera on banana pi and one with haproxy (on Banana Pi smart router - separate machine) and i notice that when i'm connecting through haproxy node, new connections (i mean opening new connections) take a lot of time (much,much more time than direct)

Direct connect from Haproxy(Banana Pi smart router without haproxy) node works very fast but with Haproxy it takes forever  :Smile: .

Second thing that i want to solve is ... performance.

Sysbench test through direct connect (100 threads, no restriction or limit on connection count) gives me on both nodes

```
OLTP test statistics:

    queries performed:

        read:                            21546

        write:                           7695

        other:                           3078

        total:                           32319

    transactions:                        1539   (49.18 per sec.)

    deadlocks:                           0      (0.00 per sec.)

    read/write requests:                 29241  (934.43 per sec.)

    other operations:                    3078   (98.36 per sec.)

Test execution summary:

    total time:                          31.2928s

    total number of events:              1539

    total time taken by event execution: 3086.7654

    per-request statistics:

         min:                               1087.02ms

         avg:                               2005.70ms

         max:                               3504.86ms

         approx.  95 percentile:            2658.12ms

Threads fairness:

    events (avg/stddev):           15.3900/2.28

    execution time (avg/stddev):   30.8677/0.34

```

but with haproxy

```
OLTP test statistics:

    queries performed:

        read:                            16548

        write:                           5910

        other:                           2365

        total:                           24823

    transactions:                        1180   (35.33 per sec.)

    deadlocks:                           3      (0.09 per sec.)

    read/write requests:                 22458  (672.46 per sec.)

    other operations:                    2365   (70.82 per sec.)

Test execution summary:

    total time:                          33.3966s

    total number of events:              1180

    total time taken by event execution: 3296.7437

    per-request statistics:

         min:                                 57.14ms

         avg:                               2793.85ms

         max:                              20223.17ms

         approx.  95 percentile:           18962.81ms

Threads fairness:

    events (avg/stddev):           11.8000/12.21

    execution time (avg/stddev):   32.9674/0.35

```

max is 10 times slower through haproxy!!!

Some configuration and information:

1 sysbench machine config (this one connect through haproxy)

- Intel i5, SSD, 8GB or RAM DDR3 (Linux elitebook 3.18.11-gentoo #1 SMP Mon May 18 19:01:30 CEST 2015 x86_64 Intel(R) Core(TM) i5 CPU M 520 @ 2.40GHz GenuineIntel GNU/Linux)

command that is invoked on that machine:

```
sysbench --test=oltp  --num-threads=100 --max-time=30 --max-requests=0  --mysql-table-engine=InnoDB  --mysql-db=test --mysql-user=root --mysql-password=passs --mysql-host=r1.dom  --mysql-engine-trx=yes --oltp-table-size=2000000 run
```

2 sysbnech machine (that one with haproxy)

- BanaPi R1 (smart router) with Allwinner A20(2 cores, 1GHz each), SSD, 1GB or RAM DDR3 (Linux r1 3.4.104+ #1 SMP PREEMPT Thu Jan 8 15:40:40 CET 2015 armv7l ARMv7 Processor rev 4 (v7l) sun7i GNU/Linux)

command that is invoked on that machine:

```
sysbench --test=oltp  --num-threads=100 --max-time=30 --max-requests=0  --mysql-table-engine=InnoDB  --mysql-db=test --mysql-user=root --mysql-password=passs --mysql-host=node{one or two}.dom  --mysql-engine-trx=yes --oltp-table-size=2000000 run
```

on this machine i have 2 vlans one for external connection (my lan) and second for internal connections (mariadb nodes)

haproxy config

```
global

  # log redirection (syslog)

  #log 127.0.0.1 local0

  #log 127.0.0.1 local1 debug

  log /dev/log local0

  log /dev/log local1 notice

  #apache2

  tune.ssl.default-dh-param 2048

  # maximum of connections for haproxy

  maxconn 2048

  #maxpipes 512

  # chroot for security reasons

  chroot /var/lib/haproxy

  # user/group for haproxy process

  user haproxy

  group haproxy

  #command to use followed hatop -s /var/lib/haproxy/stats

  stats socket /var/lib/haproxy/stats mode 777 level admin

  # act as a daemon

  daemon

defaults

  # use gloval log declaration

  log global

  # default check type

  #mode http

  # only log when closing session

  # only log failed connections

  # retry 3 times before setting node as failed

  # redispatch traffic to other servers

  option tcplog

  option dontlognull 

  option redispatch

  # maximum connection for the frontend

  maxconn 1024

  # timeouts

  timeout client 1s

  timeout client-fin 1s

  timeout server 1s

  timeout connect 1s

  retries 3

  timeout queue 1m

  timeout check 20s

# enable web check health interface on port 80

listen haproxy 192.168.2.5:8080

  stats enable

  mode http

  fullconn 1024

  #stats hide-version

  #stats uri /stats

  #stats realm Haproxy\ Statistics

  # set credentials

  stats auth admin:password

# loadbalance on slaves  - we need acces from eth0_101 and eth0_102

frontend MDBG

  #bind 192.168.2.5:3306

  bind 0.0.0.0:3306

  timeout client 57600s

  #timeout client-fin 57600s

  default_backend mariadb-galera

backend mariadb-galera

  fullconn 1024

  #option splice-auto

  timeout server 57600s

  timeout connect 5s

  # use tcp method

  mode tcp

  # round robin mechanism

  #balance roundrobin

  balance leastconn

  # tcp keepalive (pipelining) on both side (clt/srv)

  option tcpka

  # perform http request

  #option httpchk

  # perform mariadb connection with haproxy user

  option mysql-check user haproxy

  # set all read only nodes

  # inter: interval of check in milliseconds

  # check port 9200 inter 2000 rise 3 fall 3

  server vulcanone 172.16.1.21:3306 check inter 2000 rise 3 fall 3 maxconn 512

  server vulcantwo 172.16.1.22:3306 check inter 2000 rise 3 fall 3 maxconn 512

frontend LB

   bind 192.168.2.5:80

   timeout client 57600s

   #timeout client-fin 57600s

   reqadd X-Forwarded-Proto:\ http

   default_backend apache2

frontend LBS

   #bind 0.0.0.0:443

   bind 192.168.2.5:443 ssl crt /etc/haproxy/ssl/localhost.pem

   timeout client 57600s

   timeout client-fin 57600s

   reqadd X-Forwarded-Proto:\ https

   default_backend apache2

backend apache2

  fullconn 1024

  timeout server 57600s

  timeout connect 5s

  redirect scheme https if !{ ssl_fc }

  timeout http-request    20

  timeout http-keep-alive 30

  mode http

  balance roundrobin

  cookie PHPSESSID prefix indirect nocache

  option httplog

  option http-server-close

  option httpclose

  option forwardfor

  #http-request set-header X-Forwarded-Port %[dst_port]

  #http-request add-header X-Forwarded-Proto https if { ssl_fc }

  option httpchk HEAD / HTTP/1.1\r\nHost:localhost

  server markone 172.16.1.11:80 cookie markone check maxconn 512

  server marktwo 172.16.1.12:80 cookie marktwo check maxconn 512

  #server web4-srv 192.168.0.124:80 check backup # backup fail-over Server, If three of the above fails this will be activated.

```

Ping from Intel machine to haproxy machine:

```
h3x0r@elitebook ~ $ ping r1.dom

PING r1.dom (192.168.2.5) 56(84) bytes of data.

64 bytes from r1.dom (192.168.2.5): icmp_seq=1 ttl=64 time=0.335 ms

64 bytes from r1.dom (192.168.2.5): icmp_seq=2 ttl=64 time=0.255 ms

64 bytes from r1.dom (192.168.2.5): icmp_seq=3 ttl=64 time=0.320 ms

64 bytes from r1.dom (192.168.2.5): icmp_seq=4 ttl=64 time=0.194 ms
```

Ping from haproxy machine to mariadb nodes:

```
[h3x0r@r1 ~]$ ping vulcanone.dom

PING vulcanone.dom (172.16.1.21) 56(84) bytes of data.

64 bytes from vulcanone.dom (172.16.1.21): icmp_seq=1 ttl=64 time=0.339 ms

64 bytes from vulcanone.dom (172.16.1.21): icmp_seq=2 ttl=64 time=0.225 ms

^C

--- vulcanone.dom ping statistics ---

2 packets transmitted, 2 received, 0% packet loss, time 1001ms

rtt min/avg/max/mdev = 0.225/0.282/0.339/0.057 ms

[h3x0r@r1 ~]$ ping vulcantwo.dom

PING vulcantwo.dom (172.16.1.22) 56(84) bytes of data.

64 bytes from vulcantwo.dom (172.16.1.22): icmp_seq=1 ttl=64 time=0.432 ms

64 bytes from vulcantwo.dom (172.16.1.22): icmp_seq=2 ttl=64 time=0.283 ms

64 bytes from vulcantwo.dom (172.16.1.22): icmp_seq=3 ttl=64 time=0.329 ms

```

I have 1Gb network between my machines, i never notice any problems with CPU or RAM or any machine - so machine resources are fine (during heavy load haproxy machine has almost all ram free and CPU around 20-25%, nodes - of course - almost 200% CPU, 50% free ram, intel machine ...  :Smile:  no load at all).

Is there any way to make haproxy more "transparentable" ? How to reduce this performance issues ?Last edited by sebekk23 on Sat Jun 27, 2015 7:08 pm; edited 1 time in total

----------

## sebekk23

Ok - i've got it!!

I've eliminated delays in connection time by moving haproxy bind address from external (external VLAN) to internal  :Smile: 

I think that iptables was an issue (netfilter has been working too long ?) - now haproxy listen on internal IP and iptables doing DNAT to this IP from external one  :Very Happy: 

Problem solved  :Wink: 

----------

