haproxy的配置文件存在重复项目,导致reload失败,新规则未生效

作者: 李佶澳   转载请保留:原文地址   发布时间:2017/12/21 11:12:14

现象

在haproxy中做了如下配置,通过host转发请求:

listen defaulthttp
    bind 10.39.1.45:80
    mode http
    option forwardfor       except 127.0.0.0/8
    errorfile 503 /etc/haproxy/errors/503.http
    redirect scheme https code 301 if { hdr(Host) -i  www.lijiao.cn pay.lijiao.cn } !{ ssl_fc }
    acl lijiao-xdx-test-8230 hdr(host) -i  lijiao-xdx-test.inter.lijiaocn.cn
    use_backend lijiao-xdx-test-8230 if lijiao-xdx-test-8230
    acl lijiao-d-multi-lijiao-plat-80 hdr(host) -i  lijiao-d-multi-lijiao-plat.inter.lijiaocn.cn
    use_backend lijiao-d-multi-lijiao-plat-80 if lijiao-d-multi-lijiao-plat-80
    acl dashboard-demo-multi-lijiao-plat-8080 hdr(host) -i  dashboard-demo-multi-lijiao-plat.inter.lijiaocn.cn
    use_backend dashboard-demo-multi-lijiao-plat-8080 if dashboard-demo-multi-lijiao-plat-8080
    acl se-file-lijiaocn-prod-8080 hdr(host) -i  se-file-lijiaocn-prod.inter.lijiaocn.cn
    use_backend se-file-lijiaocn-prod-8080 if se-file-lijiaocn-prod-8080
    acl portal-d-multi-lijiao-plat-80 hdr(host) -i  portal-d-multi-lijiao-plat.inter.lijiaocn.cn
    use_backend portal-d-multi-lijiao-plat-80 if portal-d-multi-lijiao-plat-80

然后通过域名访问的时候,有一部分返回503错误。

调查

查看haproxy监听的端口,80端口被进程4265使用。

# netstat -lntp |grep 80
tcp        0      0 10.39.1.45:18090        0.0.0.0:*               LISTEN      4265/haproxy
tcp        0      0 10.39.1.45:17805        0.0.0.0:*               LISTEN      4265/haproxy
tcp        0      0 10.39.1.45:28080        0.0.0.0:*               LISTEN      4265/haproxy
tcp        0      0 10.39.1.45:80           0.0.0.0:*               LISTEN      4265/haproxy
tcp        0      0 10.39.1.45:18004        0.0.0.0:*               LISTEN      4265/haproxy
tcp        0      0 10.39.1.45:28090        0.0.0.0:*               LISTEN      4265/haproxy
tcp        0      0 10.39.1.45:18080        0.0.0.0:*               LISTEN      4265/haproxy
tcp        0      0 10.39.1.45:51809        0.0.0.0:*               LISTEN      4265/haproxy
tcp        0      0 10.39.1.45:38082        0.0.0.0:*               LISTEN      4265/haproxy

查看系统上的所有haproxy进程:

# ps aux
PID   USER     TIME   COMMAND
   41 haproxy    2:34 haproxy -f /etc/haproxy/haproxy.cfg -db -sf 25025
   56 haproxy    0:49 haproxy -f /etc/haproxy/haproxy.cfg -db -sf 41 41
   90 haproxy    0:47 haproxy -f /etc/haproxy/haproxy.cfg -db -sf 73 41 56 73
 1574 haproxy    0:26 haproxy -f /etc/haproxy/haproxy.cfg -db -sf 1557 41 56 90 775 843 945 1217 1285 1438 1455 1540 1557 28619 31968 32036 32189 32223 32257 32359 32648 32682 32716
 4265 haproxy    3:58 haproxy -f /etc/haproxy/haproxy.cfg -db -sf 4248 41 56 90 1574 2004 2021 2429 2446 2463 2888 3058 3177 3313 3381 3466 3483 3500 3551 3568 3670 3789 3857 3908 3976 4027 4078 4129 418
 4703 haproxy    0:25 haproxy -f /etc/haproxy/haproxy.cfg -db -sf 4686 41 56 90 1285 1574 3053 3903 4107 4448 4499 4516 4618 4652 4686 28619 32036 32189 32223 32359 32716
 5044 haproxy    0:25 haproxy -f /etc/haproxy/haproxy.cfg -db -sf 5027 41 56 90 1285 1574 3053 3903 4107 4618 4652 4703 4891 5010 5027 28619 32036 32189 32223 32359 32716
 6319 haproxy    0:21 haproxy -f /etc/haproxy/haproxy.cfg -db -sf 6302 41 56 90 1285 1574 4652 4703 5044 5248 5401 5435 5809 5979 6098 6149 6234 6285 6302 28619 32036 32189 32223 32359 32716
 6574 haproxy    0:23 haproxy -f /etc/haproxy/haproxy.cfg -db -sf 6557 41 56 90 1574 4652 4703 5044 5435 5809 5979 6098 6149 6234 6319 6540 6557 28619 32036 32189 32223 32716
 6795 haproxy    0:23 haproxy -f /etc/haproxy/haproxy.cfg -db -sf 6778 41 56 90 1574 4652 4703 5044 5435 6149 6234 6319 6574 6676 6778 28619 32036 32189 32223 32716
 9311 haproxy    0:21 haproxy -f /etc/haproxy/haproxy.cfg -db -sf 9294 41 56 90 1574 4703 5044 6319 6574 6795 7390 7611 8325 9107 9124 9294 28619 32189 32223 32716
28619 haproxy    0:30 haproxy -f /etc/haproxy/haproxy.cfg -db -sf 28602 41 56 90 243 430 617 26868 27582 28296 28330 28364 28398 28585 28602
32189 haproxy    0:27 haproxy -f /etc/haproxy/haproxy.cfg -db -sf 32172 41 56 90 28619 30421 30455 31203 31866 31883 31934 31968 32036 32070 32155 32172
32223 haproxy    0:28 haproxy -f /etc/haproxy/haproxy.cfg -db -sf 32206 41 56 90 28619 30421 30455 31203 31866 31883 31934 31968 32036 32070 32189 32206
32716 haproxy    0:23 haproxy -f /etc/haproxy/haproxy.cfg -db -sf 32699 41 56 90 28619 31203 31934 31968 32036 32070 32189 32223 32257 32359 32376 32393 32546 32563 32580 32648 32682 32699

发现4254进程不是最新的haproxy进程,之前遇到类似的问题:

使用端口复用(SO_REUSEPORT)、反复对haproxy进行reload操作,导致访问haproxy间歇性返回503

与上次不同的是,haproxy的日志里有报错,还需要仔细调查一下。

[ALERT] 354/110745 (9900) : Parsing [/etc/haproxy/haproxy.cfg:1466]: proxy 'test-new-1-lijiao-new-qas.inter.lijiaocn.cn' has the same name as proxy 'test-new-1-lijiao-new-qas.inter.lijiaocn.cn' declared at /etc/haproxy/haproxy.cfg:1378.
[ALERT] 354/110745 (9900) : Parsing [/etc/haproxy/haproxy.cfg:1652]: proxy 'web-manager-0-lijiao-new-qas.inter.lijiaocn.cn' has the same name as proxy 'web-manager-0-lijiao-new-qas.inter.lijiaocn.cn' declared at /etc/haproxy/haproxy.cfg:1619.
[ALERT] 354/110745 (9900) : Parsing [/etc/haproxy/haproxy.cfg:1672]: proxy 'test-new-0-lijiao-new-qas.inter.lijiaocn.cn' has the same name as proxy 'test-new-0-lijiao-new-qas.inter.lijiaocn.cn' declared at /etc/haproxy/haproxy.cfg:1218.
[ALERT] 354/110745 (9900) : Parsing [/etc/haproxy/haproxy.cfg:1692]: proxy 'web-manager-0-lijiao-new-qas.inter.lijiaocn.cn' has the same name as proxy 'web-manager-0-lijiao-new-qas.inter.lijiaocn.cn' declared at /etc/haproxy/haproxy.cfg:1619.
[ALERT] 354/110745 (9900) : Parsing [/etc/haproxy/haproxy.cfg:1722]: proxy 'test-new-0-lijiao-new-qas.inter.lijiaocn.cn' has the same name as proxy 'test-new-0-lijiao-new-qas.inter.lijiaocn.cn' declared at /etc/haproxy/haproxy.cfg:1218.
[ALERT] 354/110745 (9900) : Parsing [/etc/haproxy/haproxy.cfg:1748]: proxy 'web-manager-0-lijiao-new-qas.inter.lijiaocn.cn' has the same name as proxy 'web-manager-0-lijiao-new-qas.inter.lijiaocn.cn' declared at /etc/haproxy/haproxy.cfg:1619.
[ALERT] 354/110745 (9900) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg
[ALERT] 354/110745 (9900) : Fatal errors found in configuration.

检查发现haproxy的配置文件中存在重名的listen配置的时候,haproxy会直接报错,启动不成功。

例如配置文件中有下面的重复项:

listen service-0-devops.idev.lijiaocn.cn
    bind 10.39.0.105:31796
    mode tcp
    balance roundrobin
    server service-2790983151-624tf 192.168.185.14:80 maxconn 2000
listen service-0-devops.idev.lijiaocn.cn
    bind 10.39.0.105:31796
    mode tcp
    balance roundrobin
    server service-2790983151-624tf 192.168.185.14:80 maxconn 2000

运行haproxy时,会报错,haproxy进程退出:

# haproxy -f /haproxy.cfg  -db -sf 93
[ALERT] 354/160618 (123) : Parsing [/haproxy.cfg:94]: proxy 'service-0-devops.idev.lijiaocn.cn' has the same name as proxy 'service-0-devops.idev.lijiaocn.cn' declared at /haproxy.cfg:89.
[ALERT] 354/160618 (123) : Error(s) found in configuration file : /haproxy.cfg
[ALERT] 354/160618 (123) : Fatal errors found in configuration.

参考

  1. 使用端口复用(SO_REUSEPORT)、反复对haproxy进行reload操作,导致访问haproxy间歇性返回503

欢迎加微信,最好备注姓名和方向

QQ交流群

区块链实践互助QQ群:576555864

Kubernetes实践互助QQ群:947371129

Prometheus实践互助QQ群:952461804

API网关Kong实践互助QQ群:952503851

Ansible实践互助QQ群:955105412

Copyright @2011-2018 All rights reserved. 转载请添加原文连接,合作请加微信lijiaocn或者发送邮件: lijiaocn@foxmail.com,备注网站合作 友情链接: lijiaocn github.com