haproxy返回的http头中没有keep-alive

作者:李佶澳  更新时间:

  问题    haproxy    刷新

目录

现象

haproxy有两个规则,指向的是相同的backend,nginx服务器。

第一个规则是tcp的:

listen nginx-0-x-project.paas.XXXX.cn
	bind 10.39.1.67:20941
	mode tcp
	balance roundrobin
	server nginx-2853504753-u0git 192.168.60.131:80 maxconn 500
	server nginx-2853504753-ni62e 192.168.237.228:80 maxconn 500
	server nginx-2853504753-2ck44 192.168.239.75:80 maxconn 500
	server nginx-2853504753-e41uz 192.168.162.3:80 maxconn 500
	server nginx-2853504753-tqs7j 192.168.156.151:80 maxconn 500
	server nginx-2853504753-gqkcg 192.168.155.141:80 maxconn 500

通过nginx-0-x-project.paas.XXXX.cn:20941访问得到的http头中,有”keep-alive”

$curl -I http://nginx-0-x-project.paas.XXXX.cn:20941
HTTP/1.1 404 Not Found
Server: nginx/1.13.1
Date: Fri, 11 Aug 2017 09:03:55 GMT
Content-Type: text/html
Content-Length: 169
Connection: keep-alive

第二个规则backend server与第一个规则是相同的,但是是http模式:

listen defaulthttp
	bind 10.39.1.67:80
	mode http
	option forwardfor       except 127.0.0.0/8
	...
	acl x-project-nginx-nginx-0 hdr(host) -i  x-project.XXXX.cn
	use_backend x-project-nginx-nginx-0 if x-project-nginx-nginx-0
...
backend x-project-nginx-nginx-0
	server nginx-2853504753-u0git 192.168.60.131:80 cookie nginx-2853504753-u0git check maxconn 500
	server nginx-2853504753-ni62e 192.168.237.228:80 cookie nginx-2853504753-ni62e check maxconn 500
	server nginx-2853504753-2ck44 192.168.239.75:80 cookie nginx-2853504753-2ck44 check maxconn 500
	server nginx-2853504753-e41uz 192.168.162.3:80 cookie nginx-2853504753-e41uz check maxconn 500
	server nginx-2853504753-tqs7j 192.168.156.151:80 cookie nginx-2853504753-tqs7j check maxconn 500
	server nginx-2853504753-gqkcg 192.168.155.141:80 cookie nginx-2853504753-gqkcg check maxconn 500

通过x-project.XXXX.cn访问,得到的http头中,没有”keep-alive”:

$ curl -I http://x-project.XXXX.cn
HTTP/1.1 404 Not Found
Server: nginx/1.13.1
Date: Fri, 11 Aug 2017 15:43:08 GMT
Content-Type: text/html
Content-Length: 169

调查

haproxy中的option http-keep-alive用来控制http连接是否keep-alive,默认是enabled的。

同时,在haproxy-doesnt-keep-alive-http-connection看到一个说法:

It would appear that your assumption is that haproxy will add a:
    Connection: keep-alive
header. That is not the case. Instead, the keep-alive mode (KAL, default in 1.5.x) will just 
refrain from closing connections or adding Connection: close headers.
Your configuration is probably fine. You can test keepalive using nc for example, or ab -k.

意思是说,连接实际是保活的,但是haproxy不会在http响应头中添加”keep-alive”。

用ab进行测试

首先,使用非keep-alive的方式访问第二个规则的服务地址:

ab -n 100000  http://x-project.XXXX.cn/

在haproxy上,可以看到创建了非常多的连接, 10.4.110.62是client端的ip:

/ # netstat -nt |grep 10.4.110.62
tcp        0      0 10.39.1.67:80           10.4.110.62:58808       TIME_WAIT   -
tcp        0      0 10.39.1.67:80           10.4.110.62:59174       TIME_WAIT   -
tcp        0      0 10.39.1.67:80           10.4.110.62:60200       TIME_WAIT   -
tcp        0      0 10.39.1.67:80           10.4.110.62:58870       TIME_WAIT   -
...

然后,使用keep-alive的方式:

ab -n 100000 -k http://x-project.XXXX.cn/

在haproxy上,可以看到只建立了一个连接:

/ # netstat -nt |grep 10.4.110.62
tcp        0     36 10.39.1.67:22           10.4.110.62:57158       ESTABLISHED

可以看到默认就是支持keep-alive,但是支持不等于会在响应头中写入Connection: keep-alive

但是为什么第一条规则中添加了Connection: keep-alive呢?怀疑是因为第一个规则是tcp,第二个是http的缘故。

验证tcp和http

准备了一个测试环境,两个listen规则指向同一个backend,一个是http模式,一个是tcp模式。

haproxy配置:

listen lb1
    bind *:5001
    mode http
    balance roundrobin
    server  lb1-1 nginx:80 maxconn 500

listen lb2
    bind *:5002
    mode tcp
    balance roundrobin
    server  lb2-1 nginx:80 maxconn 500

访问结果如下:

mode http:

~ # curl -I 127.0.0.1:5001
HTTP/1.1 200 OK
Server: nginx/1.13.1
Date: Fri, 11 Aug 2017 10:11:54 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 30 May 2017 17:15:54 GMT
ETag: "592da8ca-264"
Accept-Ranges: bytes

mode tcp:

~ # curl -I 127.0.0.1:5002
HTTP/1.1 200 OK
Server: nginx/1.13.1
Date: Fri, 11 Aug 2017 10:11:56 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 30 May 2017 17:15:54 GMT
Connection: keep-alive
ETag: "592da8ca-264"
Accept-Ranges: bytes

直接访问后端的情况是:

$curl -I nginx
HTTP/1.1 200 OK
Server: nginx/1.13.1
Date: Fri, 11 Aug 2017 10:08:00 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 30 May 2017 17:15:54 GMT
Connection: keep-alive
ETag: "592da8ca-264"
Accept-Ranges: bytes

看起来好像是因为mode的原因,但是在抓包的时候,又发现了新情况!

抓包验证

执行ab -n 100000 -k http://x-project.XXXX.cn/时,抓取的报文内容:

GET / HTTP/1.0
Connection: Keep-Alive
Host: x-project.XXXX.cn
User-Agent: ApacheBench/2.3
Accept: */*

HTTP/1.1 404 Not Found
Server: nginx/1.13.1
Date: Fri, 11 Aug 2017 14:57:57 GMT
Content-Type: text/html
Content-Length: 169
Connection: keep-alive

<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.13.1</center>
</body>
</html>

可以很明确的看到,返回的头中有Connection: keep-alive。但是用curl访问时,却没有。

执行curl -v -X GET -H "Connection: Keep-Alive" http://x-project.XXXX.cn/,抓取的报文:

GET / HTTP/1.1
Host: x-project.XXXX.cn
User-Agent: curl/7.51.0
Accept: */*
Connection: Keep-Alive

HTTP/1.1 404 Not Found
Server: nginx/1.13.1
Date: Fri, 11 Aug 2017 15:31:48 GMT
Content-Type: text/html
Content-Length: 169

<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.13.1</center>
</body>
</html>

仔细比对后会发现,curl使用的协议是HTTP/1.1,而ab中则是HTTP/1.0

强制curl使用HTTP1.0,执行curl -v -0 -X GET -H "Connection: Keep-Alive" http://x-project.XXXX.cn/得到:

GET / HTTP/1.0
Host: x-project.XXXX.cn
User-Agent: curl/7.51.0
Accept: */*
Connection: Keep-Alive

HTTP/1.1 404 Not Found
Server: nginx/1.13.1
Date: Fri, 11 Aug 2017 15:34:28 GMT
Content-Type: text/html
Content-Length: 169
Connection: keep-alive

<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.13.1</center>
</body>
</html>

可以看到返回头中有keep-alive。

结论

对于haproxy而言,有两种情况可以在响应头中看到keep-alive:

1.  mode tcp,backend返回的响应头中带有keep-alive
2.  mode http的时候,client使用http1.0,并且请求头中包含`Connection: Keep-Alive`

另外还可以通过http-reponse指令,强行插入响应头,例如:

listen lb1
    bind *:5001
    mode http
    balance roundrobin
    http-response add-header Connection keep-alive
    server  lb1-1 nginx:80 maxconn 500

参考

  1. option http-keep-alive
  2. haproxy-doesnt-keep-alive-http-connection
  3. haproxy http-response

关注加微信,一般不闲聊(直接说事)

Copyright @2011-2019 All rights reserved. 转载请添加原文连接,合作请加微信lijiaocn或者发送邮件: [email protected],备注网站合作

友情链接:  微信公众号精选文章  发现知识星球