排查SpringCloudGateway的readAddress(..) failed: Connection reset by peer问题（利用arthas）

阅读：评论：0

问题

[id:48b8a8f5-1, L:/网关:37187 - R:应用/应用:应用端口] The connection observed an error, the request cannot be retried as the headers/body were sent
ioty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer
[2021-12-28T11:31:06,329] DEBUG  Stopping retries since predicate returned false, retry context: iteration=1 exception=ioty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer backoff={0ms}
[2021-12-28T11:31:06,330] ERROR  ==>返回错误信息
ioty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peerSuppressed: publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):|_ checkpoint ⇢ org.s.reactive.CorsWebFilter [DefaultWebFilterChain]...|_ checkpoint ⇢ HTTP POST "接口" [ExceptionHandlingWebHandler]
Stack trace:

分析

大体一看，是netty报的io异常，然后被reactor-netty抛出到网关返回了。
reactor-netty是SCG的底层依赖，去

发现有同样的issue。
大体总结就是

后端应用tomcat有个keepalivetimeout，默认与connecttimeout相同，为20s。也就是说闲置20s的连接会被断开丢弃，主要跟keepalive长连接有关。
SCG网关有个连接池，存放与后端应用tomcat的通道连接，默认不释放回收，取用连接方式为FIFO先入先出（取用最早建立的连接）
以上两者，其中至少一个没有配置正确，导致SCG连接池释放闲置时间大于tomcat与网关的断开时间，就会导致浏览器向网关请求，网关获取到的连接是已经断开的连接，继续向tomcat请求，tomcat发现使用的是废弃的连接则返回reset，网关抛出异常，但后续接口不受影响。
可简单参考下图。.

解决

①配置SCG网关

spring:cloud:gateway:httpclient:response-timeout: 10spool:type: fixedmax-idle-time: 5000max-connections: 200acquire-timeout: 45000

②配置网关启动参数：reactor-netty取用连接的规则LIFO

-Dreactorty.pool.leasingStrategy=lifo

③配置tomcat的闲置断开时间

server:tomcat:connection-timeout: 10000 #根据需要

⑤配置nacos 元数据

spring: cloud:nacos:discovery:metadata:response-timeout: 10000connect-timeout: 3000

④配合nginx关于keepalive的配置

upstream gateway {
#	ip_hash;:xx weight=1;:xx weight=1;keepalive 100;keepalive_requests 30000;keepalive_timeout 300s;
}

location /xxx{
proxy_pass gateway;
proxy_http_version 1.1;
proxy_set_header Connection "";
}

用arthas测试(其他)

使用arthas
.html

curl -O .jar
java -jar arthas-boot.jar

附加到网关

 vmtool --action getInstances --classLoaderClass org.springframework.boot.loader.LaunchedURLClassLoader --className  sources.PooledConnectionProvider --express 'instances[0].defaultPoolFactory.leasingStrategy'