TIL: Go HTTP client reuse connection
I was recently working on a weird problem related to the build-in net/http package.
# How to reuse HTTP client’s connection?
As described in godoc (opens new window), by default the http.Client would reuse the TCP connection, but only if you read all the data from the http.Response Body and close the http.Response Body as well:
If the returned error is nil, the Response will contain a non-nil Body which the user is expected to close. If the Body is not both read to EOF and closed, the Client’s underlying RoundTripper (typically Transport) may not be able to re-use a persistent TCP connection to the server for a subsequent “keep-alive” request.
To disable reuse of the connection, one could:
- Set http.Response (opens new window) Close to true
- Use http.Transport (opens new window) and set DisableKeepAlives to true
But still, it is recommended to reuse the connection, it saves some time and resources to establish a new connection.
# If the server closes the reused connection on the client-side?
OK, now we all know that by default the connection won’t be easily closed in the client-side, but what if the server closes that connection?
Unfortunately, the error handling is not very ideal in the current implementation (Go 1.12 and Go 1.13). All you’ll get is an EOF error, which is caused by reading the data from the connection buffer.
This EOF error is actually a transportReadFromServerError when *persistConn is processing it: https://github.com/golang/go/blob/release-branch.go1.12/src/net/http/transport.go#L1625 (opens new window)
But back to the *Transport, it only returns the wrapped error (the EOF): https://github.com/golang/go/blob/ecde0bfa1fb11328133bb335af80fc2a48a8f82a/src/net/http/transport.go#L574 (opens new window)
In the end, the user would only receive an *url.Error which is generated by the uerr (opens new window) anonymous function, but just contains an EOF error.
# Tragedy: the race condition
The weird problem I have recently is related to the two descriptions above.
In my service, we send data periodically to the external database every n seconds, and the connections have been reuse without a problem, but:
The server closes the idle connection after n seconds
And then, here comes the race condition …
What would the client do first:
- It founds out the connection has been closed by the server
- It tries to send the data to the external database
Of course, there’s no problem for the first case, the client would just open a new connection, and my data is sent via this new connection — everything is just fine.
However, if it tries to send the data first, then for sure the data won’t be stored into the database, but even worse — you only get an EOF error. What the hell does that EOF mean!
# The solution, hotfix, or workaround
As simple as you could imagine: Retry.
It’s just making no sense to receive an EOF error, or more precisely — an *url.Error.