So I’ve been debugging a problem for a better part of today. We first noticed an issue when our test suite was taking forever to finish and it turns out that a certain server we integrated was timing out on every single test. We initially chalked it up to the server being slow, but when we have 10 tests each taking 60 seconds to timeout, it adds a big chunk of time to run our test suites.
To provide some more background, our Rails on Ruby app uses ActiveMerchant to connect to NMI to process transactions. We kept getting the following error: “The connection to the remote server timed out”
The weird thing was that it was only happening on our Macs running OSX 10.8.3 (Mountain Lion), but not on our production server which is running Ubuntu.
So I decided to spend some time debugging the issue. I found out if I switched back to ruby-1.9.3-p392, everything worked fine. I thought maybe my ruby was compiled incorrectly, so I recompiled ruby-1.9.3-p429, but that didn’t seem to fix the problem.
Tracing the code:
which eventually generates an Net::HTTP connection and makes the SSL request.
So I wrote a little test to see what happens:
h = Net::HTTP.new('secure.networkmerchants.com', 443).tap do |http|
http.use_ssl = true
In p392, I would get:
#<Net::HTTPOK 200 OK readbody=true>
But in p429, I would get:
Errno::ECONNRESET: Connection reset by peer - SSL_connect
Searching for that error string, I eventually came upon openssl. I found out that in p429, it had switched to using homebrew’s version of openssl (version 1.0.1e) instead of the system’s version of openssl (version 0.9.8r).
Using openssl 0.9.8r, everything worked fine, but when using openssl 1.0.1e, the connection was timing out and getting the following error:
$ openssl s_client -connect secure.networkmerchants.com:443
no peer certificate available
No client certificate CA names sent
SSL handshake has read 0 bytes and written 322 bytes
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
I contacted the openssl users mailing list and got back the following response:
This is most likely another case of the frequently reported (and discussed) issue that 1.0.1 implements TLS1.2, which has more ciphersuites enabled by default and additional extensions, which together make the ClientHello bigger, and some server implementations apparently can’t cope. It appears in at least many cases the cutoff is 256 bytes, suggesting these servers don’t handle 2-byte length right.
It’s unlikely that this would be explicitly configured on a server, rather it would be an implementation flaw that previously did not cause a problem. It might occur in an older version of server software fixed in a newer version.
For many details see
Short answer is that restricting to TLS1(.0), and/or a smaller list of ciphersuites (but still enough to intersect with the server), likely works. Both do for me using 1.0.1e to your example host. You can use -msg in s_client to see exactly how much (and what) is sent for different options.
So I tried setting the ssl version to :TLSv1, but that didn’t seem to work. Setting it to ssl version to SSLv3 did though.
http.ssl_version = :SSLv3
Following the example from Forcing SSL TLSv1, I was able to override the ssl_version of the http connection that ssl_post creates:
class SSLv3Connection < ActiveMerchant::Connection
http.ssl_version = :SSLv3