The problem is not with OpenSSL, but with JSSE (Java's SSL stack). We use JSSE for the initial request. If that request fails, we do not proceed further. There is a small number of sites, like yours, where we get the handshake_failure message, and it's an JSSE interoperability issue. Because JSSE supports TLS 1.0 we can assess servers that do not support SSL v3. I suspect the problem is somewhere else.
If you have a better idea of the problem, please forward me the details. If not, I will proceed to investigate and fix the problem.
Edit: Maybe you meant SSL v2 handshake?
Ahh, your are using a Java implementation. I guess it works the same way as the older OpenSSL versions then. We try to stay away from everything that has something to do with Java, so it's hard to tell how your implementation exactly works.
You are right, we meant SSLv2 because SSLv2 support is completly disabled on our server (disabled during compilation of OpenSSL).
OpenSSL is horrible when it comes to server security. When you try to disable SSLv3 during the compilation of OpenSSL, you will also loose support for TLSv1.0. And if we do that, only our own FLWEB webbrowser, the latest Opera and the latest IE could connect to our sites.
So, I think this is the solution to your Java implementation. When it cannot handshake with SSLv2, you must assume SSLv2 is completly disabled (an give extra credits to the servers that doesn't support it, because it makes clear that the user is using a very old webbrowser).
The best way to do the testing in this following order:
* Check for TLSv1.2 support
* Check for TLSv1.1 support
* Check for TLSv1.0 support (Required for insecure browsers like FF and GC. They can't handle TLSv1.1 or TLSv1.2)
* Check for SSLv3 support (and continue checking when it cannot do a handshake.)
* Check for SSLv2 support (and continue checking when it cannot do a handshake)
I hope this will be enough information to solve those problems.
As far as I can tell, the issue is that there are no common cipher suites in the initial handshake. I see that you support Camellia, but do you support any other cipher suites? I've noticed that your site does not work in Internet Explorer, for example.
There may be another issue, which is making my troubleshooting more difficult. When I test for cipher suite support in bulk, I can't even negotiate Camellia. Rate limiting? Because of it I cannot easily inspect your server to determine which cipher suites are supported.
Edit: Ah, no; I forgot to enable full protocol testing. I can see you supporting:
Is that correct?
Yes, we only support AES-256, Camellia and cbc/gcm modes, which should be supported by any modern webbrowser.
To use IE, you first have to enable TLSv1.1 and TLSv1.2 support. I haven't checked if the latest IE version enables them by default. Maybe I should do that again anytime soon, although we Linux users cannot really stand working or testing in Windows
The reason why it works is because we now also accept insecure/old handshakes for compatability reasons. It's not that we want to, but otherwise we cannot provide SSL encryption for customers who do not already use the Fortress Linux Operating system.
The funny thing is that our own web-browser (FLWEB) is the only webbrower that can handle the latest and most secure handshakes, TLS ciphers, MAC's, EC's etc. These encryption technolgies are not even supported yet by Opera, IE, Firefox, Chrome or Safari.
We can offer you paid support and even a test web server (containing the most powerful SSL encryption possible today) for your SSL check script.
We've monitored the data that was send from your servers and we found quite some mistakes in the protocol handling and some other test exchanges. The Qualys SSL script also cannot handle the true TLSv1.2 encryption of our test server.
Please contact us by email if you're interested. The Quals SSL checker is a pretty good program and I assume you want it to stay this way.
We are now in the top-rated list of most secure SSL servers. But we have noticed that something is still missing from your scanner:
Support for the Elliptic Curve Diffie-Hellman Exchange with AES256 Galios/Counter mode.
This GCM variant is the most secure SSL encryption method available today, though it is only supported by our own latest FLWEB web-browser. But eventually all browsers should support this TLS cipher.
The scanner should also give a higher rating to servers in the server top-lists when this GCM cipher is supported and when the DH key is at least 4096 Bits.
We recently discovered that almost any other TLS cipher which uses DH or just RSA are vulnerable to a new TLS/SSL attack. But we cannot expose that for the moment
Our servers currently support RSA 8192/16384/32768 bits certificates with SHA512withRSA signing. Problem is that no CA supports them at the moment.
Maybe you know a CA who does support it?
Can you be more specific about what's missing? At the moment, the test currently lists your server as supporting 0xc030, which is TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384. We test for all suites specified in RFC 5289. If you support a suite that we're not detecting them, please send me a PCAP of one successful handshake and I will investigate.
I expect that we will be updating the rating guide early next year. Generally speaking, we expect to go deeper and incentivise security quality. I don't know the details yet.
As for CA support for better signature algorithms -- I have never come across such a certificate.