Tuesday, August 30, 2016

Ephemeral DH public key size is less than the required minimum

This error appeared for me after upgrading from 8.3.01 to 9.1.0, and not on appliances that were delivered with the newer 9.0.0 image (and patched to 9.1.0).

After running a TCP dump I found that the ciphers used by the gateway for its outbound connection in the client-hello did not match the default list shown in the routing assertion.

I found that upon changing the cipher list for the routing assertion that the client-hello was then correct and the error ceased to appear.

Support was able to clarify the underlying meaning of the error:
"The error relates to the Diffie-Hellman (DH) keys of sizes less than 1024 bits are deprecated because of their insufficient strength that is being presented to the Gateway.  There was changing JAVA along the way that disabled supported key size. (https://docs.oracle.com/javase/8/docs/technotes/guides/security/enhancements-8.html )

Support stronger ephemeral DH keys in the SunJSSE provider: Make ephemeral DH key match the length of the certificate key during SSL/TLS handshaking in the SunJSSE provider. A new system property, jdk.tls.ephemeralDHKeySize, is defined to customize the ephemeral DH key sizes. The minimum acceptable DH key size is 1024 bits, except for exportable cipher suites or legacy mode (jdk.tls.ephemeralDHKeySize=legacy). See Customizing Size of Ephemeral DH Keys and REF 6956398."

Ultimately the reason that the solution of changing the list worked is because:

  1. The cipher list being used when 'default' is selected is not actually what is indicated
    • My preference is to go to the bottom of the list and move the two SSL ciphers to the bottom of the checked list, this reordering is sufficient to cause the gateway to use what it says.
  2. The list that is actually being used by default has DHE ciphers prioritized over ECDHE (unlike the list indicted within the routing assertion)
    • Because the gateway is the client it controls the negotiation and therefore setting ECDHE ciphers ahead of DHE by minimal alteration to the routing assertions' cipher list may be sufficient to resolve the issue; however, depending on the cipher list available on the backend server you may need to alter the list further.
  3. "Server endpoint is using OLDER versions of SSL libraries (JDK 7, openssl, etc.) than currently supported by CA API Gateway"
    • Which is also why it does not appear for all backend systems, only ones that are not current on their security suites.
Special thanks to CA support for their time and effort in rooting out the cause of this issue.