The SourceIP Problem (aka The SSL Termination Dilemma)

For any server it is desirable to know the originating IP address of client traffic, even when the network firewall is relied upon for IP filtering.  It is still indispensable in identifying traffic patterns, usage trends, and threat analysis.

The most direct method, I think, is to allocate the server's IP on the same subnet as the load balancer.  This allows the load balancer to forward traffic to the server instead of terminating IP and routing it.  The catch is that the gateway (or server) needs to be smart enough to route the traffic that arrives via the load balancer back out through the load balancer and all other traffic, out bound connections and direct non load balanced traffic, through the network gateway.  There are a few ways to achieve this.

1)  Setup NAT (Network Address Translation) for every IP that the server needs to communicate with directly, and add a hosts file entry where applicable to resolve SSL hostname mismatch errors as necessary.

2)  Setup explicit route entries on the server for targets that it is expected to communicate with directly, in or out bound.  One restriction of this approach is that all IPs communicated with must either go through the load balancer or through the network gateway, not both.

3) Terminate IP traffic at the load balancer and inject x-forwarded-for headers.  The gateway will automatically interpret these headers as the source IP in policy, this approach will however prevent you from seeing the client certificate of the requesters because SSL is being terminated at the load balancer.  But as far as routing goes, the gateway only has to communicate with the network firewall and does not need to be subnet adjacent to the load balancer (not the method I advocate, but frequently used).

4)  Dynamically route the traffic out the load balancer based on the incoming IP and default all other traffic through the network gateway.  This is the best of both worlds because the gateway sees the source IP of all incoming traffic and SSL is not terminated ahead either, this means it can still see the client certificates as well.  Supposedly there is a way to do this using metric and TCP flags, but I find it easier to have two ethernet ports enabled with dynamic routing, which turned out to be much simpler than expected.  This is what I describe in detail below.

You'll need to setup two network interfaces for the gateway (this should work for any linux server really).  One will be used as the default for all traffic (the outbound routing and direct commutations for management/administration), and the other interface will be for all traffic through the load-balancer.

We will use the following addresses for these objects in this example:
192.168.0.1    network.gateway.firewall
192.168.0.11   loadbalancer.local.interface
192.168.0.100  apigateway.eth0.default
192.168.0.101  apigateway.eth1.loadbalanced

You should setup both of these interfaces through the ssgconfig shell script with the network gateway as the gateway for eth0 and the load balancer's local interface as the gateway for eth1, with eth0 and it's gateway as the default when prompted (and restart for changes to take effect).  Then using the following commands from the root shell setup the dynamic routing (these also require another restart to take effect).

echo '252     internal' >> /etc/iproute2/rt_tables
echo 'default via 192.168.0.11 dev eth0 table internal' >> /etc/sysconfig/network-scripts/route-eth0
echo 'from 192.168.0.101 dev eth0 table internal' > /etc/sysconfig/network-scripts/rule-eth1

These will have added a custom route table called 'internal', then setup eth0 on startup to set the 'internal' route table to hold an entry making the system's default gateway to be the load balancer instead of the network gateway when this table is used, and finally eth1 on startup to use a routing rule that says when traffic arrives through eth1 it should use the 'internal' routing table.

Your firewall can block all traffic to the gateway's eth1 since it should only communicate with the load balancer's local interface (which does not have to go through the firewall as it is on the local subnet).  Then eth0 can be made accessible to the other gateways in the cluster and wherever your administrators run policy manager from, as well as whatever other system monitoring is in place.  The gateway's eth0 end up being your source for outbound firewall rules regardless of destination.  The only traffic that will leave the gateway though eth1 will be synchronous responses to incoming traffic from the load balancer (an inbound established connection, not outbound connections).

You can use the following commands to review the rules that are in place.

ip rule show
ip route show
ip route show table internal

SSL for Beginners, A Primer

Overview

The Secure Socket Layer (SSL) is used to encrypt communications between a client and a server.  There are two flavors of SSL one-way aka server-side and mutual aka bi-lateral.  As you might have guessed, mutual SSL makes use, in part, of server side SSL.

Mechanics

When the client establishes a connection with the server the client determines if the communication will occur over SSL.  For example, HTTP is unsecure and HTTPS is HTTP traffic over an SSL connection (really it could be SSL or TLS, but lets not get into that).

First off the client will call the server to open the connection.  They will negotiate to agree on an encryption algorithm (if they can't find one that they both use then you may see 'negotiation' failures).  Then the server will send it's certificate, which the client will validate in the following ways:

1. The server certificate's subject name (or one of it's subject alternative names) must match the hostname requested by the client or you will get a hostname mismatch error.
2. The client must trust the issuer of the server certificate, normally a third party like Verisign, Entrust, etc.  In some cases clients may be configured to trust the specific certificate of the server they are connecting to or trust the certificate by virtue of it's issuer and also require that it have a specific subject DN.  If the server's certificate does not meet the expectations of the client then the error will be something like 'could not establish trust', which is normally corrected by adding the appropriate third party issuer to the client's trusted root authorities list.
3. If the client is configured to check for certificate revocations then it will reach out to the issuer of the server's certificate to verify that it has not been revoked.
4. Of course the certificate must also not be expired.

The difference between the subject name (Subject CN) and the subject DN is that the CN is the name i.e. server.domain.com and the DN looks more like "CN=server.domain.com,DN=domain,DN=com" which will look familiar if you have worked with LDAP.

At this point your server side SSL connection can move forward, however if you are using mutual authentication then the client certificate is the next step.  Some clients may be configured to not send their certificate until they receive an authentication challenge while some servers expect that the clients send their certificates without being challenged and therefore will not send one, this failure mode can be difficult to recognize.  In the layer 7 environment it is fairly easy to see if a request is coming in without being signed by a client certificate, the problem becomes confirming that request is from the client you are troubleshooting and not from someone else.  The incoming IP address of the request will help you with this, unless you are behind a network architecture that is not providing an x-forwarded-for header.

If you are using mutual SSL then the server should validate the client certificate in a similar manner to the way the client validated the server's certificate, the point of which is to restrict traffic to only be accepted from known clients.  The best way to do this is to have a copy of the client certificate's public key on file for the server to use for validation.  You can use the client's issuer and/or the client's subject name (or even more properties), but even if you use the name and the issuer then another certificate with the same name from the same issuer would be able to get in.  I can understand the temptation because this means that you would not have to make any changes when the client renews their certificate, but it is not as secure as the 'shared secret' of an explicit client certificate.

Anatomy of a Certificate

This is important to understand and I have seen a lot of confusion regarding Public and Private key pairs.  When you create a certificate you generate a key which uses a random number and a password of your choosing.  You should then generate a 'Certificate Signing Request' which you hand off to the issuing authority and they give you back your public certificate (cer, crt, or other extension), you can then merge your key with the certificate to get your private certificate.  The process differs greatly from one issuer (signing authority) to another.

Think of the private key like your drivers license or debit card, when combined with your social or pin (the password) someone else can impersonate you and gain access to sensitive information.  In other words, don't give anyone that private key that should not have it.  Even if you don't give them the password; giving anyone else that key compromises it's security.

The public key on the other hand is much more innocuous.  Its more like your drivers license number, without the card itself there is not much they can do beyond confirming that they are being presented with your card when they see it and it lets them know some information about you (which is what the public key is for), but most importantly they cannot use it to impersonate you.

Both server and client certificates have private and public keys.

• The private server certificate gets installed on the server to host the site or service
• The public server certificate is delivered to clients connecting to the secured URL
• The client certificate private key is used to sign requests outgoing to a secured URL
• The client certificate public key is used by the server to verify the signature of the incoming request from the client

A single certificate can be used for both (as well as other) purposes, but the private key would have to be implemented on a client and a server in order to actually act as both.  When you examine the properties of a certificate you can see what it can be used for in the 'enhanced key usage' (multi-value) property.

Trust Relationships

A third party issuer will have a self-signed root authority.  This means that the certificate does not have an issuer, if you trust it that is because you trust who owns it.  As an issuer they can use it to sign other certificates, such as server, client, and intermediate authorities.  For example the third party trusted authority may issue your company a intermediate authority, and then your company could use that intermediate authority to sign your own server and client certificates.  Then when others look at your certificate public key (because you wouldn't be handing out the private keys) they would see your server certificate's trust chain consists of server.domain.com issued by your.intermediate.authority which is issued by third.party.authority.  Then when they have third.party.authority in their trusted root list they can accept the connection to your server.

In reality if they do not have the third party root in their trusted root authorities list and your intermediate authority in their trusted intermediate authorities list then they may not see your trust chain properly while examining your certificate as that information is populated by the local system unless the certificate is configured to send the entire chain with it.  Also, in order for them to trust a certificate that has an intermediate authority they must either have that authority in their trust lists or trust the root authority and have the certificate delivered with it's complete trust chain.

Certificates issued by the trusted third party are trusted by extension because the third party is expected to have performed due diligence in verifying the identity of those requesting that the third party issue certificates to them.  That is really what you are trusting the third party to do when you add their self-signed issuing root into your trusted root authorities list.  Though if there are intermediate authorities you still need them delivered at some point in order to make that connection between the server or client certificate and the trusted root.