The most direct method, I think, is to allocate the server's IP on the same subnet as the load balancer. This allows the load balancer to forward traffic to the server instead of terminating IP and routing it. The catch is that the gateway (or server) needs to be smart enough to route the traffic that arrives via the load balancer back out through the load balancer and all other traffic, out bound connections and direct non load balanced traffic, through the network gateway. There are a few ways to achieve this.
1) Setup NAT (Network Address Translation) for every IP that the server needs to communicate with directly, and add a hosts file entry where applicable to resolve SSL hostname mismatch errors as necessary.
2) Setup explicit route entries on the server for targets that it is expected to communicate with directly, in or out bound. One restriction of this approach is that all IPs communicated with must either go through the load balancer or through the network gateway, not both.
3) Terminate IP traffic at the load balancer and inject x-forwarded-for headers. The gateway will automatically interpret these headers as the source IP in policy, this approach will however prevent you from seeing the client certificate of the requesters because SSL is being terminated at the load balancer. But as far as routing goes, the gateway only has to communicate with the network firewall and does not need to be subnet adjacent to the load balancer (not the method I advocate, but frequently used).
4) Dynamically route the traffic out the load balancer based on the incoming IP and default all other traffic through the network gateway. This is the best of both worlds because the gateway sees the source IP of all incoming traffic and SSL is not terminated ahead either, this means it can still see the client certificates as well. Supposedly there is a way to do this using metric and TCP flags, but I find it easier to have two ethernet ports enabled with dynamic routing, which turned out to be much simpler than expected. This is what I describe in detail below.
You'll need to setup two network interfaces for the gateway (this should work for any linux server really). One will be used as the default for all traffic (the outbound routing and direct commutations for management/administration), and the other interface will be for all traffic through the load-balancer.
We will use the following addresses for these objects in this example:
192.168.0.1 network.gateway.firewall
192.168.0.11 loadbalancer.local.interface
192.168.0.100 apigateway.eth0.default
192.168.0.101 apigateway.eth1.loadbalanced
You should setup both of these interfaces through the ssgconfig shell script with the network gateway as the gateway for eth0 and the load balancer's local interface as the gateway for eth1, with eth0 and it's gateway as the default when prompted (and restart for changes to take effect). Then using the following commands from the root shell setup the dynamic routing (these also require another restart to take effect).
echo '252 internal' >> /etc/iproute2/rt_tables
echo 'default via 192.168.0.11 dev eth0 table internal' >> /etc/sysconfig/network-scripts/route-eth0
echo 'from 192.168.0.101 dev eth0 table internal' > /etc/sysconfig/network-scripts/rule-eth1
These will have added a custom route table called 'internal', then setup eth0 on startup to set the 'internal' route table to hold an entry making the system's default gateway to be the load balancer instead of the network gateway when this table is used, and finally eth1 on startup to use a routing rule that says when traffic arrives through eth1 it should use the 'internal' routing table.
Your firewall can block all traffic to the gateway's eth1 since it should only communicate with the load balancer's local interface (which does not have to go through the firewall as it is on the local subnet). Then eth0 can be made accessible to the other gateways in the cluster and wherever your administrators run policy manager from, as well as whatever other system monitoring is in place. The gateway's eth0 end up being your source for outbound firewall rules regardless of destination. The only traffic that will leave the gateway though eth1 will be synchronous responses to incoming traffic from the load balancer (an inbound established connection, not outbound connections).
You can use the following commands to review the rules that are in place.
ip route show
ip route show table internal
