The most direct method, I think, is to allocate the server's IP on the same subnet as the load balancer. This allows the load balancer to forward traffic to the server instead of terminating IP and routing it. The catch is that the gateway (or server) needs to be smart enough to route the traffic that arrives via the load balancer back out through the load balancer and all other traffic, out bound connections and direct non load balanced traffic, through the network gateway. There are a few ways to achieve this.
1) Setup NAT (Network Address Translation) for every IP that the server needs to communicate with directly, and add a hosts file entry where applicable to resolve SSL hostname mismatch errors as necessary.
2) Setup explicit route entries on the server for targets that it is expected to communicate with directly, in or out bound. One restriction of this approach is that all IPs communicated with must either go through the load balancer or through the network gateway, not both.
3) Terminate IP traffic at the load balancer and inject x-forwarded-for headers. The gateway will automatically interpret these headers as the source IP in policy, this approach will however prevent you from seeing the client certificate of the requesters because SSL is being terminated at the load balancer. But as far as routing goes, the gateway only has to communicate with the network firewall and does not need to be subnet adjacent to the load balancer (not the method I advocate, but frequently used).
4) Dynamically route the traffic out the load balancer based on the incoming IP and default all other traffic through the network gateway. This is the best of both worlds because the gateway sees the source IP of all incoming traffic and SSL is not terminated ahead either, this means it can still see the client certificates as well. Supposedly there is a way to do this using metric and TCP flags, but I find it easier to have two ethernet ports enabled with dynamic routing, which turned out to be much simpler than expected. This is what I describe in detail below.
You'll need to setup two network interfaces for the gateway (this should work for any linux server really). One will be used as the default for all traffic (the outbound routing and direct commutations for management/administration), and the other interface will be for all traffic through the load-balancer.
We will use the following addresses for these objects in this example:
192.168.0.1 network.gateway.firewall
192.168.0.11 loadbalancer.local.interface
192.168.0.100 apigateway.eth0.default
192.168.0.101 apigateway.eth1.loadbalanced
You should setup both of these interfaces through the ssgconfig shell script with the network gateway as the gateway for eth0 and the load balancer's local interface as the gateway for eth1, with eth0 and it's gateway as the default when prompted (and restart for changes to take effect). Then using the following commands from the root shell setup the dynamic routing (these also require another restart to take effect).
echo '252 internal' >> /etc/iproute2/rt_tables
echo 'default via 192.168.0.11 dev eth0 table internal' >> /etc/sysconfig/network-scripts/route-eth0
echo 'from 192.168.0.101 dev eth0 table internal' > /etc/sysconfig/network-scripts/rule-eth1
These will have added a custom route table called 'internal', then setup eth0 on startup to set the 'internal' route table to hold an entry making the system's default gateway to be the load balancer instead of the network gateway when this table is used, and finally eth1 on startup to use a routing rule that says when traffic arrives through eth1 it should use the 'internal' routing table.
Your firewall can block all traffic to the gateway's eth1 since it should only communicate with the load balancer's local interface (which does not have to go through the firewall as it is on the local subnet). Then eth0 can be made accessible to the other gateways in the cluster and wherever your administrators run policy manager from, as well as whatever other system monitoring is in place. The gateway's eth0 end up being your source for outbound firewall rules regardless of destination. The only traffic that will leave the gateway though eth1 will be synchronous responses to incoming traffic from the load balancer (an inbound established connection, not outbound connections).
You can use the following commands to review the rules that are in place.
ip route show
ip route show table internal
It has come to my attention that the new version of the gateway (v10+) on CentOS (as opposed to older RHEL) uses Network Manager over iproute2 and thus setting up persistent routes requires different commands.
ReplyDeleteYou will still have to add the route table to rt_tables file:
echo '252 management' >> /etc/iproute2/rt_tables
Then add default route to that table for the secondary device:
nmcli connection modify ssg_eth1 +ipv4.routes "192.168.0.11 table=252"
Lastly, add the routing rule to use the route table when traffic arrives on the device:
nmcli conneciton modify ssg_eth1 +ipv4.routing-rules "proprity 32765 from "192.168.0.11 table=252"
(I renamed my secondary device "ssg_eth1" via /etc/sysconfig/network-scripts/ifcfg-ens### per Broadcom documenation referenced below)
Alternatively... you could set "net.ipv4.conf.all.rp_filter = 2" in /etc/sysctl.conf, which tells your network manager to ignore all your routing tables and simply return responses back out the same path they arrived on, which sounds great, however I would be remiss if I did not point out that this will cause your server to respond to traffic out along routes that may not be desirable in the event that traffic was able to reach the server without going through your default network gateway and firewall.
Ref:
https://techdocs.broadcom.com/us/en/ca-enterprise-software/layer7-api-management/api-gateway/10-1/reference/network-deployment-guide.html
https://developer-old.gnome.org/NetworkManager/unstable/nmcli.html
oops, typos
Deletenmcli connection modify ssg_eth1 +ipv4.routing-rules "priority 32765 from "192.168.0.11 table=252"
... and now version 11 on Debian:
ReplyDeleteFirst be sure to start with https://techdocs.broadcom.com/us/en/ca-enterprise-software/layer7-api-management/api-gateway/11-0/reference/network-deployment-guide.html.
Edit /etc/iftab to add the other NIC with it's MAC address
nmcli con modify "Wired connection #" connection.interface-name "ssg_eth1"
(optional) nmcli con modify "Wired connection #" connection.id "ssg_eth1"
And then: mv /etc/NetworkManager/system-connections/Wired\ connection\ 1.nmconnection /etc/NetworkManager/system-connections/ssg_eth1.nmconnection
This last step is not in the Layer7 docs, but since we've renamed the object the file doesn't match and needs to be updated.
Reboot, then use the ssgconfig menu to add and configure the interface.
I've done this where both NICs are on the same subnet and therefore have the same network gateway, if they have a different network gateway there will likely be more steps to setup correct routing. If I have to do that on one then I will add another note here with those added steps.