The 128T GRE plugin can be used for creating IPv4 GRE tunnels between a 128T router and a remote GRE tunnel destination. For services such as Zscaler, this provides better throughput compared to other tunneling mechanisms.
The instructions for installing and managing the plugin can be found here.
The GRE plugin can be configured on an individual router. In order to configure the tunnel, it is important to collect the following information up-front:
- Remote tunnel information
- Ingress traffic to be sent through the tunnel
- WAN interfaces to be used for sending the tunnel traffic
Remote GRE tunnel information
First and foremost, it is important to identify the address and other probe parameter for the remote GRE endpoints.
In the above example, you will notice there are two tunnels configured on the router. Each
destination represents a single tunnel interface and allows the user to configure an IPv4 address for the destination. The
enabled-node configuration allows the user to control which 128T router node will be operating which tunnel. In addition the
enabled-node > tenant can be used to apply a specific tenant for the GRE tunnel traffic. For each destination on each enabled-node, the 128T router creates a unique KNI interface and the configured tenant is applied to this interface.
Tunnel ICMP health check parameters
The GRE tunnels do not have an inherent mechanism to detect the availability of remote peers. As a result, the GRE plugin allows the user to configure ICMP probes to the destination. The configuration is enabled by default with the following settings:
The time interval for the attributes are in seconds.
link-test-interval an icmp check is performed to determine the availability of the remote tunnel peer. For an unresponsive peer, a total of
number-of-retries + 1 icmp ping attempts will be made within the
retry-interval. If the peer does not respond to any of these ping attempts, then its considered as down. In the above config, assuming an unresponsive peer, first ping is sent at 10 seconds, followed by 5 more pings at 1 second interval each. In total taking the system about 15 seconds and 6 pings to detect a peer as down. Once a peer is considered down, the next attempt to detect the tunnel liveliness is made after 10 seconds (or the
In the above example, the two tunnels
sec-tunnel create two additional KNI interfaces called
gre-1 respectively. When a tunnel is determined to be non-responsive, the corresponding
gre-x interface is brought down. For example, in the above config, when the
pri-tunnel goes down, the corresponding
gre-0 interface is brought down as well. This allows for traffic to fail over to a secondary tunnel if available. More details on this will be explained later in the document.
ICMP health check to private address
Some GRE tunnel providers require the endpoint to ping an internal private address to detect the tunnel liveliness. The
icmp-keep-alive > address-type can be used to configure a private address for keep-alive detection. Consider the following example:
In the above configuration, the
address-type > custom is used to set a private icmp-address of
192.168.10.13. In doing so, the icmp-health check algorithm described above will be run on the private address of
192.168.10.13 instead of the default
destination > host. The behavior in terms of declaring the tunnel as down and continuous monitoring remains the same.
When using a private ICMP address, its important to also use an in-subnet address for the generated KNIs. This can be accomplished by configuring the appropriate
plugin-network as illustrated in the example above.
128T services to transport over the tunnel
Next step is to identify the the prefix or the subnet to be transported over the tunnel. In some cases, it might be desirable to transport all internet traffic through the tunnel, so the prefix could be as simple as 0.0.0.0/0. This can be done by capturing the prefix in a 128T service and setting the next-hop as the
gre-x interfaces. As noted in the previous section, each destination on a given node corresponds to a
gre-x inteface. By configuring the next-hop as the appropriate GRE interfaces, it allows the incoming traffic to be service-function chained to a GRE tunnel towards a WAN interface.
In the example above, all the
lan tenant traffic in the
192.168.10.0/24 subnet will be sent to the
gre-1 network interfaces. These
gre-x-intf are auto generated by the conductor and correspond to the configured destination. In the above config, when the
gre-0 interface will be used as primary target for lan subnet while that tunnel is up. If the
pri-tunnel goes down, all new sessions will automatically be routed to the
sec-tunnel via the
WAN interfaces for sending the tunnel packets
Another piece of configuration that is auto-generated is the service corresponding to the two configured tunnels. In the above example, the two tunnels
sec-tunnel will trigger two auto-generated services, one for each of the destination. The generated service will look something like this:
The next step, is to configure the service-routes or other routing configuration for these generated services. Typically, such routes are directed towards the WAN interface and user has full control over how & where this traffic can be routed.
Static Source NAT considerations
This section can be skipped for WAN interface types of PPPoE and LTE
Please note that the
next-hop is making use of a
gre-dpdk2-nat-pool for example. This nat-pool is necessary for performing a source nat of the GRE tunnel traffic depending on which egress interface are being used.
network-interface > source-nat flag does not support GRE, hence the
shared-nat-pool is required.
shared-nat-pool configuration is as follows:
The TCP MSS (maximum segment size) is the amount of data that the interface is willing to accept in a single TCP segment. This option is negotiated by the client and server based on their local environments. However, tunneling adds extra overhead in terms of packet bytes so its important to be able to adjust the MSS (maximum segment-size) for TCP connections by the routers. By default, the
enforced-mss is set to be
path-mtu which allows us to automatically adjust the MSS based on the MTU of the tunnel interface. In addition, user can override the value to a static number as shown in the example below.
Debugging & Troubleshooting
When the plugin is installed on the conductor, each commit triggers two scripts called
generate_pillar to auto-generate KNI, services etc and to generate pillar data for each router. Please check the following locations for debugging information.
- Logs for the config and pillar generation for each commit can be found here
GRE Tunnel not working on the router
When the config and pillar data are successfully generated, a
t128-setup-gre RPM is installed on the router itself. As part of this process, a script called
handle_gre_config is executed which will create all the necessary config etc on the running system.
- Logs for the config generation on the router can be found here
- Debugging the runtime status of the GRE tunnels can be done by monitoring the journal for the following tags
- For debugging the linux network namespace, here are some of the common commands along with the relevant output of how a healthy system would look like
ip netns pri-tunnel ip addr
ip netns pri-tunnel ip route show table all
ip netns pri-tunnel ip rule list
ip netns exec pri-tunnel iptables -nvL
The status of the tunnel and other data is available via the auto generated tunnel interfaces. Here's an example:
In addition, a
ping-monitor service is started for each configured tunnel, the
systemctl status ping-monitor-namespace@<tunnel-name> can be used to query the status of the ping service.
Release 1.1.2, 2.1.2
PLUGIN-677 GRE plugin doesn't start up correctly post reboot
Resolution Implemented a config watcher service to handle startup conditions and dynamically apply configuration changes at runtime.
Release 1.1.3, 2.1.3
PLUGIN-799 Pillar data for GRE tunnel is false by default
Resolution Set the enabled key to true by default when generating the pillar data
PLUGIN-479 Address range checking not valid for GRE tunnel
Resolution Use non-strict mode when getting plugin network in config generation