128T 4.1 Release Notes

Release 4.1.10

Issues Fixed

I95-30610 RTP is not properly classified for subsequent 128T routers

I95-33279 Path MTU discovery unresolvable when no ICMP is generated

I95-34058 Session setup fails for paths configured as outbound-only when first packet of a flow exceeds MTU (typically UDP)

I95-34068 SVR sessions fail to establish due to waypoint allocation failures after HA node failover.

Symptom: The following warning log is generated:
```
Mar 03 09:25:10.813 [HWMC| – ] WARN (icmpManager ) Base Exception: failed to allocate ports for WayPoint; intf=5.0; local=192.0.2.100; remote=198.51.100.128
```
Until the system is upgraded to 4.1.10, this issue can be mitigated by removing the corresponding adjacency configuration and adding it back.

I95-34090 A network-interface configured with multiple neighborhoods, where one of the neighborhoods defines a port range, will result in traffic being dropped on the defined range

Symptoms: SVR traffic is dropped when destined for a port range configured on its peer 128T router's neighborhood

I95-34158 Modifying a port-range configuration value to include an overlapping range results in a list with an incorrect range.

Conditions: An existing range list is modified to include an overlapping range

Until the system is upgraded to 4.1.10, this issue can be mitigated by entering a unique range of values that do not overlap.

I95-34310 Secure fields from the 128T configuration are in the commit audit events from config diff operations.

I95-34744 highway process can fault when a DHCP server assigns the IP address 0.0.0.0 to the 128T router

Release 4.1.9

Issues Fixed

I95-35138 A vulnerability in the SaltStack code allows for unauthenticated salt-minions to execute any script on the salt-master.
info
This fix is required only on the 128T Conductor.

Release 4.1.8

note

The 4.1.8 release is a superset of the 4.1.7 release. Features and corrections in the 4.1.7 release are not provided in these release notes. Please refer to the 4.1.7 release notes for further information.

Issues Fixed

I95-18857 Support for automatic loopback has been added to Sangoma T1 devices

I95-33536 Fixed highway terminate condition on shutdown with large number of peer paths

Symptom: highwayManager process aborts on shutdown or restart

Conditions: 128T router with greater than 2500 active peer paths restarted with systemctl restart 128T

Corrective Action: None required, system will automatically recover.

I95-33485 Upgrading a HA node of a 128T router can result in traffic being dropped

Symptom: Existing traffic for some services is blackholed during upgrade

Corrective Action: Complete upgrade of both nodes or restart non-upgraded 128T

I95-33323 highwayManager process can fault on processing DNS responses for services with addresses defined as FQDN

Symptom: Traffic stops forwarding while process restarts

Corrective Action: This condition is rare and is exacerbated by DNS responses that change for the same request. Typically the order of the A records have changed for load balancing purposes. This can be mitigated by ensuring the DNS responses are consistent, or removing the FQDN from the service configuration.

I95-33296 Removing a redundant device-interface and its corresponding redundancy-group as part of the same commit will cause the commit operation to fail.

Symptom: Unable to commit configuration changes

Corrective Action: Perform two commit operations. The first commit must be to remove the redundancy-group.

I95-32843 System can fault when routing loop is created with OSPF and BGP

Symptom: highwayManager process faults after configuration is loaded.

Corrective Action: Restore existing configuration to remove routing loop created by OSPF.

Release 4.1.7

note

The 4.1.7 release is a superset of the 4.1.6 release. Features and corrections in the 4.1.6 release are not provided in these release notes. Please refer to the 4.1.6 release notes for further information.

Issues Fixed

I95-31170 NodeMonitor Application fault on shutdown.

I95-32449,I95-25567,I95-31060,I95-31675 failed to reserve ports for WayPoint resulting in loss of traffic

Release 4.1.6

note

The 4.1.6 release is a superset of the 4.1.5 release. Features and corrections in the 4.1.5 release are not provided in these release notes. Please refer to the 4.1.5 release notes for further information.

Issues Fixed

I95-21979 An ethernet interface with DHCP enabled cannot be created from GUI. Validation does not pass and throws the error: "There must be at least one address configured when type is not pppoe and dhcp is disabled".

I95-25553 Disabling and re-enabling a service is not real time configurable and required a restart of the 128T service.

I95-25984, I95-32181 Power outage may result in a zero byte global.init file, rendering 128T unable to start

I95-26634 BGP routes are not updated when VLANed interface is operationally down

I95-28103 After a restart of the highway process, any interfaces that were managed by 128T that were not gracefully restored, now provide detailed logging information regarding their state and corresponding transition.

I95-28535 Unable to obtain DHCP lease after redundant interface state transitions back to operationally up

I95-29115 service-route / max-sessions is not real-time configurable and requires a restart of the 128T service

I95-29741 Salt connection between Router and Conductor may not reestablish on a connection failure

I95-29811 Session created with a non-standard protocol (e.g. security audits performing "fuzzing") can cause flows to never be deleted and exhaust the flow table

I95-29812 Service ping does not work with peer-connectivity set to outbound-only between 128T routers

I95-29821 Packet fragmentation for SVR paths is larger than configured MTU by the L4 packet header size

I95-29990 When a KNI interface starts as operationally down, the state remains the default of unknown and never transition to down.

I95-30002 Service route generation skipped for generation set to true if another service with the same name is set to generation false

I95-30078, I95-30268 Traffic does not switch to standby interface on management path communication failure

I95-30315 DHCP Server fails to start after system power failure and power recovery

I95-30327 On peer path failure, peer path may remain down if going through a nat to get to the remote peer

I95-30334 Shared interfaces can operate with both interfaces on each HA nodes as active (each acting in standalone mode)

I95-30354, I95-30687 admin user or 128t-admin group missing after upgrade

I95-30383 Connection error to node may result in configuration not being applied

I95-30401 Asset remains in pending state on initialization if the connection is terminated during pending state operations

I95-30448 Application module retry interval takes an extended time after a failure. With this correction, if a failure is encountered the retries will start at 60 minutes and increase exponentially for each subsequence failure after the second failure.

I95-30734 "worker-core packet-processing-utilization" would show 100% due to improper handling of unidirectional flows when packet-retransmission feature is enabled

I95-30742 Incorrect packet fragmentation when first packet is a jumbo packet

I95-30781 Warning not provided when neighborhood does not have an end port range

I95-30833 BGP over SVR neighbor not connecting due to missing route

I95-30934 Services with matching tenants and protocols with port ranges ending in 65535 may result in services failing.

I95-31005 128T does not safely handle the case where a DHCP server provides a client lease with an address of 0.0.0.0

I95-31200 BGP timer configuration is not dynamically configurable

I95-31208 continuous configuration updates do to network anomalies lead to increased memory usage

I95-31232 Highway application may fault during network port scanning of the router

I95-31244 When a pinhole session is restored upon failover, if the routing table is not up-to-date, the packet will incorrectly be routed to the same interface from which it came from.

I95-31251 Peer intermittently does not recover after system restart

I95-31255 Intermittently managed routers may send a "No connectivity" alarm to the Conductor

I95-31333 highwayManager application fault during multiple interface state transitions

I95-31515 highwayManager application faults if no paths are available for session duplication

I95-31570 Changing the order of custom charts can sometimes render the dashboard page inoperable

I95-31742 routingManager application can fault when a device interface is removed

I95-32065 NAPT translation does not work with for a service-route with a next-hop

I95-32257 Router behind a NAT not peering after upgrade

Release 4.1.5

Product Enhancements

Overlapping IP address support for adjacencies

Prior versions of the 128T software did not support two routers with the same WAN IP address communicating with common infrastructure. The most common manifestation of this issue is when two routers both share the same private IP address (e.g., 192.168.1.100) and are peering with the same head end router. In this scenario, the head end router cannot disambiguate between the two devices, which may lead to routing issues (e.g., traffic intended for one location may be sent to another location with the same address). It is for this reason that large-scale projects were discouraged/prevented from using private IP addresses, due to the high likelihood of IP collision.

Issues Fixed

I95-30078 - HA node communication failure results in two systems both taking control of a shared (redundant) interface

Symptom: Traffic egressing a highly available device may get pinned to the wrong node in a highly available pair.

Mitigation (pre-4.1.5): Manually purge specific traffic flows that are pinned to the wrong node, to allow them to regenerate.

I95-30387 - ‘path MTU is unresolvable' alarm after 128T restart or BFD link erroneously shows 100% packet loss when device is behind a NAT

Symptom: when two 128T devices communicate with one another and there is an intervening NAT device (as is common in LTE networks), the path will show 100% packet loss for various NAT types.

Mitigation (pre-4.1.5): this problem was introduced in 4.1.5-1 and fixed in 4.1.5-2.

I95-25787 Overlapping IP address support for adjacencies (See Special Considerations)

I95-27406 Ctrl-c does not terminate PCLI tab completion

I95-28337 When two HA systems have a temporary disconnection between one another, it may prevent administrative (PCLI) logins.

I95-29135 A 128T node may permanently lose "asset" connectivity to Conductor in the wake of a connection failure. The asset connectivity affects the Conductor's ability to remotely start/stop/upgrade a 128T router, but does not impact other administrative functions.

I95-29136 The PCLI now warns users logging in if their login exceeds the recommended number (4).

I95-29173 Tab complete produced unexpected output at various places within the PCLI.

I95-29188 Performance improvements for distributing large configurations from Conductor to routers

I95-29206 Routers establish connections to the Conductor in a way that avoids "thundering herd" problems

I95-29217 Inaccurate "There are changes waiting to be validated. Click the validate button to refresh the list." message in the configuration section of the conductor UI

I95-29238 Entering new configuration with non-interactive commands result in configuration not being applied.

I95-29252 Reconfiguring VLANs on interfaces would occasionally require a restart of 128T software.

I95-29278 Revertible path failover can sometimes not revert a session. This would cause a device to continue to use a secondary path in the event that a primary path fails, traffic is migrated to the secondary path, but later the primary path is restored.

I95-29282, I95-27182 Restart of the standby conductor node may impact administrative access to active Conductor node for up to 10 minutes.

I95-29283 The system's ARP table size now corresponds to the number of peering routers. This prevents ARP table overflow issues that may cause outages as nodes can no longer communicate with one another.

I95-29289 Certain dual failure conditions could result in a redundant interface being inactive on both nodes in a highly available pair. This could occur, for example, if one node was rebooted and its counterpart had a software failure (crash) before the first node was able to resynchronize completely. This would previously require a reboot to restore.

I95-29364 Node not transitioning from starting state on initial deployment. This affects the time to turn up a new system due to an extra restart being required.

I95-29367 Application can fault when administratively disabling a T1 Interface, requiring a software restart.

I95-29377 Application can fault when disabling an interface that is configured for DHCP, requiring a software restart.

I95-29430 Fixed a cosmetic issue when a progress indicator remained during a configuration commit action that resulted in validation errors.

I95-29688 An internal software communication failure resulted in the PCLI command 'show system' showing partial data.

I95-29841 Two services (routes) with the same IP address, where one has a security policy assigned and another does not, may cause an application fault when traffic hits that security policy. This would result in full traffic loss until the application restarted (approx. 30 seconds).

I95-29744 Restart of a node may result in failed sessions, requiring rebooting the counterpart node to restore service. This has only been observed during upgrade scenarios.

I95-30857 BGP routes not being updated after a large number of peer path state changes in conjunction with a configuration change. This may result in the 128T from advertising future route updates.

I95-30845 FIB entries become invalid after race condition with simultaneous configuration change and routing updates, resulting in routing failure.

I95-30598 Automated Provisioner issues continuous salt minion restarts for a node in a disconnected state, resulting in unnecessary reattempts.

I95-30424 With BGP, there exists a race condition on startup that results in FIB entries missing some routes, resulting in routing failure.

I95-30143 High state is applied to nodes when configuration has changed, causing unnecessary Conductor to router communication. This has been changed to only apply high state when necessary. (Note: when a node is disconnected and reconnects high state is always applied)

I95-30011 SVR packets dropped briefly when standby node in an HA router pair is restarted.

Release 4.1.4

Issues Fixed

I95-15146 Large NETCONF client requests can fail to be processed

I95-22948 show ospf shows incorrect data when OSPF is not configured

I95-24724 Jumbo frames are dropped on a KNI interface

I95-24811 Maximum network-interface mtu has been reduced to 9198 to avoid fragmentation issues

I95-25863 Requesting metrics beyond what is available returns empty data

I95-25964 Removed access-policy configuration for dhcp-server

I95-26059 Changing a device-interface name will stop pcap capture

I95-26079 A system fault can occur if a commit operation is cancelled

I95-26089 show network-interface can sometimes output no data

I95-26199 SVR port range is restricted to 1025-16383. This has been increased to 1025-65535

I95-26237 Custom charts for device-interfaces can have incorrect session count metrics

I95-26288 A ping flood can stop future ICMP processing

I95-26517 Unable to provision conductor service on a device-interface type of T1

I95-26601 MTU configuration on a bridged KNI interface is not dynamically reconfigurable

I95-27044 System fault can occur when changing encryption keys

I95-27170 Added numerous tranmission stats for DHCP clients

I95-27242 Attempting to upgrade a managed router to a version greater than the Conductor fails silently

I95-27281 Session migration does not work when a session is initiated from a router with peer-connectivity=bidirectional to a router with peer-connectivity=outbound-only

I95-27328 Unable to execute yum commands on nodes after ztp

I95-27339 Sessions for a DHCP relay service can linger, causing subsequent DHCP request failures

I95-27371 BGP route reflector can not be disabled by setting client false

I95-27487 Invalid config produces the message % Error: Config does not exist in Zookeeper

I95-27521 forwarding-core-count is not honored in configuration

I95-27536 LTE interfaces that are unable to connect to the carrier's network can cause system instability

I95-27540, I95-27943 An erroneous message of "download timed out, try again" very shortly after downloading a software upgrade, even though no error occurred

I95-27600 T1 interface shows up even if the w1g1ppp is down

I95-27601 show assets does not show configured asset after changing its asset id

I95-27614 When operating in promiscuous mode, the 128T can send ICMP unreachables for dest macs that are not its own

I95-27635 Configuring BGP local-as under neighbor causes BGP configuration to not work properly

I95-27636 Bandwidth of UDP traffic is displayed incorrectly when traffic shaping is enabled

I95-27673 GUI may display There may be changes waiting to be validated… after validate with no errors

I95-27683 Unable to shutdown BGP peer

I95-27708 System can fault when generating ntpd.conf

I95-27728 sssd restarts when changing user system preferences

I95-27730 Node reboots after 128T upgrade when reboot is unnecessary

I95-27733 128status.sh script has been added to provide detailed 128T system health information

I95-27752 Session duplication only works with security-policy hmac-mode regular

I95-27763 BGP neighbor local-as is ignored when set to neighbor remote-as

I95-27830 System can fault when multiple "first" packets are processed simultaneously

I95-27878 Database process can consume large amount of CPU due to internal debugging data. This database has been removed

I95-27889 System can fault on shutdown

I95-27899 Router is stuck in the state of "Key Accepted" during assets registration

I95-27922 System can fault when flow is selectively deleted from ingress router between two 128T peers with ongoing traffic

I95-27925 Sparklines do not show numerical value when time window is greater than 5 seconds

I95-27928 Terminal emulators that do not support 256 colors will produce the error % Error: Unhandled AttributeError: 'NoneType' object has no attribute 'text'

I95-27937 System can fault when executing save tech-support-info on high-end server platforms with greater than 16 cores

I95-27940 Asset software version is still displayed after disconnect

I95-27953 bgpd process can fault in execution of show ip bgp neighbor <neighbor-addr> received-routes when communities match and are configured with a strip policy

I95-27959 Unable to initiate install of software after download through the asset page

I95-27979 Provisioning a service-route through the GUI allows entry of a service name. The service name should only provide options from a drop-down of provisioned services.

I95-27986 Unable to provision service-route next-hop through GUI

I95-28031 Unable to disable generated flag on DHCP relay service

I95-28053 Inconsistency between units in spark lines and value on graphs

I95-28056 Event History page can result in poor system-wide network performance in large deployments. The Event History page now requires a selection of a single router from which to obtain event history

I95-28060, I95-28380 System can fault when a configuration is applied with active sessions that contain overlapping private IPs

I95-28161 GUI displays no service-routes for services that have service-routes

I95-28162 Swagger document has wrong placeholders for Clone functions

I95-28191 Session optimization feature can cause 100% CPU utilization with low bandwidth

I95-28208 ISIS packets generate an exception in highwayManager.log files

I95-28219 Non-admin users are able to edit and view other users within the GUI

I95-28220 show assets now displays if an asset never connected

I95-28245, I95-28311, I95-28312 Control buffer pool can be exhausted for routers with a large number of peers

I95-28271 systemServicesCoordinator process can fault when processing IPC

I95-28347 Router upgrade can restart after rebooting a Conductor node before the upgrade completes

I95-28351 Invoking lshw can cause the system to fault

I95-28380 Service validation check not handling implicit /32 address resulting in an application fault

I95-28389 Malformed or missing user data store will prevent the 128T from starting

I95-28412 System can reinitialize multiple times during initial ZTP process

I95-28446 Sessions can be dropped during HA node failover

I95-28448 High CPU utilization for prolonged periods of time can induce a split-brain situation

I95-28480 Incorrect directory permissions can cause the Conductor-hosted software repository to fault

I95-28498 Redirecting output from PCLI show commands to a file produces multiple instances of Redirecting output... in the file

I95-28510 Patch available icon is shown on assets that are unmanaged

I95-28528 An empty option exists in Peer Authority and Router dropdowns

I95-28592 GraphQL API has been added for system connectivity status

I95-28673 ping command did not display round trip time

I95-28691 Large configurations can cause timeouts during processing

I95-28694 Import/Export page can fail to load

I95-28730 System can fault on receipt of BGP packet to an interface that does not exist

I95-28748 When configured for LDAP, users may not be able to login

I95-28827 Malformed environment configuration will cause 128T to fail to start

I95-28863 PCLI may fail to load resulting in message % Error: Unhandled UnboundLocalError: local variable 'model' referenced before assignment

I95-29032 In rare cases, the routingManager process can fault processing configuration

I95-29042 AWS ENA driver will fault if interface is disabled and then enabled.

Release 4.1.3

note

The 4.1.3 release requires the 128T-installer 2.3.0 or greater. By default, the latest installer will automatically be used.

Issues Fixed

I95-27792 Asset status not updated correctly

I95-27339 DHCP stuck flows

Release 4.1.2

note

The 4.1.2 release requires the 128T-installer 2.3.0 or greater. By default, the latest installer will automatically be used.

Issues Fixed

I95-22743 Installation now enables Kernel crash capturing

I95-24835 Creation of new security-policy generates error upon commit

I95-24963 show event alarm rows all reports the error Failed to retrieve alarm events

I95-25744 Tab-ing in GUI dialog boxes does not toggle between buttons or fields

I95-25790 Peer path statistics on PCLI may be missing for routers with multiple paths

I95-26021 WEB/GUI page scrolling stops working after router AP upgrade

I95-26154 System can fault when executing save tech-support-info

I95-26293 Changing the router name will prevent future logins

I95-26361 Asset reports downloading software after download has completed

I95-26447, I95-26509 Conductor software proxy is configured when conductor-only is not set

I95-26492 Duplicate addresses returned from a DNS request are not deduplicated

I95-26620 BGP routes not populated in the RIB when the eBGP neighbor's ttl is set to 1

I95-26690 Stats may not be displayed until 128T is restarted

I95-26696 Connectivity alarms are incorrectly re-evaluated on configuration changes resulting in retransmitting the alarm

I95-26706 Empty global ID for non-redundant interface produces validation warning

I95-26732 Router-based services are incorrectly shared with other neighborhoods

I95-26797 saltmaster.log and logrotate.d message filling root mail

I95-26845 Waypoint Exhaustion Post Change

I95-26910, I95-26911, I95-26912, I95-26927, I95-26972, I95-27046, I95-27067, I95-27076, I95-27125, I95-27181, I95-27184, I95-27248, I95-27357, I95-27404, I95-27421 Validation can take a long time on large configurations

I95-26943 Custom charts displays "all" series, when it should say "selected router"

I95-26956 Piping show commands through grep on large topologies never completes

I95-26981 Alarm shelving is not honored after 128T restarts

I95-26982, I95-27057, I95-27063, I95-27215, I95-27427 PCLI can have long response times with large configurations

I95-26985 128T can get hung on shutting down when removing kni kernel module

I95-27023 PCLI can get stuck in Starting the PCLI…

I95-27029 show asset software incorrectly prompts for confirmation

I95-27032 HA indicator overlaid on top of router selector in custom charts

I95-27035, I95-27382, I95-27415, I95-27430 Web server may have poor responsiveness with large configurations

I95-27051 Congestion control algorithm for session optimization has been reverted to no fairness

I95-27066 Validatate operations can be executed multiple times for the same configuration

I95-27084 Topology map zooms in then back out after selecting a router

I95-27102 Large configurations increase the load time of the topology page A loading indicator was added to provide a better user experience

I95-27105 128T not restarted after reinitialization

I95-27107 NTP service does not start before 128T which can cause HA synchronization issues if time is incorrect between nodes

I95-27120 PCLI does not honor log level changes

I95-27121 128T can crash when flow capacity has been exceeded

I95-27136, I95-27216 Terminated PCLI sessions can still linger preventing future logins

I95-27137 Configurations that contain a large number of hostnames/FQDNs can cause commit to take a long time

I95-27153 Large numbers of shelving alarms changes result in CPU saturation spikes

I95-27173 Requesting both IPv4 and IPv6 DHCP addresses for LTE does not work for some carriers

I95-27209, I95-27560 Large configurations can take a long time to commit

I95-27235 Subsequent requests to refreshing "Available Versions" do nothing

I95-27241 PCLI can hang on quit after committing configuration

I95-27246 Changing system clock to an earlier time will cause the 128T to not produce any metrics

I95-27248 Large configurations can take a long time to retrieve on both the PCLI and GUI

I95-27250 Configurations that have a large number of peer paths on a Router can cause HA to fail on those nodes. This is a result of running out of memory within one of the processes. This setting is configurable to fit the needs of the deployment

I95-27282, I95-27369 Large configurations can cause delays in GraphQL responsiveness

I95-27285 Large configurations can cause save tech-support-info to take a long time to complete

I95-27300 Unable to properly allocate forwarding core count to leverage multi-socket systems

I95-27310 It is possible to oversubscribe the system by allocating more forwarding cores than exist on the platform

I95-27401, I95-27511 Unclean shutdowns can cause the analytics database to become corrupt

I95-27444 Flow audit events are created even if auditing is disabled

I95-27522 highwayManager.log file is unnecessarily overflowing with the message:

[HWMC|AEM ] ERROR (ActionEventManag) The High queue has overflowed. Dropping message

I95-27524 128status.sh script has been added for troubleshooting

I95-27604 FIB Next Hops is <none> when there are only service-routes with services with application-name

I95-27614 The 128T would send ICMP unreachable responses for destination MACs that were not its own

I95-27799 Per-router services fail validation when a service applies to more than one router-group

I95-27830 2 or more ICMP packets within milliseconds resulted in a software fault

Release 4.1.1

Issues Fixed

I95-21219 In large deployments, displaying all peer paths on the topology page can cause poor performance. Peer path lines for a given router are only drawn when selecting a router.

I95-23631 "All Routers" option on custom dashboards can cause poor performance for large deployments

I95-23634 Large configurations can cause the web server performance to degrade

I95-25297 auditd and plugin files are unnecessarily rotated

I95-25367 The 128T does not respond to ICMP packets destined for its interfaces

I95-25897 Installing from an ISO can cause an incorrect core count allocation

I95-26021 Window scroll bar become inoperable requiring a complete browser page reload

I95-26247 Configuring a prefix-list in a BGP neighbor can cause a system fault

I95-26266 When an interface configured for DHCP changes state to operationally down, the system can lock up

I95-26284 DHCP packets being relayed through 128T can get dropped

I95-26328 Login failure when LDAP username and home directory do not match

I95-26385 Added audit events for global.init and local.init

I95-26392 Upgrading an asset through the Conductor was restricted to only use 2.2.x versions of the installer

I95-26397 Cannot access dashboard tabs when exceeding screen limit

I95-26428 GUI fails to commit cloned router config, producing error: "reverse SSH port must be unique within an authority"

I95-26457 System can fault on shutdown when unbinding interfaces

I95-26467 Setting DHCP option interface-mtu causes DHCP server to fail to start

I95-26504 Transmitting large ICMP packets over SVR can cause the system to fault

I95-26582 SSH keep-alives between conductor and router are happening too frequently

I95-26683 Factory defaults can be inadvertently applied when bringing up a HA node

I95-26721 kni processes may linger upon shutdown causing security keys to not install properly

I95-26765 There exists no way to clear DHCP client leases. The command dhcp-agent release-current-dhcp-lease has been added

I95-26788 The system can fault when unprovisioning a redundant interface

I95-26799 Errors generated by assets are erroneously duplicated

I95-26802 Upgrading or downloading software upgrades on all routers can impair performance of the Conductor. The asset selector has been restricted to a maximum of 10 routers to avoid overwhelming the Conductor with simultaneous download and upgrade operations

I95-26828 In large deployments, issuing a PCLI command to all routers from the Conductor can impact performance. A warning is issued when issuing commands to all routers

I95-26838 Events on the GUI dashboard are displayed out of time order

I95-26842 PCLI does not honor log-level and always logs at DEBUG

I95-26846 mount has been added to the output of save tech-support-info

I95-26848 Commiting a configuration that contains router validation errors from Conductor is not commited

I95-26858 128T may start up with 0 byte length ntpd.conf and global.init files

I95-26862 Asset state timers are unnecessarily aggressive

I95-26868 TCP timers for assets are unnecessarily aggressive

I95-26870, I95-26889, I95-26908 Large configurations can cause the PCLI to time out

I95-26873 Commit and Validatate operations can be executed multiple times for the same configuration. A message now appears preventing the user from executing the same operation twice

I95-26875, I95-26890, I95-26901 Large configuration can cause validation to time out

I95-26924 show assets command erronously prompts for confirmation

I95-26925 Executing save tech-support-info can time out

I95-26928 128T can fault on shutdown

I95-26929 128T can fault on startup if it contains BGP configuration

Release 4.1.0

New Features and Improvements

I95-18018 Upgrade SSR-OS

I95-18889 GUI Dashboard with Improved Navigation

I95-19231 Loopback for BGP Peering

I95-19370 DHCP Server

I95-21279 Azure Accelerated Networking Support

I95-23620 IPv6 Services

I95-23494 Router-Based Services

I95-24541 UDP Transform NAT Traversal

I95-25350 Core Isolation Mask Readability

I95-25543 Non-traffic audit events are now enabled by default.

I95-25647 Quectel EC25-A LTE card support

note

The 4.1.0 release requires the 128T-installer 2.2.0 or greater. By default, this installer will automatically be used.

Issues Fixed

I95-22879 Alarm count in bubble may not reflect the alarms in UI

I95-22926 Configuring dynamic-hostname writes all FQDN-based peers to /etc/hosts.

I95-23122 The output of show platform may display the incorrect number of cores.

I95-23761 Changing the web server's listening port does not affect the forwarding port for host-service.

I95-24337, I95-25803 Interface alarms reference internal device-interface IDs which do not exist in customer visible configuration.

I95-24404 When changing an interface from redundant to non-redundant, no validation will occur if the global-id remains configured.

I95-24453 128T would not start on systems with less than 7GB of RAM.

I95-24469 The user is now presented with a dialog that the 128T-installer will be updated prior to downloading an update of the 128T software.

I95-24598 Alarm overlay will scroll outside of chart bounds when moved with the slider.

I95-24879 Failed to communicate from core network in data center to branch office due to "Error adding session".

I95-25051 Blue update status icon remains blinking on routers.

I95-25164 An invalid node name sent over NETCONF for a service-route can cause the system to fault.

I95-25174 Eroneous journal entries appear on restart of 128T when Zscaler plugin is enabled.

I95-25311 GUI FIB table is inconsistent with output of show fib on PCLI.

I95-25329 PCLI in Conductor GUI creates nested navigation bars

I95-25425 DHCP Relay with /32 and FIB next-hop set to none requires additional service configuration

I95-25429 Rare race condition when applying configuration can cause the system to fault.

I95-25498 Neighbor solicitation messages received on an interface without a IPv6 address will silently drop the messages without incrementing any counters.

I95-25532 The autocomplete on the assets page does not display the asset id correctly.

I95-25536 If a more specific static route matching a kernel route is removed, the kernel route remains inactive.

I95-25539 A GUI initiated upgrade of a Conductor node failes to restart the Conductor service upon completion of the upgrade.

I95-25570 128T process can fault on system shutdown.

I95-25580 Copy-and-pasting a router configuration can cause validation issues.

I95-25655 An unresolvable hostname can cause the rsyslogd service to fault.

I95-25693 Cloning a network-interface causes a configuration warning with the message: 'qp-value' has been deprecated.

I95-25698 Number of Custom charts over 20-25 may result in charts disappearing

I95-25729 The use of overlapping IP addresses for services can cause a failure to route packets.

I95-25788 Static route option for DHCP enabled egress interface not available

I95-25855 The GUI may return null for the PCI address of a device interface.

I95-25864 In the rare case when two HA nodes of the same Router are simultaneously started with factory default configuration, a race condition can occur where configuration from the Conductor may not be applied to the Router.

I95-25891 In very rare cases where networks can have over 18 seconds of delay, the use of the Session Optimization can cause a system fault.

I95-25895 Standby system can fault when upgrading the HA pair from 4.0 to 4.1.

I95-25917 After an upgrade of a system installed from an ISO, 128T is not enabled upon system reboot.

I95-25937 Reverse SSH service fails when the server-side SSH keys change.

I95-25949, I95-26091 IGMPv2 joins are dropped by the 128T.

I95-25981 Setting user preferences to display configuration as tiles or tables is not honored.

I95-25982 Configuring a service next-hop as a network-interface provisioned for DHCPv6 can cause the system to fault.

I95-26007 Executing save tech-support-info can sometimes cause interface state to be incorrectly reported.

I95-26058 System can fault when receiving a malformed packet.

I95-26066 Session classified as applications without a common name can cause the standby node to fault when the data is synchronized.

I95-26107 Path MTU discovery may not resolve when behind a NAT

Special Considerations

When configuring network-interface adjacencies via the PCLI the peer field is now required. If the peer is missing the PCLI will provide a message that it is required:
```
"Argument 'peer' is required"
```

Prior to 4.1.5, any peering with routers that have the same IP addresses was not supported. In 4.1.5 or greater remote peers having the same IP address is now supported in one direction, with the use of "outbound-only". 128T currently does not support ALL routers having the same IP address, the same IP address support is unidirectional. For example: The following is supported:
```
  	R1(172.16.1.1) --- peered --- R2 (192.168.1.1)
  	          |------- peered --- R3 (192.168.1.1)
```
The following is not supported:
```
  	R1(192.168.1.1) --- peered --- R2 (192.168.1.1)
  	           |------- peered --- R3 (192.168.1.1)     
```

Caveats

I95-30103 Entering flat configuration into PCLI does not always create the configuration

Symptom: - When performing configuration using flat (or cut and paste of the complete flat configuration line) the configuration is not applied

Conditions: - When a configuration object does not previously exist and setting an attribute of that configuration object. For example in the following configuration line:
```
configure authority tenant one name one
```
If the "tenant one" configuration object does not exist, the tenant object will not be created. If it does exist then the command will set the attribute "name" to "one"

Corrective Action: - On initial creation, do not use flat configuration operations for creating the configuration.

I95-29842 Nodes with Overlapping DHCP addresses will not be displayed when 'show peers' command is run

Symptom: Nodes with Overlapping DHCP addresses will not be displayed in the output of show peers

Conditions: After upgrading to pre 4.1.5 to 4.1.6 and if Overlapping DHCP addresses are being used in the network topology

Corrective Action: After upgrading the Conductors perform a commit operation from either the PCLI or the Conductor GUI

I95-29733 Conductor UI may not provide an indication that a refresh is in progress (flashing blue dot)

Symptom: When selecting the Router to fresh the available versions to upgrade, the flashing blue indicator may not be present

Conditions: Shortly after both HA conductors have been upgraded and the refresh button is selected for a router

Corrective Action: N/A, no user corrective action can be performed. Waiting for a moment will result in the appearance of the solid blue dot if an upgrade is available (Note: Both conductors must be running a version greater than or equal to the target router version)

I95-29592 Conductor UI and/or PCLI may not update the asset software version correctly

Symptom: The Conductor UI and/or the PCLI may not correctly reflect the software version running on the asset

Conditions: After the asset has been upgraded

Corrective Action: If the asset is not updated after ~5 minutes after an upgrade is performed, the salt-minion will need a restart on the asset node that does not update the version. This is done with the following command on the node as the root linux user:
```
systemctl restart salt-minion
```

I95-29271 PCLI/Netconf subsystem may fault on exit

Symptom: PCLI or Netconf session may fault

Conditions: On session exit (either exit from PCLI or direct Netconf session exit)

Corrective Action: N/A, no user corrective action is required.

I95-29134 save tech-support-info fails to create tech support file

Symptom: save tech-support-info fails with the following error message:
```
"Error: Failed to execute the 'save-tech-support-info' RPC: Fatal error creating tarball"
```
Conditions: When configuration exports have been saved with spaces it in the name of the exported configuration file

Corrective Action: Remove the saved configuration files with spaces in the name and avoid using spaces when exporting configuration. Note: Exporting configuration files with spaces in the name may be prevented in a future release.

I95-28766 Conductor PCLI shows configuration change when no changes have been performed

Symptom: Conductor PCLI may incorrectly provide an * that there is a candidate configuration change

Conditions: Unknown

Corrective Action: None, if the configuration has not changed this indicator can be ignored. A comparison can be performed with compare config running candidate

I95-27946 Commit may fail on Conductor when node in router pair is stopped

Symptom: When performing a commit to a router where one of the nodes is offline, the commit from the Conductor may not respond or may fail. Performing a validate operation a second time may provide the following error response:
```
“✖ Validating...
 % Error: Candidate configuration is invalid:
 1. A request of type validate is already in progress”
```
Conditions: When a node in the router pair is offline.

Corrective Action: The validate operation is sent from the conductor to the nodes to verify that the configuration is correct. The validate will timeout to the node that is offline. Bring the node back online and perform the operation a second time.

I95-27944 Network error may cause upgrade to fail and not retry.

Symptom: The following message is provided on the Conductor UI
```
“The upgrade failed because: NetworkError when attempting to fetch resource.”
```
Conditions: When upgrading the Conductor from the Conductor GUI

Corrective Action: Verify the conductor can reach the 128T repository, once verified to be accessible, perform the upgrade operation again.

I95-27808 sync peer addresses router force from conductor may not trigger router to send address information from peer

Symptom: When performing the following command on the Conductor PCLI, sync peer addresses router force the router may not provide the peer address information

Conditions: Unknown

Corrective Action: Perform the PCLI command on the router to update the information on the conductor.

I95-27722 Alarms for "Peer not reachable" may not clear and will persist after nodes are back and operational

Symptom: Alarms for "peer not reachable" provided in on the Conductor

Conditions: Unknown, seen after system upgrade

Corrective Action: NA, The alarms will clear within 15 minutes.

I95-25947 The upgrade to 4.1 can take upwards of 40 minutes to complete. The increase in installation time is due to the underlying OS upgrade.

I95-25828 Rollback to the previous version of software is not supported due to the underlying operating system upgrade.

I95-25659 When 128T learns the peer IP of a 128T Router through BFD source NAT detection, it uses the learned address for most things, but not for flow moves. Flow move detection relies on the statically configured adjacency within the configuration, not the updated addresses that BFD provides. To prevent for move failures when waypoints are behind a NAT, the administrator should configure external-nat-address on the corresponding adjacency. The value given to that field should be that same source NATed address.

I95-25823 kernel warnings can be seen in journalctl when running the 128T within Azure. An example of the warning messages can be seen below.

      Aug 29 02:23:32 sct1_dut1 kernel: Modules linked in: rte_kni(OE) igb_uio(OE) uio vfat fat intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aes
      Aug 29 02:23:32 sct1_dut1 kernel: CPU: 1 PID: 1620 Comm: fastLane Tainted: G           OE  ------------   3.10.0-862.14.4.el7.x86_64 #1
      Aug 29 02:23:32 sct1_dut1 kernel: Hardware name: ADI Engineering RCC-VE2/RCC-VE2, BIOS ADI_RCCVE2-01.00.00.00 08/25/2017
      Aug 29 02:23:32 sct1_dut1 kernel: Call Trace:
      Aug 29 02:23:32 sct1_dut1 kernel:  [<ffffffff83713754>] dump_stack+0x19/0x1b
      Aug 29 02:23:32 sct1_dut1 kernel:  [<ffffffff830945d8>] __warn+0xd8/0x100
      Aug 29 02:23:32 sct1_dut1 kernel:  [<ffffffff8309471d>] warn_slowpath_null+0x1d/0x20
      Aug 29 02:23:32 sct1_dut1 kernel:  [<ffffffff831256b9>] update_cpumasks_hier+0x3c9/0x410
      Aug 29 02:23:32 sct1_dut1 kernel:  [<ffffffff83122cd0>] ? cpuset_read_u64+0x100/0x100
      Aug 29 02:23:32 sct1_dut1 nodeMonitor[1624]: Processed configuration update (2)
      Aug 29 02:23:32 sct1_dut1 kernel:  [<ffffffff83125953>] cpuset_write_resmask+0x253/0xa10
      Aug 29 02:23:32 sct1_dut1 kernel:  [<ffffffff8311b020>] ? cgroup_rightmost_descendant+0x80/0x80
      Aug 29 02:23:32 sct1_dut1 kernel:  [<ffffffff83125700>] ? update_cpumasks_hier+0x410/0x410
      Aug 29 02:23:32 sct1_dut1 kernel:  [<ffffffff8311d411>] cgroup_file_write+0x1d1/0x2d0
      Aug 29 02:23:32 sct1_dut1 kernel:  [<ffffffff832221c8>] ? __sb_start_write+0x58/0x110
      Aug 29 02:23:32 sct1_dut1 kernel:  [<ffffffff832d5437>] ? security_file_permission+0x27/0xa0
      Aug 29 02:23:32 sct1_dut1 kernel:  [<ffffffff8321f240>] vfs_write+0xc0/0x1f0
      Aug 29 02:23:32 sct1_dut1 kernel:  [<ffffffff8322006f>] SyS_write+0x7f/0xf0
      Aug 29 02:23:32 sct1_dut1 kernel:  [<ffffffff8372579b>] system_call_fastpath+0x22/0x27
      Aug 29 02:23:32 sct1_dut1 kernel: ---[ end trace e9955ec11f51ffa9 ]---
      Aug 29 02:23:32 sct1_dut1 kernel: ------------[ cut here ]------------
      Aug 29 02:23:32 sct1_dut1 kernel: WARNING: CPU: 1 PID: 1620 at kernel/cpuset.c:994 update_cpumasks_hier+0x3c9/0x410
      Aug 29 02:23:32 sct1_dut1 kernel: Modules linked in: rte_kni(OE) igb_uio(OE) uio vfat fat intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aes
      Aug 29 02:23:32 sct1_dut1 kernel: CPU: 1 PID: 1620 Comm: fastLane Tainted: G        W  OE  ------------   3.10.0-862.14.4.el7.x86_64 #1
      Aug 29 02:23:32 sct1_dut1 kernel: Hardware name: ADI Engineering RCC-VE2/RCC-VE2, BIOS ADI_RCCVE2-01.00.00.00 08/25/2017
      Aug 29 02:23:32 sct1_dut1 kernel: Call Trace:
      Aug 29 02:23:32 sct1_dut1 kernel:  [<ffffffff83713754>] dump_stack+0x19/0x1b
      Aug 29 02:23:32 sct1_dut1 kernel:  [<ffffffff830945d8>] __warn+0xd8/0x100
      Aug 29 02:23:32 sct1_dut1 kernel:  [<ffffffff8309471d>] warn_slowpath_null+0x1d/0x20
      Aug 29 02:23:32 sct1_dut1 kernel:  [<ffffffff831256b9>] update_cpumasks_hier+0x3c9/0x410
      Aug 29 02:23:32 sct1_dut1 kernel:  [<ffffffff83122cd0>] ? cpuset_read_u64+0x100/0x100
      Aug 29 02:23:32 sct1_dut1 kernel:  [<ffffffff83125953>] cpuset_write_resmask+0x253/0xa10
      Aug 29 02:23:32 sct1_dut1 kernel:  [<ffffffff8311b020>] ? cgroup_rightmost_descendant+0x80/0x80
      Aug 29 02:23:32 sct1_dut1 kernel:  [<ffffffff83125700>] ? update_cpumasks_hier+0x410/0x410
      Aug 29 02:23:32 sct1_dut1 kernel:  [<ffffffff8311d411>] cgroup_file_write+0x1d1/0x2d0
      Aug 29 02:23:32 sct1_dut1 kernel:  [<ffffffff832221c8>] ? __sb_start_write+0x58/0x110
      Aug 29 02:23:32 sct1_dut1 kernel:  [<ffffffff832d5437>] ? security_file_permission+0x27/0xa0
      Aug 29 02:23:32 sct1_dut1 kernel:  [<ffffffff8321f240>] vfs_write+0xc0/0x1f0
      Aug 29 02:23:32 sct1_dut1 kernel:  [<ffffffff8322006f>] SyS_write+0x7f/0xf0
      Aug 29 02:23:32 sct1_dut1 kernel:  [<ffffffff8372579b>] system_call_fastpath+0x22/0x27
      Aug 29 02:23:32 sct1_dut1 kernel: ---[ end trace e9955ec11f51ffaa ]---

Release 4.1.10​

Issues Fixed​

Release 4.1.9​

Issues Fixed​

Release 4.1.8​

Issues Fixed​

Release 4.1.7​

Issues Fixed​

Release 4.1.6​

Issues Fixed​

Release 4.1.5​

Product Enhancements​

Overlapping IP address support for adjacencies​

Issues Fixed​

Release 4.1.4​

Issues Fixed​

Release 4.1.3​

Issues Fixed​

Release 4.1.2​

Issues Fixed​

Release 4.1.1​

Issues Fixed​

Release 4.1.0​

New Features and Improvements​

Issues Fixed​

Special Considerations​

Caveats​

Release 4.1.10

Issues Fixed

Release 4.1.9

Issues Fixed

Release 4.1.8

Issues Fixed

Release 4.1.7

Issues Fixed

Release 4.1.6

Issues Fixed

Release 4.1.5

Product Enhancements

Overlapping IP address support for adjacencies

Issues Fixed

Release 4.1.4

Issues Fixed

Release 4.1.3

Issues Fixed

Release 4.1.2

Issues Fixed

Release 4.1.1

Issues Fixed

Release 4.1.0

New Features and Improvements

Issues Fixed

Special Considerations

Caveats