9

Edge Routing and Networking

Edge routers, bridges, and gateways play an integral role in IoT development. One primary function of edge computing is its ability to bridge networks, route across networks, and provide advanced network management to allow for a seamless enterprise-level of management from a corporate-managed data center to a remote edge device. This requires robust, autonomous, and failsafe networking.

We have already discussed the different PAN and WAN protocols earlier in this book. A significant function of the far-edge router or gateway is to perform the bridging between non-IP networks and other IP-based networks, including cellular and 5G.

We begin this chapter by reviewing some of the fundamentals of routing in general. The edge is where we aggregate data from sensors and PAN communication as well as outbound connections to cloud sources through the Internet. IoT devices require some type of communication to the Internet. Often, these devices do not have the resources themselves to reach the Internet directly (for example, a lack of TCP/IP networking hardware and software); therefore, edge computing and routing is needed to assist with data movement.

It is important to understand and correctly design the routing of data into and out of the edge nodes. We then address topics of VLANs and software-defined networks, which are taking hold in IoT deployments. These technologies allow for edge-managed routers to intelligently separate traffic.

TCP/IP network functions at the edge

Routing refers to the steering of data between multiple clients and networks. For this discussion, we will examine TCP/IP based routing and its applicability on the edge.

Routing functions

The basic function of a router is to bridge connections between network segments. Routing is considered a layer-three function of the OSI standard model because it makes use of the IP address layer to guide packet movement. Essentially, all routers rely on a routing table to guide the flow of data. The routing table is used to find the best match to a packet's destination IP address.

There are several proven algorithms used for efficient routing. One type of routing is dynamic routing, where the algorithms react to changes in the network and topology. Information about the state of the network is shared through a routing protocol on a timed basis or upon a triggered update. Examples of dynamic routing are distance-vector routing and link-state routing. Alternatively, static routing is important and useful for small networks that need specific paths configured between routers. Static routes are non-adaptive and thus have no need to scan topologies or update metrics. These are preset on the router:

An important metric in routing is the time of convergence. Convergence occurs when all routers in a network share the same topological information and state.

Typical edge routers will support routing protocols such as Border Gateway Protocol (BGP), Open Shortest Path First (OSPF), Routing Information Protocol (RIP), and RIPng. An architect using edge routers in the field needs to be aware of the congestion and costs of using a certain routing protocol over others, especially if the interconnection between routers is a data-capped WAN connection:

A typical routing table is as follows:

[administrator@Edge-Computer: /]$ route 
Table: wan
Destination         Gateway      Device UID         Flags    Metric    Type
default             96.19.152.1    wan              onlink   0         unicast
Table: main
Destination         Gateway      Device UID         Flags    Metric    Type
96.19.152.0/21      *            wan                         0         unicast
172.86.160.0/20     *            iface:cloud-edge1           0         unicast
172.86.160.0/20     None         None                        256       blackhole
192.168.1.0/24      *            primarylan                  0         unicast
2001:470:813b::/48  *            *iface:cloud-edge1          256       unicast
fe80::/64           *            lan                         256       unicast
Table: local
Destination         Gateway      Device UID         Flags    Metric    Type
96.19.152.0         *            wan                         0         broadcast
96.19.153.13        *            wan                         0         local
96.19.159.255       *            wan                         0         broadcast
127.0.0.0           *            *iface:lo                   0         broadcast
.
.

In this example, there are three tables: wan, main, and local. A router may have a single routing table or multiple routing tables. The example above was from a real edge routing system in the field that had a used cellar WAN uplink and had a local subnet with multiple networks and a virtual LAN. Each table contains particular routing paths specific to that interface:

It should also be noted in the preceding example that IPv6 addresses are intermixed with IPv4 addresses. For example, 2001:470:813b::/48 in the main table is an IPv6 address with a /48 bit subnet.

PAN-to-WAN bridging

A critical area of IoT and bridging is the conversion of non-IP communication to IP-based WAN networking. An example is converting from captured Bluetooth advertising packets to MQTT streams across 5G. This is a complex process involving multiple hardware components, radios, device drivers, and communication stacks. However, it is fundamental to the process.

At this point, we assume there is the proper PAN and necessary WAN hardware installed on the edge computer. Normally, the process will either be event-driven or polled. For event-driven traffic generation, the arrival of PAN data will signal an event (the arrival of a Bluetooth advertising message). If the event is qualified, it may have the following actions:

While this is a simple representation of a bridge, the logic within the control system is what makes the edge a powerful technology with rules engines, machine learning, advanced power management techniques, and edge analytics. These concepts will be discussed in detail later in this book.

An example of such bridging is shown in the following figure. Here, a typical cellular NBIOT module is used (in this case a Telit cellular module). These modules typically connect to a hosting computer across UART or USB interfaces (for up to 480 Mbit/s communications). Modules such as the Telit cellular module can operate using a built-in Hayes-compatible AT command set and use a TCP stack built into the module or connect in a raw format across USB using the Remote Network Driver Interface Specification (RNDIS) or Ethernet over USB. Both are protocols that transport Ethernet frames over USB.

Since typical 802.3 Ethernet transports data in frames and USB doesn't, a protocol must be used if you attempt to use a USB connection as a TCP networking component. Ethernet Control Model (ECM) and Ethernet Emulation Model (EEM) are two such protocols. Of the two, ECM is the simplest to implement and use.

The non-IP side of the bridge is controlled by the edge router as captured packets or advertisements from a Bluetooth device propagate up the typical BLE stack to the application. In this case, the binding application is the edge control application that performs all the necessary initialization and network bindings to bridge the two disparate network segments.

The application is also the correct entity to filter, denature (filtering data based on rules such as privacy rules), cache, and analyze the incoming stream of data and make the decision to transmit the data up towards the WAN:

A screenshot of a cell phone Description automatically generated

Figure 1: PAN-to-WAN conversion on edge routing. Here a Telit Cellular NB-IoT module is used that communicates to the hosting edge computer over a high-speed USB interface. RNDIS is used to force ethernet frames across the USB link. The edge control application performs most of the setup, binding, and network interfacing. Additionally, it controls the rules engine and logic as to which packets should be propagated from the PAN to the WAN.

While all cellular radios require a SIM to identify the device to the cellular carrier, there are two methods to consider for edge designs. A traditional SIM card can be used and built into a card carrier on the PCB board. Alternatively, devices like the Telit HE910 support "SIM-on-chip," which is a fixed and nonremovable SIM design intended to be surface-mounted permanently on the PCB. This is intended for resiliency in harsh or hostile IoT deployments like off-road vehicles or where moisture is a problem.

The following example shows how a communication session will work with a cellular radio, such as the Telit HE910, using AT commands. The edge computer or processor in this case can have multiple applications intending to open network sockets and transmit or receive data across the Internet. Each application will be assigned an AT (attention) virtual port (AT0, AT1 and so on). The Hayes AT command set operates in either COMMAND mode or ONLINE mode. In COMMAND mode, AT prefixed instructions configure and control the modem, whereas in ONLINE mode, data entered by the application or user is freely sent upstream to the network. An IP address is assigned to each AT port on the modem to match the application.

Here's an example of opening a cellular-based socket on the Telit module.

In COMMAND mode, open the socket using:

AT#SD=3,0,80,"host name"
CONNECT
[data is delivered]
+++ 
OK

The AT command SD signifies "socket dial" and will open a TCP/UDP connection to a known host like azure.microsoft.com. The command is parsed as:

AT#SD=<connId>,<txProt>, <rPort>, <IPaddr> [, <closureType> [, <IPort>[,<connMode>]]]

There are hundreds of unique and vendor-defined AT commands. This has been a very short look into how the radio forms an IP-based connection. Different models of modules may use different commands or interfaces. It is up to the architect to understand the radio characteristics for the solution they are developing.

Figure 2: Telit HE910 cellular radio example. Here, three applications create virtual AT ports that reach across a USB connection from the edge processor to the Telit radio module.

Failover and out-of-band management

Failover is a critical feature for certain IoT edge routers, particularly for moving vehicles and patient care applications. Failover, as the name implies, is switching from one WAN interface to another when a primary source is lost. Loss of WAN could be attributed to the loss of cellular connectivity in a tunnel. A logistics company with a fleet of cold-storage trucks may need to guarantee connectivity where cellular service varies nationwide. A failover from one cellular provider to another by use of multiple SIM identities can help mitigate and smooth over connectivity. Another use case could be using customer Wi-Fi as a primary WAN interface for in-home health monitoring but enabling a failover to cellular WAN if the Wi-Fi signal is lost. Failover should be seamless and automatic without packet loss or a noticeable impact on data latency.

Out-of-band management (OOBM) should also be considered for IoT devices. OOBM is useful in conditions where a dedicated and isolated channel is needed for the management of equipment. Sometimes this is called light-out management (LOM) if the primary systems are offline, damaged, or have power loss; one is still able to manage and inspect the equipment remotely through a sideband channel. In IoT, this can be useful for situations that need guaranteed uptime and remote management, such as oil and gas monitoring or industrial automation. A well-designed OOBM system should have no reliance on the system being monitored to be functional. Typical management planes, such as VNC or SSH tunnels, require the device to be booted and functional.

When building an IoT architecture, you must ask what assets need OOBM. Typically, you want that form of management for crucial points in your edge-to-cloud network. If an edge router could be compromised or is crucial to manage (for example, fiduciary transactions or emergency services), it is useful to have OOBM to determine the health of the machine without relying on the normal network. This allows an IT manager to reset the device, stop malware attacks, and reprogram the device even if the traditional network path is lost.

An OOBM needs to be adjunct and isolated from the system, as shown in the following figure:

Figure 3: Example configuration for out-of-band management

An example of a use case where failover is used in IoT deployments is in areas of supermarket retail. Modern grocers, as well as the new Amazon (cashier-less grocers), will have point-of-sale machines like cash registers as well as IoT sensors, Bluetooth beacons, and security systems. They also include Internet-based music services and customer Wi-Fi. In the event of a failure with a traditional Internet service, the system will switch to a failover cellular service. That service has less bandwidth and costly service agreements, so the edge systems cap the quality of service for things like customer Wi-Fi and Internet-based music in preference of keeping the cashiers and financial transactions working.

Edge-level network security

Security must be considered from sensor to cloud when architecting IoT solutions. The edge has a significant role in secure data communication as well as isolating different data paths from each other. This section will explore some network standards that should be deployed at the edge.

VLANs

A virtual LAN (VLAN) functions like any other physical LAN, but it enables computers and other devices to be grouped together even if they are not physically attached to the same network switch. The partitioning occurs in the data link layer (two) of the OSI model. VLANs are a form of network segmentation of devices, applications, or users, although they share the same physical network. A VLAN can also group hosts together, although they are not on the same network switch, essentially easing the burden of partitioning networking without running extra cables. IEEE 802.1Q is the standard by which VLANs are built. Essentially, a VLAN uses an identifier or tag consisting of 12 bits in the Ethernet frame. Therefore, there is a hard limit of 4,096 potential VLANs on a single physical network.

A switch can assign a port to directly map to a particular VLAN. Since the VLAN is designed in layer two of the stack, traffic can be tunneled through layer three, allowing geographically separated VLANs to share a common topology.

The following figure shows a corporate point-of-sale (POS) and a VOIP system that is virtually isolated from a set of IoT devices as well as the guest Wi-Fi. This is done through VLAN addressing, although the system shares the same physical network. Here, we assume this is a smart IoT deployment where all edge IoT devices and sensors carry an IP stack and are addressable through the LAN:

Figure 4: An example VLAN architecture in a franchise or retail scenario

VLAN design is particularly useful in the IoT space. Isolating IoT devices from other corporate functions is a typical scenario. VLANs are only useful when dealing with IP addressable devices.

VPN

VPN tunnels are used to establish a secure connection to a remote network over a public network. For example, VPN tunnels can be used across the Internet by an individual to connect to a secure corporate network while traveling, or by two office networks to function as one network. The two networks set up a secure connection to the (normally) unsecured Internet by assigning VPN encryption protocols.

For IoT deployment, a VPN is necessary to move data from remote sensors and edge devices into a corporate or private LAN. Typically, a corporation will be behind a firewall, and a VPN is the only means to move data directly into a private on-premises server. In these cases, a VPN may be a necessary component of the router to bridge the networks. Later in this chapter, a discussion of software-defined networks will present an alternative method to securing networks.

Several VPNs exist:

When comparing VPN functionality, it is important to look at functions such as logging (logs of user bandwidth, number of users' connections) if the service has its own DNS, IPV6 support, SSL and SSH support, data encryption and handshake encryption methods (for example, AES256), and SSL service ratings. SSL service ratings are reported by Qualsys SSL Lab test tools (www.qualsys.com).

A VPN must either trust the underlying network protocols or provide its own security. VPN tunnels typically use IPsec to authenticate and encrypt packets exchanged across the tunnels. To set up a VPN tunnel router on one end, there must be another device (usually a router) that also supports IPsec on the other end. Internet Key Exchange (IKE) is the security protocol in IPsec. IKE has two phases. The first phase is responsible for establishing a secure communication channel, and in the second phase, the established channel is used by IKE peers. The router has several different security protocol options for each phase, but the default selections will be sufficient for most users. Each IKE exchange uses one encryption algorithm, one hash function, and one DH group to make a secure exchange:

In IKE phase one, you can only select one DH group if you are using an aggressive exchange mode.

The algorithms are listed in order of priority. You can reorder this priority list by clicking and dragging algorithms up or down. Any selected algorithm may be used for IKE, but the algorithms at the top of the list are likely to be used more often.

A word of caution on mobile and power-constrained IoT deployments: A traditional VPN cannot withstand moving in and out of a persistent network connection (such as cellular roaming, carrier switching, or occasionally powered devices). If the network tunnel is disrupted, it causes timeouts, disconnects, and failures. Some mobile VPN software, such as Host Identity Protocol (HIP) from the IETF, attempts to resolve the problem by disassociating different IP addresses used when roaming to the VPN logical connection. Another alternative is software-defined networking (SDN), which will be covered later in this chapter.

Traffic shaping and QoS

Traffic shaping and quality of services (QoS) features are useful in deployments that need a guaranteed level of service when dealing with congestion or variable network loads. For example, in an IoT use case, when mixing live video streams as well as public Wi-Fi, the video feeds may need priority and a guaranteed level of quality, especially in a situation of public safety or surveillance. Left to itself, incoming data from the WAN to an edge router will be serviced on a first-come, first-served basis:

Dynamic shaping and packet priority allow rules to be created based on the type of data or application rather than just the IP address or port.

A method to classify and manage network traffic is differentiated services (DiffServ). DiffServ uses a 6-bit differentiated service code point (DSCP) in the IP header for packet classification. The concept of DiffServ is that complex functions (such as packet classification and policing) can be carried out at the edge of the network by edge routers, which then mark the packet to receive a particular type of per-hop behavior. Traffic entering a DiffServ router is subject to classification and conditioning. Additionally, a DiffServ router is free to change the classification of a previously marked packet by a different router. DiffServ is a coarse-grain tool for traffic management since the chain of routers in a link do not all need to support it. The router will then manage the different packet classes through QoS features. Alternatively, IntServ, which stands for integrated services, assists in QoS and mandates that all routers in the chain support it. This is a form of fine-grain QoS.

Another aspect of network quality is the mean opinion score (MOS). MOS is the arithmetic mean of individual values on a scale of the quality of a system from a user perspective. This is commonly used in Voice over Internet Protocol (VOIP) applications but certainly can be used for vision systems, imaging, streaming data, and user interface usability. It is based on a subjective rating of one to five (one being the worst quality, five being the best quality) and should be used in a feedback loop to increase capacity or reduce data sizes to match capacity.

An edge router bridging a PAN to an IP-based WAN has several options at its disposal to respond to changes in link quality and degradation of network services, for example, in an IoT deployment for fleet trucking where the carrier signal may degrade. In these situations, the router can employ TCP performance-enhancing proxies (PEPs) to overcome and compensate for quality changes (RFC 3135). PEPs can be used in the transport layer or application layer of the stack and differ based on the physical medium. Forms of PEP include:

A PEP consists of the following functions:

Security functions

An edge router or gateway fulfills another important role by providing a layer of security between the WAN, Internet, and the underlying PAN/IoT devices. Many devices will lack the necessary resources, memory, and computational power to provide robust security and provisioning. Whether the architect is building their own gateway service or purchasing one, the following features should be considered to secure IoT components.

Firewall protection is the most basic form of security. There are two basic forms of firewalls for telecommunications. The first is a network firewall that filters and controls information flow from one network to another. The second is a host-based firewall that protects applications and services local to that machine. In the case of IoT edge routers, we focus on network firewalls. By default, a firewall will prevent certain types of network traffic from flowing into a firewall-protected zone, but any traffic originating within that zone is allowed to flow outward. A firewall will find and isolate information based on packets, states, or applications depending on the sophistication of the firewall. Typically, zones will be created around network interfaces with rules designed to control traffic flow between zones. An example would be an edge router with a guest Wi-Fi zone and a corporate private zone.

A packet firewall can isolate and contain certain traffic based on the source or destination IP, ports, MAC addresses, IP protocols, and other information contained in the packet header. A stateful firewall operates in layer four of the OSI stack. It gathers and aggregates packets looking for patterns and state information such as new connections versus existing connections. Application filtering is yet more sophisticated in that it can search for certain types of application network flows, including FTP traffic or HTTP data.

A firewall can also utilize a demilitarized zone (DMZ). A DMZ is simply a logical zone. A DMZ host is effectively not firewalled, in the sense that any computer on the Internet may attempt to remotely access network services at the DMZ IP address. Typical uses involve running a public web server and sharing files. The DMZ host is usually specified by a direct IP address.

Port forwarding is a concept that enables certain ports behind a firewall to be exposed. Several IoT devices need an open port to provide services that are controlled by cloud components. Again, a rule is constructed that allows a specified IP address within the firewalled zone to have an exposed port.

Both DMZs and port forwarding open ports and interfaces within an otherwise protected network. Caution should be exercised to ensure this is the intention of the architect. In a mass IoT deployment with many edge routers, a generic rule to open a port may be useful for one location but a significant security risk for another. Additionally, keeping an audit of DMZ and open ports should be a security process as network topology and configurations change with time. A DMZ, at any one time, could lead to an open hole in the network security later.

Metrics and analytics

In many instances, IoT edge devices will be under service-level agreements or data caps when using metered data such as 4G LTE. In other situations, the edge router or gateway is a host of other PAN networks and IoT devices. It should serve as a central (but local) authority on the health of the PAN network/mesh. Metrics and analytics are useful for collecting and resolving connectivity and cost challenges, especially as the scale of IoT grows. Typical metrics and collections should include the following:

An example of metrics usage is in mobile IoT for municipal vehicles. In this case, a large Midwestern municipality required that their fleet of snow-removal vehicles be connected to the Internet and report information such as GPS location, salt status, and plow status. This was needed to create real-time maps of the city streets and efficiently position plows where needed. Metrics were gathered to understand signal integrity and WAN uptime. Some areas were in cellular "shadows" in more hilly areas. This information was used to build an edge solution that cached data until the cellular signal was reestablished.

These types of metrics should be collected and monitored on a scheduled and automatic basis. Additionally, advanced routers should be able to construct rules and alerts when certain events or abnormal behaviors are seen at the edge.

Software-defined networking

Software-defined networking (SDN) is a method of decoupling the software and algorithms that define the networking control plane from the underlying hardware that manages the forwarding plane.

Additionally, network function virtualization (NFV) is defined as providing network functions that run on vendor-agnostic hardware. NFV describes virtualizing network features typically found in layers four through seven of the stack. These two paradigms provide the industry methods to build, scale, and deploy significantly complex network architecture in a very flexible manner. Preceding all, this greatly reduces enterprise cost in network infrastructure since most of the services can run in the cloud.

Why is this important for devices at the edge and where does it fit in with the IoT? We have spent a good deal of this book detailing data movement from a sensor to a cloud and yet have taken for granted how the overall inter-networking infrastructure will scale to a billion additional nodes on a network. One needs only to ask an IT administrator to put an additional million endpoints on a corporate network where the nodes are heterogeneous, in remote locations, and some moving in vehicles to understand the impact on the overall networking infrastructure. Traditional networking does not scale, and we are forced to consider alternative means for building large networks with minimal impact and cost.

SDN is important for IoT deployments and should be considered when dealing with devices that must be segregated for security or performance reasons. For example, a mobile and moving edge system that is SDN capable can establish a secure cloud SDN host. The edge system can move between different carriers and communication systems but always maintain a static IP address to the Internet.

SDN architecture

The journal article Software-Defined Networking: A Comprehensive Survey (D. Kreutz, F. M. V. Ramos, P. E. Veríssimo, C. E. Rothenberg, S. Azodolmolky and S. Uhlig, in Proceedings of the IEEE, vol. 103, no. 1, pp. 14-76, Jan. 2015) defines SDN as having four characteristics:

An SDN's infrastructure is similar to a traditional network since it utilizes similar hardware: switches, routers, and middleboxes. The principal difference, however, is that an SDN utilizes the fast server-class off-the-shelf computing power without complex and unique embedded control hardware. These server platforms are usually in the cloud performing network services in software rather than custom ASICs. The edge routers are essentially dumb without autonomous control. The SDN architecture separates the control plane (the logic and function control) and the data plane (that executes datapath decisions and forwards traffic). The data plane consists of routers and switches that have an association with an SDN controller.

Everything above the data plane forwarding hardware typically can reside in the cloud or on private data center hardware, as shown in the following figure:

Figure 5: Typical SDN architecture abstraction

The illustration shows simplified switching and forwarding nodes that reside on the data plane and marshal information along prescribed paths determined by an abstracted SDN controller that can live in a cloud instance. The SDN controller manages the control plane via a southbound interface to the forwarding nodes. Network applications can reside on top of the SDN controller and manipulate the data plane with services such as threat monitoring and intrusion detection. These services typically require custom and unique hardware solutions to be deployed and managed by the customer.

Traditional internetworking

A typical internetworking architecture will use a collection of managed hardware/software components that are single-purposed and contain embedded software/solutions. Often, these use no-commodity hardware and dedicated ASIC designs. Typical functions include routing, managed switches, firewalls, deep packet inspection and intrusion detection, load balancers, and data analyzers. Such dedicated appliances need to be managed by the customer and staffed with trained network IT personnel to maintain and administer them. These components may come from multiple vendors and require significantly different methods of management.

In this configuration, the data plane and control plane are unified. When the system needs to add or remove another node or set up a new data path, many of the dedicated systems need to be updated with new VLAN settings, QoS parameters, access control lists, static routes, and firewall pass-throughs. This may be manageable when dealing with several thousand endpoints. However, when we scale to millions of nodes that are remote, moving and connecting/disconnecting, such traditional technology regularly becomes untenable:

Figure 6: Traditional networking components. In a typical inter-networking scenario, systems providing services such as security, deep packet inspection, load balancing, and metric gathering require custom vendor hardware and management systems. This introduces difficulty in management and scaling for large installations, remote devices, and moving systems since the control and data planes are unified.

SDN benefits

An SDN model of networking should be considered for mass IoT deployments, especially when a customer needs to establish the provenance and security of a wide deployment of nodes. An architect should consider the following situations when using an SDN:

Three aspects of SDN makes it attractive to IoT deployments:

Later in this book, Chapter 13, IoT and Edge Security, will explore Software-Defined Perimeters (SDPs) as another example of network function virtualization and how it can be used to create micro-segments and device isolation, which is critical for IoT security.

Summary

Edge routers, bridges, and gateways play an integral role in IoT development. The functionality provided by an edge router allows enterprise-level security, routing, resiliency, and quality of service. The bridge/gateway plays an important role in translating non-IP networks into the IP-based protocols necessary for Internet and cloud connectivity. It is also important to realize that the growth of IoT to billions of nodes will take shape with low-cost and constrained electronics. Features such as enterprise routing, tunneling, and VPNs require significant hardware and processing capabilities, and it makes sense to use routers and gateways for that service function. Later in this book, we will also look at how edge routers play a vital role in edge processing and fog computing.

The next chapter will go deep into IoT-based protocols such as MQTT and CoAP, with working examples. These protocols are the lightweight language of IoT and more often than not use gateways and edge routers as translators.