]> Verizon

60 Sylvan Rd Waltham MA 02451 United States of America damascene.joachimpillai@verizon.com

Mojatatu Networks

Suite 200, 15 Fitzgerald Rd. Ottawa K2H 9G1 Ontario Canada hadi@mojatatu.com

Routing Internet Engineering Task Force ForCES Inter-FE This document describes how to extend the Forwarding and Control Element Separation (ForCES) Logical Functional Block (LFB) topology across Forwarding Elements (FEs) by defining the inter-FE LFB class. The inter-FE LFB class provides the ability to pass data and metadata across FEs without needing any changes to the ForCES specification. The document focuses on Ethernet transport.

In the ForCES architecture, a packet service can be modeled by composing a graph of one or more LFB instances. The reader is referred to the details in the ForCES model. The ForCES model describes the processing within a single Forwarding Element (FE) in terms of Logical Functional Blocks (LFBs), including provision for the Control Element (CE) to establish and modify that processing sequence, and the parameters of the individual LFBs. Under some circumstances, it would be beneficial to be able to extend this view and the resulting processing across more than one FE. This may be in order to achieve scale by splitting the processing across elements or to utilize specialized hardware available on specific FEs. Given that the ForCES inter-LFB architecture calls for the ability to pass metadata between LFBs, it is imperative to define mechanisms to extend that existing feature and allow passing the metadata between LFBs across FEs. This document describes how to extend the LFB topology across FEs, i.e., inter-FE connectivity without needing any changes to the ForCES definitions. It focuses on using Ethernet as the interconnection between FEs.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in .

This document depends on the terms (below) defined in several ForCES documents: , , , , , and . Control Element (CE) Forwarding Element (FE) FE Model LFB (Logical Functional Block) Class (or type) LFB Instance LFB Model LFB Metadata ForCES Component LFB Component ForCES Protocol Layer (ForCES PL) ForCES Protocol Transport Mapping Layer (ForCES TML)

The scope of this document is to solve the challenge of passing ForCES-defined metadata alongside packet data across FEs (be they physical or virtual) for the purpose of distributing the LFB processing.

The FEs involved in the inter-FE LFB belong to the same Network Element (NE) and are within a single administrative private network that is in close proximity. The FEs are already interconnected using Ethernet. We focus on Ethernet because it is commonly used for FE interconnection. Other higher transports (such as UDP over IP) or lower transports could be defined to carry the data and metadata, but these cases are not addressed in this document.

To illustrate the problem scope, we present two use cases where we start with a single FE running all the LFBs functionality and then split it into multiple FEs achieving the same end goals.

A sample LFB topology depicted in demonstrates a service graph for delivering a basic IPv4-forwarding service within one FE. For the purpose of illustration, the diagram shows LFB classes as graph nodes instead of multiple LFB class instances. Since the purpose of the illustration in is to showcase how data and metadata are sent down or upstream on a graph of LFB instances, it abstracts out any ports in both directions and talks about a generic ingress and egress LFB. Again, for illustration purposes, the diagram does not show exception or error paths. Also left out are details on Reverse Path Filtering, ECMP, multicast handling, etc. In other words, this is not meant to be a complete description of an IPv4-forwarding application; for a more complete example, please refer to the LFBLibrary document. The output of the ingress LFB(s) coming into the IPv4 Validator LFB will have both the IPv4 packets and, depending on the implementation, a variety of ingress metadata such as offsets into the different headers, any classification metadata, physical and virtual ports encountered, tunneling information, etc. These metadata are lumped together as "ingress metadata". Once the IPv4 validator vets the packet (for example, it ensures that there is no expired TTL), it feeds the packet and inherited metadata into the IPv4 unicast LPM (Longest-Prefix-Matching) LFB.

| +------------->| | | | | + ingress | | + ingress |IPv4 | IPv4 pkt | | | metadata | | metadata |Ucast+------------>| +--+ | +----+ |LPM | + ingress | | | +-+-+ IPv4 +-----+ + NHinfo +---+ | | | Validator metadata IPv4 | | | LFB NextHop| | | LFB | | | | | | IPv4 pkt | | | + {ingress | +---+ + NHdetails} Ingress metadata | LFB +--------+ | | Egress | | <--+ |<-----------------+ | LFB | +--------+ ]]>

The IPv4 unicast LPM LFB does an LPM lookup on the IPv4 FIB using the destination IP address as a search key. The result is typically a next-hop selector, which is passed downstream as metadata. The NextHop LFB receives the IPv4 packet with associated next-hop (NH) information metadata. The NextHop LFB consumes the NH information metadata and derives a table index from it to look up the next-hop table in order to find the appropriate egress information. The lookup result is used to build the next-hop details to be used downstream on the egress. This information may include any source and destination information (for our purposes, which Media Access Control (MAC) addresses to use) as well as egress ports. (Note: It is also at this LFB where typically, the forwarding TTL-decrementing and IP checksum recalculation occurs.) The details of the egress LFB are considered out of scope for this discussion. Suffice it to say that somewhere within or beyond the Egress LFB, the IPv4 packet will be sent out a port (e.g., Ethernet, virtual or physical).

demonstrates one way that the router LFB topology in may be split across two FEs (e.g., two Application-Specific Integrated Circuits (ASICs)). shows the LFB topology split across FEs after the IPv4 unicast LPM LFB.

| +------------->| | | | | | + ingress | | + ingress |IPv4 | | | +----------+ metadata | | metadata |Ucast| | | ^ +----+ |LPM | | | | IPv4 +--+--+ | | | Validator | | | LFB | | +---------------------------------------------------|---------+ | IPv4 packet + {ingress + NHinfo} metadata FE2 | +---------------------------------------------------|---------+ | V | | +--------+ +--------+ | | | Egress | IPv4 packet | IPv4 | | | <-----+ LFB |<----------------------+NextHop | | | | |{ingress + NHdetails} | LFB | | | +--------+ metadata +--------+ | +-------------------------------------------------------------+ ]]>

Some proprietary interconnections (for example, Broadcom HiGig over XAUI ) are known to exist to carry both the IPv4 packet and the related metadata between the IPv4 Unicast LFB and IPv4NextHop LFB across the two FEs. This document defines the inter-FE LFB, a standard mechanism for encapsulating, generating, receiving, and decapsulating packets and associated metadata FEs over Ethernet.

In this section, we show an example of an arbitrary Network Function that is more coarsely grained in terms of functionality. Each Network Function may constitute more than one LFB.

| +------------->| | | | | 1 | + NF1 | | + NF1/2 |NF3 | | | +----------+ metadata | | metadata | | | | ^ +----+ | | | | | +--+--+ | | | | | | | | +---------------------------------------------------|---------+ V ]]>

The setup in is typical of most packet processing boxes where we have functions like deep packet inspection (DPI), NAT, Routing, etc., connected in such a topology to deliver a packet processing service to flows.

The setup in can be split across three FEs instead of as demonstrated in . This could be motivated by scale-out reasons or because different vendors provide different functionality, which is plugged-in to provide such functionality. The end result is having the same packet service delivered to the different flows passing through.

| +------------->| | | 1 | + NF1 | | + NF1/2 |NF3 | +----------+ metadata | | metadata | | ^ +----+ | | | +--+--+ | V ]]>

We address the inter-FE connectivity requirements by defining the inter-FE LFB class. Using a standard LFB class definition implies no change to the basic ForCES architecture in the form of the core LFBs (FE Protocol or Object LFBs). This design choice was made after considering an alternative approach that would have required changes to both the FE Object capabilities (SupportedLFBs) and the LFBTopology component to describe the inter-FE connectivity capabilities as well as the runtime topology of the LFB instances.

The distributed LFB topology described in is re-illustrated in to show the topology location where the inter-FE LFB would fit in. As can be observed in , the same details passed between IPv4 unicast LPM LFB and the IPv4 NH LFB are passed to the egress side of the inter-FE LFB. This information is illustrated as multiplicity of inputs into the egress inter-FE LFB instance. Each input represents a unique set of selection information.

| +------------->| | | | | | + ingress | | + ingress |IPv4 | | | +----------+ metadata | | metadata |Ucast| | | ^ +----+ |LPM | | | | IPv4 +--+--+ | | | Validator | | | | LFB | | | | IPv4 pkt + metadata | | | {ingress + NHinfo} | | | | | | | +..--+..+ | | | |..| | | | | +-V--V-V--V-+ | | | Egress | | | | Inter-FE | | | | LFB | | | +------+----+ | +---------------------------------------------------|---------+ | Ethernet Frame with: | IPv4 packet data and metadata {ingress + NHinfo + Inter-FE info} FE2 | +---------------------------------------------------|---------+ | +..+.+..+ | | |..|.|..| | | +-V--V-V--V-+ | | | Ingress | | | | Inter-FE | | | | LFB | | | +----+------+ | | | | | IPv4 pkt + metadata | | {ingress + NHinfo} | | | | | +--------+ +----V---+ | | | Egress | IPv4 packet | IPv4 | | | <-----+ LFB |<----------------------+NextHop | | | | |{ingress + NHdetails} | LFB | | | +--------+ metadata +--------+ | +-------------------------------------------------------------+ ]]>

The egress of the inter-FE LFB uses the received packet and metadata to select details for encapsulation when sending messages towards the selected neighboring FE. These details include what to communicate as the source and destination FEs (abstracted as MAC addresses as described in ); in addition, the original metadata may be passed along with the original IPv4 packet. On the ingress side of the inter-FE LFB, the received packet and its associated metadata are used to decide the packet graph continuation. This includes which of the original metadata and on which next LFB class instance to continue processing. In , an IPv4NextHop LFB instance is selected and the appropriate metadata is passed to it. The ingress side of the inter-FE LFB consumes some of the information passed and passes it the IPv4 packet alongside with the ingress and NHinfo metadata to the IPv4NextHop LFB as was done earlier in both Figures and .

describes some of the issues related to using Ethernet as the transport and how we mitigate them. defines a payload format that is to be used over Ethernet. An existing implementation of this specification that runs on top of Linux Traffic Control is described in .

There are several issues that may occur due to using direct Ethernet encapsulation that need consideration.

Because we are adding data to existing Ethernet frames, MTU issues may arise. We recommend: Using large MTUs when possible (example with jumbo frames). Limiting the amount of metadata that could be transmitted; our definition allows for filtering of select metadata to be encapsulated in the frame as described in . We recommend sizing the egress port MTU so as to allow space for maximum size of the metadata total size to allow between FEs. In such a setup, the port is configured to "lie" to the upper layers by claiming to have a lower MTU than it is capable of. Setting the MTU can be achieved by ForCES control of the port LFB (or some other configuration. In essence, the control plane when explicitly making a decision for the MTU settings of the egress port is implicitly deciding how much metadata will be allowed. Caution needs to be exercised on how low the resulting reported link MTU could be: for IPv4 packets, the minimum size is 64 octets and for IPv6 the minimum size is 1280 octets .

A raw packet arriving at the inter-FE LFB (from upstream LFB class instances) may have Class-of-Service (CoS) metadata indicating how it should be treated from a Quality-of-Service perspective. The resulting Ethernet frame will be eventually (preferentially) treated by a downstream LFB (typically a port LFB instance) and their CoS marks will be honored in terms of priority. In other words, the presence of the inter-FE LFB does not change the CoS semantics.

Most of the traffic passing through FEs that utilize the inter-FE LFB is expected to be IP based, which is generally assumed to be congestion controlled . For example, if congestion causes a TCP packet annotated with additional ForCES metadata to be dropped between FEs, the sending TCP can be expected to react in the same fashion as if that packet had been dropped at a different point on its path where ForCES is not involved. For this reason, additional inter-FE congestion-control mechanisms are not specified. However, the increased packet size due to the addition of ForCES metadata is likely to require additional bandwidth on inter-FE links in comparison to what would be required to carry the same traffic without ForCES metadata. Therefore, traffic engineering SHOULD be done when deploying inter-FE encapsulation. Furthermore, the inter-FE LFB MUST only be deployed within a single network (with a single network operator) or networks of an adjacent set of cooperating network operators where traffic is managed to avoid congestion. These are Controlled Environments, as defined by Section 3.6 of . Additional measures SHOULD be imposed to restrict the impact of inter-FE-encapsulated traffic on other traffic; for example: rate-limiting all inter-FE LFB traffic at an upstream LFB managing circuit breaking Isolating the inter-FE traffic either via dedicated interfaces or VLANs

The Ethernet wire encapsulation is illustrated in . The process that leads to this encapsulation is described in . The resulting frame is 32-bit aligned.

The Ethernet header (illustrated in ) has the following semantics: The Destination MAC Address is used to identify the Destination FEID by the CE policy (as described in ). The Source MAC Address is used to identify the Source FEID by the CE policy (as described in ). The ethertype is used to identify the frame as inter-FE LFB type. Ethertype TBA1 is to be used. The 16-bit metadata length is used to describe the total encoded metadata length (including the 16 bits used to encode the metadata length). One or more 16-bit TLV-encoded metadatum follows the Metadata length field. The TLV type identifies the metadata ID. ForCES metadata IDs that have been registered with IANA will be used. All TLVs will be 32-bit-aligned. We recognize that using a 16-bit TLV restricts the metadata ID to 16 bits instead of a ForCES-defined component ID space of 32 bits if an Index-Length-Value (ILV) is used. However, at the time of publication, we believe this is sufficient to carry all the information we need; the TLV approach has been selected because it saves us 4 bytes per metadatum transferred as compared to the ILV approach. The original packet data payload is appended at the end of the metadata as shown.

The Ethernet inter-FE LFB has two LFB input port groups and three LFB output ports as shown in . The inter-FE LFB defines two components used in aiding processing described in .

Decapsulated Packet -------------->|IngressInGroup | + metadata Ethernet Frame | | | | raw Packet + | OUT1+--> Encapsulated Ethernet -------------->|EgressInGroup | Frame Metadata | | | EXCEPTIONOUT +--> ExceptionID, packet | | + metadata +-----------------+ ]]>

The inter-FE LFB (instance) can be positioned at the egress of a source FE. illustrates an example source FE in the form of FE1. In such a case, an inter-FE LFB instance receives, via port group EgressInGroup, a raw packet and associated metadata from the preceding LFB instances. The input information is used to produce a selection of how to generate and encapsulate the new frame. The set of all selections is stored in the LFB component IFETable described further below. The processed encapsulated Ethernet frame will go out on OUT1 to a downstream LFB instance when processing succeeds or to the EXCEPTIONOUT port in the case of failure. The inter-FE LFB (instance) can be positioned at the ingress of a receiving FE. illustrates an example destination FE in the form of FE1. In such a case, an inter-FE LFB receives, via an LFB port in the IngressInGroup, an encapsulated Ethernet frame. Successful processing of the packet will result in a raw packet with associated metadata IDs going downstream to an LFB connected on OUT2. On failure, the data is sent out EXCEPTIONOUT.

The egress inter-FE LFB receives packet data and any accompanying metadatum at an LFB port of the LFB instance's input port group labeled EgressInGroup. The LFB implementation may use the incoming LFB port (within the LFB port group EgressInGroup) to map to a table index used to look up the IFETable table. If the lookup is successful, a matched table row that has the IFEInfo details is retrieved with the tuple (optional IFETYPE, optional StatId, Destination MAC address (DSTFE), Source MAC address (SRCFE), and optional metafilters). The metafilters lists define a whitelist of which metadatum are to be passed to the neighboring FE. The inter-FE LFB will perform the following actions using the resulting tuple: Increment statistics for packet and byte count observed at the corresponding IFEStats entry. When the MetaFilterList is present, walk each received metadatum and apply it against the MetaFilterList. If no legitimate metadata is found that needs to be passed downstream, then the processing stops and the packet and metadata are sent out the EXCEPTIONOUT port with the exceptionID of EncapTableLookupFailed . Check that the additional overhead of the Ethernet header and encapsulated metadata will not exceed MTU. If it does, increment the error-packet-count statistics and send the packet and metadata out the EXCEPTIONOUT port with the exceptionID of FragRequired . Create the Ethernet header. Set the Destination MAC address of the Ethernet header with the value found in the DSTFE field. Set the Source MAC address of the Ethernet header with the value found in the SRCFE field. If the optional IFETYPE is present, set the ethertype to the value found in IFETYPE. If IFETYPE is absent, then the standard inter-FE LFB ethertype TBA1 is used. Encapsulate each allowed metadatum in a TLV. Use the metaID as the "type" field in the TLV header. The TLV should be aligned to 32 bits. This means you may need to add a padding of zeroes at the end of the TLV to ensure alignment. Update the metadata length to the sum of each TLV's space plus 2 bytes (a 16-bit space for the Metadata length field). The resulting packet is sent to the next LFB instance connected to the OUT1 LFB-port, typically a port LFB. In the case of a failed lookup, the original packet and associated metadata is sent out the EXCEPTIONOUT port with the exceptionID of EncapTableLookupFailed . Note that the EXCEPTIONOUT LFB port is merely an abstraction and implementation may in fact drop packets as described above.

An ingressing inter-FE LFB packet is recognized by inspecting the ethertype, and optionally the destination and source MAC addresses. A matching packet is mapped to an LFB instance port in the IngressInGroup. The IFETable table row entry matching the LFB instance port may have optionally programmed metadata filters. In such a case, the ingress processing should use the metadata filters as a whitelist of what metadatum is to be allowed. Increment statistics for packet and byte count observed. Look at the metadata length field and walk the packet data, extracting the metadata values from the TLVs. For each metadatum extracted, in the presence of metadata filters, the metaID is compared against the relevant IFETable row metafilter list. If the metadatum is recognized and allowed by the filter, the corresponding implementation Metadatum field is set. If an unknown metadatum ID is encountered or if the metaID is not in the allowed filter list, then the implementation is expected to ignore it, increment the packet error statistic, and proceed processing other metadatum. Upon completion of processing all the metadata, the inter-FE LFB instance resets the data point to the original payload (i.e., skips the IFE header information). At this point, the original packet that was passed to the egress inter-FE LFB at the source FE is reconstructed. This data is then passed along with the reconstructed metadata downstream to the next LFB instance in the graph. In the case of a processing failure of either ingress or egress positioning of the LFB, the packet and metadata are sent out the EXCEPTIONOUT LFB port with the appropriate error ID. Note that the EXCEPTIONOUT LFB port is merely an abstraction and implementation may in fact drop packets as described above.

There are two LFB components accessed by the CE. The reader is asked to refer to the definitions in . The first component, populated by the CE, is an array known as the "IFETable" table. The array rows are made up of IFEInfo structure. The IFEInfo structure constitutes the optional IFETYPE, the optionally present StatId, the Destination MAC address (DSTFE), the Source MAC address (SRCFE), and an optionally present array of allowed metaIDs (MetaFilterList). The second component (ID 2), populated by the FE and read by the CE, is an indexed array known as the "IFEStats" table. Each IFEStats row carries statistics information in the structure bstats. A note about the StatId relationship between the IFETable table and the IFEStats table -- an implementation may choose to map between an IFETable row and IFEStats table row using the StatId entry in the matching IFETable row. In that case, the IFETable StatId must be present. An alternative implementation may map an IFETable row to an IFEStats table row at provisioning time. Yet another alternative implementation may choose not to use the IFETable row StatId and instead use the IFETable row index as the IFEStats index. For these reasons, the StatId component is optional.

PacketAny Arbitrary Packet InterFEFrame Ethernet frame with encapsulated IFE information bstats Basic stats bytes The total number of bytes seen uint64 packets The total number of packets seen uint32 errors The total number of packets with errors uint32 IFEInfo Describing IFE table row Information IFETYPE The ethertype to be used for outgoing IFE frame uint16 StatId The Index into the stats table uint32 DSTFE The destination MAC address of the destination FE byte[6] SRCFE The source MAC address used for the source FE byte[6] MetaFilterList The allowed metadata filter table uint32 IFE This LFB describes IFE connectivity parameterization 1.0 EgressInGroup The input port group of the egress side. It expects any type of Ethernet frame. PacketAny IngressInGroup The input port group of the ingress side. It expects an interFE-encapsulated Ethernet frame. InterFEFrame OUT1 The output port of the egress side InterFEFrame OUT2 The output port of the Ingress side PacketAny EXCEPTIONOUT The exception handling path PacketAny ExceptionID IFETable The table of all inter-FE relations IFEInfo IFEStats The stats corresponding to the IFETable table bstats ]]>

IANA has registered the following LFB class name in the the "Logical Functional Block (LFB) Class Names and Class Identifiers" subregistry of the "Forwarding and Control Element Separation (ForCES)" registry <https://www.iana.org/assignments/forces>. LFB Class Identifier LFB Class Name LFB Version Description Reference 18 IFE 1.0 An IFE LFB to standardize inter-FE LFB for ForCES Network Elements This document

This memo includes a request for a new Ethernet protocol type as described in .

The FEs involved in the inter-FE LFB belong to the same NE and are within the scope of a single administrative Ethernet LAN private network. While trust of policy in the control and its treatment in the datapath exists already, an inter-FE LFB implementation SHOULD support security services provided by Media Access Control Security (MACsec) . MACsec is not currently sufficiently widely deployed in traditional packet processing hardware although it is present in newer versions of the Linux kernel (which will be widely deployed) . Over time, we expect that most FEs will be able to support MACsec. MACsec provides security services such as a message authentication service and an optional confidentiality service. The services can be configured manually or automatically using the MACsec Key Agreement (MKA) over the IEEE 802.1x Extensible Authentication Protocol (EAP) framework. It is expected that FE implementations are going to start with shared keys configured from the control plane but progress to automated key management. The following are the MACsec security mechanisms that need to be in place for the inter-FE LFB: Security mechanisms are NE-wide for all FEs. Once the security is turned on, depending upon the chosen security level (e.g., Authentication, Confidentiality), it will be in effect for the inter-FE LFB for the entire duration of the session. An operator SHOULD configure the same security policies for all participating FEs in the NE cluster. This will ensure uniform operations and avoid unnecessary complexity in policy configuration. In other words, the Security Association Keys (SAKs) should be pre-shared. When using MKA, FEs must identify themselves with a shared Connectivity Association Key (CAK) and Connectivity Association Key Name (CKN). EAP-TLS SHOULD be used as the EAP method. An operator SHOULD configure the strict validation mode, i.e., all non-protected, invalid, or non-verifiable frames MUST be dropped. It should be noted that given the above choices, if an FE is compromised, an entity running on the FE would be able to fake inter-FE or modify its content, causing bad outcomes.

&RFC2119; &RFC5810; &RFC5811; &RFC5812; &RFC7391; &RFC7408; IEEE IEEE &RFC6956; &RFC3746; Broadcom &RFC2460;

The authors would like to thank Joel Halpern and Dave Hood for the stimulating discussions. Evangelos Haleplidis shepherded and contributed to improving this document. Alia Atlas was the AD sponsor of this document and did a tremendous job of critiquing it. The authors are grateful to Joel Halpern and Sue Hares in their roles as the Routing Area reviewers for shaping the content of this document. David Black put in a lot of effort to make sure the congestion-control considerations are sane. Russ Housley did the Gen-ART review, Joe Touch did the TSV area review, and Shucheng LIU (Will) did the OPS review. Suresh Krishnan helped us provide clarity during the IESG review. The authors are appreciative of the efforts Stephen Farrell put in to fixing the security section.