]> University of Patras

Department of Electrical and Computer Engineering Patras 26500 Greece ehalep@ece.upatras.gr

EICT GmbH

Torgauer Strasse 12-15 10829 Berlin Germany k.pentikousis@eict.de

University of Patras

Department of Electrical and Computer Engineering Patras 26500 Greece sdena@upatras.gr

Mojatatu Networks

Suite 400, 303 Moodie Dr. Ottawa, Ontario K2H 9R4 Canada hadi@mojatatu.com

Brocade

dmm@1-4-5.net

University of Patras

Department of Electrical and Computer Engineering Patras 26500 Greece odysseas@ece.upatras.gr

Routing Software-Defined Networking Software-defined Networking SDN Programmable Networks Architecture Layer Terminology Software-Defined Networking (SDN) refers to a new approach for network programmability, that is, the capacity to initialize, control, change, and manage network behavior dynamically via open interfaces. SDN emphasizes the role of software in running networks through the introduction of an abstraction for the data forwarding plane and, by doing so, separates it from the control plane. This separation allows faster innovation cycles at both planes as experience has already shown. However, there is increasing confusion as to what exactly SDN is, what the layer structure is in an SDN architecture, and how layers interface with each other. This document, a product of the IRTF Software-Defined Networking Research Group (SDNRG), addresses these questions and provides a concise reference for the SDN research community based on relevant peer-reviewed literature, the RFC series, and relevant documents by other standards organizations.

"Software-Defined Networking (SDN)" is a term of the programmable networks paradigm . In short, SDN refers to the ability of software applications to program individual network devices dynamically and therefore control the behavior of the network as a whole . Boucadair and Jacquenet point out that SDN is a set of techniques used to facilitate the design, delivery, and operation of network services in a deterministic, dynamic, and scalable manner. A key element in SDN is the introduction of an abstraction between the (traditional) forwarding and control planes in order to separate them and provide applications with the means necessary to programmatically control the network. The goal is to leverage this separation, and the associated programmability, in order to reduce complexity and enable faster innovation at both planes . The historical evolution of the research and development area of programmable networks is reviewed in detail in , starting with efforts dating back to the 1980s. As documented in , many of the ideas, concepts, and concerns are applicable to the latest research and development in SDN (and SDN standardization) and have been under extensive investigation and discussion in the research community for quite some time. For example, Rooney, et al.  discuss how to allow third-party access to the network without jeopardizing network integrity or how to accommodate legacy networking solutions in their (then new) programmable environment. Further, the concept of separating the control and forwarding planes, which is prominent in SDN, has been extensively discussed even prior to 1998 in SS7 networks, Ipsilon Flow Switching , and ATM. SDN research often focuses on varying aspects of programmability, and we are frequently confronted with conflicting points of view regarding what exactly SDN is. For instance, we find that for various reasons (e.g., work focusing on one domain and therefore not necessarily applicable as-is to other domains), certain well-accepted definitions do not correlate well with each other. For example, both OpenFlow and the Network Configuration Protocol (NETCONF) have been characterized as SDN interfaces, but they refer to control and management, respectively. This motivates us to consolidate the definitions of SDN in the literature and correlate them with earlier work at the IETF and the research community. Of particular interest is, for example, to determine which layers comprise the SDN architecture and which interfaces and their corresponding attributes are best suited to be used between them. As such, the aim of this document is not to standardize any particular layer or interface but rather to provide a concise reference that reflects current approaches regarding the SDN layer architecture. We expect that this document would be useful to upcoming work in SDNRG as well as future discussions within the SDN community as a whole. This document addresses the work item in the SDNRG charter titled "Survey of SDN approaches and Taxonomies", fostering better understanding of prominent SDN technologies in a technology-impartial and business-agnostic manner but does not constitute a new IETF standard. It is meant as a common base for further discussion. As such, we do not make any value statements nor discuss the applicability of any of the frameworks examined in this document for any particular purpose. Instead, we document their characteristics and attributes and classify them, thus providing a taxonomy. This document does not intend to provide an exhaustive list of SDN research issues; interested readers should consider reviewing and . In particular, Jarraya, et al.  provide an overview of SDN-related research topics, e.g., control partitioning, which is related to the Consistency, Availability and Partitioning (CAP) theorem discussed in . This document has been extensively reviewed, discussed, and commented by the vast majority of SDNRG members, a community that certainly exceeds 100 individuals. It is the consensus of SDNRG that this document should be published in the IRTF stream of the RFC series. The remainder of this document is organized as follows. explains the terminology used in this document. introduces a high-level overview of current SDN architecture abstractions. Finally, discusses how the SDN layer architecture relates to prominent SDN-enabling technologies.

This document uses the following terms: Software-Defined Networking (SDN) - A programmable networks approach that supports the separation of control and forwarding planes via standardized interfaces. Resource - A physical or virtual component available within a system. Resources can be very simple or fine-grained (e.g., a port or a queue) or complex, comprised of multiple resources (e.g., a network device). Network Device - A device that performs one or more network operations related to packet manipulation and forwarding. This reference model makes no distinction whether a network device is physical or virtual. A device can also be considered as a container for resources and can be a resource in itself. Interface - A point of interaction between two entities. When the entities are placed at different locations, the interface is usually implemented through a network protocol. If the entities are collocated in the same physical location, the interface can be implemented using a software application programming interface (API), inter-process communication (IPC), or a network protocol. Application (App) - An application in the context of SDN is a piece of software that utilizes underlying services to perform a function. Application operation can be parameterized, for example, by passing certain arguments at call time, but it is meant to be a standalone piece of software; an App does not offer any interfaces to other applications or services. Service - A piece of software that performs one or more functions and provides one or more APIs to applications or other services of the same or different layers to make use of said functions and returns one or more results. Services can be combined with other services, or called in a certain serialized manner, to create a new service. Forwarding Plane (FP) - The collection of resources across all network devices responsible for forwarding traffic. Operational Plane (OP) - The collection of resources responsible for managing the overall operation of individual network devices. Control Plane (CP) - The collection of functions responsible for controlling one or more network devices. CP instructs network devices with respect to how to process and forward packets. The control plane interacts primarily with the forwarding plane and, to a lesser extent, with the operational plane. Management Plane (MP) - The collection of functions responsible for monitoring, configuring, and maintaining one or more network devices or parts of network devices. The management plane is mostly related to the operational plane (it is related less to the forwarding plane). Application Plane - The collection of applications and services that program network behavior. Device and resource Abstraction Layer (DAL) - The device's resource abstraction layer based on one or more models. If it is a physical device, it may be referred to as the Hardware Abstraction Layer (HAL). DAL provides a uniform point of reference for the device's forwarding- and operational-plane resources. Control Abstraction Layer (CAL) - The control plane's abstraction layer. CAL provides access to the Control-Plane Southbound Interface. Management Abstraction Layer (MAL) - The management plane's abstraction layer. MAL provides access to the Management-Plane Southbound Interface. Network Services Abstraction Layer (NSAL) - Provides service abstractions that can be used by applications and services.

summarizes the SDN architecture abstractions in the form of a detailed, high-level schematic. Note that in a particular implementation, planes can be collocated with other planes or can be physically separated, as we discuss below. SDN is based on the concept of separation between a controlled entity and a controller entity. The controller manipulates the controlled entity via an interface. Interfaces, when local, are mostly API invocations through some library or system call. However, such interfaces may be extended via some protocol definition, which may use local inter-process communication (IPC) or a protocol that could also act remotely; the protocol may be defined as an open standard or in a proprietary manner. Day explores the use of IPC as the mainstay for the definition of recursive network architectures with varying degrees of scope and range of operation. The Recursive InterNetwork Architecture outlines a recursive network architecture based on IPC that capitalizes on repeating patterns and structures. This document does not propose a new architecture -- we simply document previous work through a taxonomy. Although recursion is out of the scope of this work, illustrates a hierarchical model in which layers can be stacked on top of each other and employed recursively as needed.

This document follows a network-device-centric approach: control mostly refers to the device packet-handling capability, while management typically refers to aspects of the overall device operation. We view a network device as a complex resource that contains and is part of multiple resources similar to . Resources can be simple, single components of a network device, for example, a port or a queue of the device, and can also be aggregated into complex resources, for example, a network card or a complete network device. The reader should keep in mind that we make no distinction between "physical" and "virtual" resources or "hardware" and "software" realizations in this document, as we do not delve into implementation or performance aspects. In other words, a resource can be implemented fully in hardware, fully in software, or any hybrid combination in between. Further, we do not distinguish whether a resource is implemented as an overlay or as a part/component of some other device. In general, network device software can run on so-called "bare metal" or on a virtualized substrate. Finally, this document does not discuss how resources are allocated, orchestrated, and released. Indeed, orchestration is out of the scope of this document. SDN spans multiple planes as illustrated in . Starting from the bottom part of the figure and moving towards the upper part, we identify the following planes: Forwarding Plane - Responsible for handling packets in the data path based on the instructions received from the control plane. Actions of the forwarding plane include, but are not limited to, forwarding, dropping, and changing packets. The forwarding plane is usually the termination point for control-plane services and applications. The forwarding plane can contain forwarding resources such as classifiers. The forwarding plane is also widely referred to as the "data plane" or the "data path". Operational Plane - Responsible for managing the operational state of the network device, e.g., whether the device is active or inactive, the number of ports available, the status of each port, and so on. The operational plane is usually the termination point for management-plane services and applications. The operational plane relates to network device resources such as ports, memory, and so on. We note that some participants of the IRTF SDNRG have a different opinion in regards to the definition of the operational plane. That is, one can argue that the operational plane does not constitute a "plane" per se, but it is, in practice, an amalgamation of functions on the forwarding plane. For others, however, a "plane" allows one to distinguish between different areas of operations; therefore, the operational plane is included as a "plane" in . We have adopted this latter view in this document. Control Plane - Responsible for making decisions on how packets should be forwarded by one or more network devices and pushing such decisions down to the network devices for execution. The control plane usually focuses mostly on the forwarding plane and less on the operational plane of the device. The control plane may be interested in operational-plane information, which could include, for instance, the current state of a particular port or its capabilities. The control plane's main job is to fine-tune the forwarding tables that reside in the forwarding plane, based on the network topology or external service requests. Management Plane - Responsible for monitoring, configuring, and maintaining network devices, e.g., making decisions regarding the state of a network device. The management plane usually focuses mostly on the operational plane of the device and less on the forwarding plane. The management plane may be used to configure the forwarding plane, but it does so infrequently and through a more wholesale approach than the control plane. For instance, the management plane may set up all or part of the forwarding rules at once, although such action would be expected to be taken sparingly. Application Plane - The plane where applications and services that define network behavior reside. Applications that directly (or primarily) support the operation of the forwarding plane (such as routing processes within the control plane) are not considered part of the application plane. Note that applications may be implemented in a modular and distributed fashion and, therefore, can often span multiple planes in . has defined the data, control, and management planes in terms of Operations, Administration, and Maintenance (OAM). This document attempts to broaden the terms defined in in order to reflect all aspects of an SDN architecture. All planes mentioned above are connected via interfaces (indicated with "Y" in . An interface may take multiple roles depending on whether the connected planes reside on the same (physical or virtual) device. If the respective planes are designed so that they do not have to reside in the same device, then the interface can only take the form of a protocol. If the planes are collocated on the same device, then the interface could be implemented via an open/proprietary protocol, an open/proprietary software inter-process communication API, or operating system kernel system calls. Applications, i.e., software programs that perform specific computations that consume services without providing access to other applications, can be implemented natively inside a plane or can span multiple planes. For instance, applications or services can span both the control and management planes and thus be able to use both the Control-Plane Southbound Interface (CPSI) and Management-Plane Southbound Interface (MPSI), although this is only implicitly illustrated in . An example of such a case would be an application that uses both and . Services, i.e., software programs that provide APIs to other applications or services, can also be natively implemented in specific planes. Services that span multiple planes belong to the application plane as well. While not shown explicitly in , services, applications, and entire planes can be placed in a recursive manner, thus providing overlay semantics to the model. For example, application-plane services can be provided to other applications or services through NSAL. Additional examples include virtual resources that are realized on top of a physical resources and hierarchical control-plane controllers . Note that the focus in this document is, of course, on the north/south communication between entities in different planes. But this, clearly, does not exclude entity communication within any one plane. It must be noted, however, that in , we present an abstract view of the various planes, which is devoid of implementation details. Many implementations in the past have opted for placing the management plane on top of the control plane. This can be interpreted as having the control plane acting as a service to the management plane. Further, in many networks, especially in Internet routers and Ethernet switches, the control plane has been usually implemented as tightly coupled with the network device. When taken as a whole, the control plane has been distributed network-wide. On the other hand, the management plane has been traditionally centralized and has been responsible for managing the control plane and the devices. However, with the adoption of SDN principles, this distinction is no longer so clear-cut. Additionally, this document considers four abstraction layers: The Device and resource Abstraction Layer (DAL) abstracts the resources of the device's forwarding and operational planes to the control and management planes. Variations of DAL may abstract both planes or either of the two and may abstract any plane of the device to either the control or management plane. The Control Abstraction Layer (CAL) abstracts the Control-Plane Southbound Interface and the DAL from the applications and services of the control plane. The Management Abstraction Layer (MAL) abstracts the Management-Plane Southbound Interface and the DAL from the applications and services of the management plane. The Network Services Abstraction Layer (NSAL) provides service abstractions for use by applications and other services. At the time of this writing, SDN-related activities have begun in other SDOs. For example, at the ITU, work on architectural and signaling requirements and protocols has commenced, but the respective study groups have yet to publish their documents, with the exception of . The views presented in as well as in are well aligned with this document.

A network device is an entity that receives packets on its ports and performs one or more network functions on them. For example, the network device could forward a received packet, drop it, alter the packet header (or payload), forward the packet, and so on. A network device is an aggregation of multiple resources such as ports, CPU, memory, and queues. Resources are either simple or can be aggregated to form complex resources that can be viewed as one resource. The network device is in itself a complex resource. Examples of network devices include switches and routers. Additional examples include network elements that may operate at a layer above IP (such as firewalls, load balancers, and video transcoders) or below IP (such as Layer 2 switches and optical or microwave network elements). Network devices can be implemented in hardware or software and can be either physical or virtual. As has already been mentioned before, this document makes no such distinction. Each network device has a presence in a forwarding plane and an operational plane. The forwarding plane, commonly referred to as the "data path", is responsible for handling and forwarding packets. The forwarding plane provides switching, routing, packet transformation, and filtering functions. Resources of the forwarding plane include but are not limited to filters, meters, markers, and classifiers. The operational plane is responsible for the operational state of the network device, for instance, with respect to status of network ports and interfaces. Operational-plane resources include, but are not limited to, memory, CPU, ports, interfaces, and queues. The forwarding and the operational planes are exposed via the Device and resource Abstraction Layer (DAL), which may be expressed by one or more abstraction models. Examples of forwarding-plane abstraction models are Forwarding and Control Element Separation (ForCES), OpenFlow, YANG model, and SNMP MIBs. Examples of the operational-plane abstraction model include the ForCES model, the YANG model, and SNMP MIBs. Note that applications can also reside in a network device. Examples of such applications include event monitoring and handling (offloading) topology discovery or ARP in the device itself instead of forwarding such traffic to the control plane.

The control plane is usually distributed and is responsible mainly for the configuration of the forwarding plane using a Control-Plane Southbound Interface (CPSI) with DAL as a point of reference. CP is responsible for instructing FP about how to handle network packets. Communication between control-plane entities, colloquially referred to as the "east-west" interface, is usually implemented through gateway protocols such as BGP or other protocols such as the Path Computation Element (PCE) Communication Protocol (PCEP). These corresponding protocol messages are usually exchanged in-band and subsequently redirected by the forwarding plane to the control plane for further processing. Examples in this category include , , and . Control-plane functionalities usually include: Topology discovery and maintenance Packet route selection and instantiation Path failover mechanisms The CPSI is usually defined with the following characteristics: time-critical interface that requires low latency and sometimes high bandwidth in order to perform many operations in short order oriented towards wire efficiency and device representation instead of human readability Examples include fast- and high-frequency of flow or table updates, high throughput, and robustness for packet handling and events. CPSI can be implemented using a protocol, an API, or even inter-process communication. If the control plane and the network device are not collocated, then this interface is certainly a protocol. Examples of CPSIs are ForCES and the OpenFlow protocol. The Control Abstraction Layer (CAL) provides access to control applications and services to various CPSIs. The control plane may support more than one CPSI. Control applications can use CAL to control a network device without providing any service to upper layers. Examples include applications that perform control functions, such as OSPF, IS-IS, and BGP. Control-plane service examples include a virtual private LAN service, service tunnels, topology services, etc.

The management plane is usually centralized and aims to ensure that the network as a whole is running optimally by communicating with the network devices' operational plane using a Management-Plane Southbound Interface (MPSI) with DAL as a point of reference. Management-plane functionalities are typically initiated, based on an overall network view, and traditionally have been human-centric. However, lately, algorithms are replacing most human intervention. Management-plane functionalities typically include: Fault and monitoring management Configuration management In addition, management-plane functionalities may also include entities such as orchestrators, Virtual Network Function Managers (VNF Managers) and Virtualised Infrastructure Managers, as described in . Such entities can use management interfaces to operational-plane resources to request and provision resources for virtual functions as well as instruct the instantiation of virtual forwarding functions on top of physical forwarding functions. The possibility of a common abstraction model for both SDN and Network Function Virtualization (NFV) is explored in . Note, however, that these are only examples of applications and services in the management plane and not formal definitions of entities in this document. As has been noted above, orchestration and therefore the definition of any associated entities is out of the scope of this document. The MPSI, in contrast to the CPSI, is usually not a time-critical interface and does not share the CPSI requirements. MPSI is typically closer to human interaction than CPSI (cf. ); therefore, MPSI usually has the following characteristics: It is oriented more towards usability, with optimal wire performance being a secondary concern. Messages tend to be less frequent than in the CPSI. As an example of usability versus performance, we refer to the consensus of the 2002 IAB Workshop : the key requirement for a network management technology is ease of use, not performance. As per , textual configuration files should be able to contain international characters. Human-readable strings should utilize UTF&nbhy;8, and protocol elements should be in case-insensitive ASCII, which requires more processing capabilities to parse. MPSI can range from a protocol, to an API or even inter-process communication. If the management plane is not embedded in the network device, the MPSI is certainly a protocol. Examples of MPSIs are ForCES, NETCONF, IP Flow Information Export (IPFIX), Syslog, Open vSwitch Database (OVSDB), and SNMP. The Management Abstraction Layer (MAL) provides access to management applications and services to various MPSIs. The management plane may support more than one MPSI. Management applications can use MAL to manage the network device without providing any service to upper layers. Examples of management applications include network monitoring, fault detection, and recovery applications. Management-plane services provide access to other services or applications above the management plane.

The definition of a clear distinction between "control" and "management" in the context of SDN received significant community attention during the preparation of this document. We observed that the role of the management plane has been earlier largely ignored or specified as out-of-scope for the SDN ecosystem. In the remainder of this subsection, we summarize the characteristics that differentiate the two planes in order to have a clear understanding of the mechanics, capabilities, and needs of each respective interface.

A point has been raised regarding the reference timescales for the control and management planes regarding how fast the respective plane is required to react to, or how fast it needs to manipulate, the forwarding or operational plane of the device. In general, the control plane needs to send updates "often", which translates roughly to a range of milliseconds; that requires high-bandwidth and low-latency links. In contrast, the management plane reacts generally at longer time frames, i.e., minutes, hours, or even days; thus, wire efficiency is not always a critical concern. A good example of this is the case of changing the configuration state of the device.

Another distinction between the control and management planes relates to state persistence. A state is considered ephemeral if it has a very limited lifespan and is not deemed necessary to be stored on non-volatile memory. A good example is determining routing, which is usually associated with the control plane. On the other hand, a persistent state has an extended lifespan that may range from hours to days and months, is meant to be used beyond the lifetime of the process that created it, and is thus used across device reboots. Persistent state is usually associated with the management plane.

As mentioned earlier, traditionally, the control plane has been executed locally on the network device and is distributed in nature whilst the management plane is usually executed in a centralized manner, remotely from the device. However, with the advent of SDN centralizing, or "logically centralizing", the controller tends to muddle the distinction of the control and management plane based on locality.

The CAP theorem views a distributed computing system as composed of multiple computational resources (i.e., CPU, memory, storage) that are connected via a communications network and together perform a task. The theorem, or conjecture by some, identifies three characteristics of distributed systems that are universally desirable: Consistency, meaning that the system responds identically to a query no matter which node receives the request (or does not respond at all). Availability, i.e., that the system always responds to a request (although the response may not be consistent or correct). Partition tolerance, namely that the system continues to function even when specific nodes or the communications network fail. In 2000, Eric Brewer conjectured that a distributed system can satisfy any two of these guarantees at the same time but not all three. This conjecture was later proven by Gilbert and Lynch and is now usually referred to as the CAP theorem . Forwarding a packet through a network correctly is a computational problem. One of the major abstractions that SDN posits is that all network elements are computational resources that perform the simple computational task of inspecting fields in an incoming packet and deciding how to forward it. Since the task of forwarding a packet from network ingress to network egress is obviously carried out by a large number of forwarding elements, the network of forwarding devices is a distributed computational system. Hence, the CAP theorem applies to forwarding of packets. In the context of the CAP theorem, if one considers partition tolerance of paramount importance, traditional control-plane operations are usually local and fast (available), while management-plane operations are usually centralized (consistent) and may be slow. The CAP theorem also provides insights into SDN architectures. For example, a centralized SDN controller acts as a consistent global database and specific SDN mechanisms ensure that a packet entering the network is handled consistently by all SDN switches. The issue of tolerance to loss of connectivity to the controller is not addressed by the basic SDN model. When an SDN switch cannot reach its controller, the flow will be unavailable until the connection is restored. The use of multiple non-collocated SDN controllers has been proposed (e.g., by configuring the SDN switch with a list of controllers); this may improve partition tolerance but at the cost of loss of absolute consistency. Panda, et al.  provide a first exploration of how the CAP theorem applies to SDN.

The Network Services Abstraction Layer (NSAL) provides access from services of the control, management, and application planes to other services and applications. We note that the term "SAL" is overloaded, as it is often used in several contexts ranging from system design to service-oriented architectures; therefore, we explicitly add "Network" to the title of this layer to emphasize that this term relates to , and we map it accordingly in to prominent SDN approaches. Service interfaces can take many forms pertaining to their specific requirements. Examples of service interfaces include, but are not limited to, RESTful APIs, open protocols such as NETCONF, inter-process communication, CORBA interfaces, and so on. The two leading approaches for service interfaces are RESTful interfaces and Remote Procedure Call (RPC) interfaces. Both follow a client-server architecture and use XML or JSON to pass messages, but each has some slightly different characteristics. RESTful interfaces, designed according to the representational state transfer design paradigm, have the following characteristics: Resource identification - Individual resources are identified using a resource identifier, for example, a URI. Manipulation of resources through representations - Resources are represented in a format like JSON, XML, or HTML. Self-descriptive messages - Each message has enough information to describe how the message is to be processed. Hypermedia as the engine of application state - A client needs no prior knowledge of how to interact with a server, as the API is not fixed but dynamically provided by the server. Remote procedure calls (RPCs) , e.g., XML-RPC and the like, have the following characteristics: Individual procedures are identified using an identifier. A client needs to know the procedure name and the associated parameters.

Applications and services that use services from the control and/or management plane form the application plane. Additionally, services residing in the application plane may provide services to other services and applications that reside in the application plane via the service interface. Examples of applications include network topology discovery, network provisioning, path reservation, etc.

We advocate that the SDN southbound interface should encompass both CPSI and MPSI. SDN controllers such as and are a collection of control-plane applications and services that implement a CPSI ( and both use OpenFlow) and provide a northbound interface for applications. The SDN northbound interface for controllers is implemented in the Network Services Abstraction Layer (NSAL) of . The above model can be used to describe all prominent SDN-enabling technologies in a concise manner, as we explain in the following subsections.

The IETF Forwarding and Control Element Separation (ForCES) framework consists of one model and two protocols. ForCES separates the forwarding plane from the control plane via an open interface, namely the ForCES protocol, which operates on entities of the forwarding plane that have been modeled using the ForCES model. The ForCES model is based on the fact that a network element is composed of numerous logically separate entities that cooperate to provide a given functionality (such as routing or IP switching) and yet appear as a normal integrated network element to external entities. ForCES models the forwarding plane using Logical Functional Blocks (LFBs), which, when connected in a graph, compose the Forwarding Element (FE). LFBs are described in XML, based on an XML schema. LFB definitions include base and custom-defined datatypes; metadata definitions; input and output ports; operational parameters or components; and capabilities and event definitions. The ForCES model can be used to define LFBs from fine- to coarse-grained as needed, irrespective of whether they are physical or virtual. The ForCES protocol is agnostic to the model and can be used to monitor, configure, and control any ForCES-modeled element. The protocol has very simple commands: Set, Get, and Del(ete). The ForCES protocol has been designed for high throughput and fast updates. With respect to , the ForCES model is suitable for the DAL, both for the operational and the forwarding plane, using LFBs. The ForCES protocol has been designed and is suitable for the CPSI, although it could also be utilized for the MPSI.

The Network Configuration Protocol (NETCONF) is an IETF network management protocol . NETCONF provides mechanisms to install, manipulate, and delete the configuration of network devices. NETCONF protocol operations are realized as remote procedure calls (RPCs). The NETCONF protocol uses XML-based data encoding for the configuration data as well as the protocol messages. Recent studies, such as and , have shown that NETCONF performs better than SNMP . Additionally, the YANG data modeling language has been developed for specifying NETCONF data models and protocol operations. YANG is a data modeling language used to model configuration and state data manipulated by the NETCONF protocol, NETCONF remote procedure calls, and NETCONF notifications. YANG models the hierarchical organization of data as a tree, in which each node has either a value or a set of child nodes. Additionally, YANG structures data models into modules and submodules, allowing reusability and augmentation. YANG models can describe constraints to be enforced on the data. Additionally, YANG has a set of base datatypes and allows custom-defined datatypes as well. YANG allows the definition of NETCONF RPCs, which allows the protocol to have an extensible number of commands. For RPC definitions, the operations names, input parameters, and output parameters are defined using YANG data definition statements. With respect to , the YANG model is suitable for specifying DAL for the forwarding and operational planes. NETCONF is suitable for the MPSI. NETCONF is a management protocol, which was not (originally) designed for fast CP updates, and it might not be suitable for addressing the requirements of CPSI.

OpenFlow is a framework originally developed at Stanford University and currently under active standards development through the Open Networking Foundation (ONF). Initially, the goal was to provide a way for researchers to run experimental protocols in a production network . OpenFlow has undergone many revisions, and additional revisions are likely. The following description reflects version 1.4 . In short, OpenFlow defines a protocol through which a logically centralized controller can control an OpenFlow switch. Each OpenFlow-compliant switch maintains one or more flow tables, which are used to perform packet lookups. Distinct actions are to be taken regarding packet lookup and forwarding. A group table and an OpenFlow channel to external controllers are also part of the switch specification. With respect to , the OpenFlow switch specifications define a DAL for the forwarding plane as well as for CPSI. The OF-CONFIG protocol, based on the YANG model, provides a DAL for the forwarding and operational planes of an OpenFlow switch and specifies NETCONF as the MPSI. OF-CONFIG overlaps with the OpenFlow DAL, but with NETCONF as the transport protocol, it shares the limitations described in the previous section.

Interface to the Routing System (I2RS) provides a standard interface to the routing system for real-time or event-driven interaction through a collection of protocol-based control or management interfaces. Essentially, one of the main goals of I2RS, is to make the Routing Information Base (RIB) programmable, thus enabling new kinds of network provisioning and operation. I2RS did not initially intend to create new interfaces but rather leverage or extend existing ones and define informational models for the routing system. For example, the latest I2RS problem statement discusses previously defined IETF protocols such as ForCES, NETCONF, and SNMP. Regarding the definition of informational and data models, the I2RS working group has opted to use the YANG modeling language. Currently the I2RS working group is developing an Information Model in regards to the Network Services Abstraction Layer for the I2RS agent. With respect to , the I2RS architecture encompasses the control and application planes and uses any CPSI and DAL that is available, whether that may be ForCES, OpenFlow, or another interface. In addition, the I2RS agent is a control-plane service. All services or applications on top of that belong to either the Control, Management, or Application plane. In the I2RS documents, management access to the agent may be provided by management protocols like SNMP and NETCONF. The I2RS protocol may also be mapped to the service interface as it will even provide access to services and applications other than control-plane services and applications.

The Simple Network Management Protocol (SNMP) is an IETF-standardized management protocol and is currently at its third revision (SNMPv3) . It consists of a set of standards for network management, including an application-layer protocol, a database schema, and a set of data objects. SNMP exposes management data (managed objects) in the form of variables on the managed systems, which describe the system configuration. These variables can then be queried and set by managing applications. SNMP uses an extensible design for describing data, defined by Management Information Bases (MIBs). MIBs describe the structure of the management data of a device subsystem. MIBs use a hierarchical namespace containing object identifiers (OIDs). Each OID identifies a variable that can be read or set via SNMP. MIBs use the notation defined by Structure of Management Information Version 2. An early example of SNMP in the context of SDN is discussed in . With respect to , SNMP MIBs can be used to describe DAL for the forwarding and operational planes. Similar to YANG, SNMP MIBs are able to describe DAL for the forwarding plane. SNMP, similar to NETCONF, is suited for the MPSI.

The Path Computation Element (PCE) architecture defines an entity capable of computing paths for a single service or a set of services. A PCE might be a network node, network management station, or dedicated computational platform that is resource-aware and has the ability to consider multiple constraints for a variety of path computation problems and switching technologies. The PCE Communication Protocol (PCEP) is used between a Path Computation Client (PCC) and a PCE, or between multiple PCEs. The PCE architecture represents a vision of networks that separates path computation for services, the signaling of end-to-end connections, and actual packet forwarding. The definition of online and offline path computation is dependent on the reachability of the PCE from network and Network Management System (NMS) nodes and the type of optimization request that may significantly impact the optimization response time from the PCE to the PCC. The PCEP messaging mechanism facilitates the specification of computation endpoints (source and destination node addresses), objective functions (requested algorithm and optimization criteria), and the associated constraints such as traffic parameters (e.g., requested bandwidth), the switching capability, and encoding type. With respect to , PCE is a control-plane service that provides services for control-plane applications. PCEP may be used as an east-west interface between PCEs that may act as domain control entities (services and applications). The PCE working group is specifying extensions that allow an active PCE to control, using PCEP, MPLS or GMPLS Label Switched Paths (LSPs), thus making it applicable for the CPSI for MPLS and GMPLS switches.

Bidirectional Forwarding Detection (BFD) is an IETF-standardized network protocol designed for detecting path failures between two forwarding elements, including physical interfaces, subinterfaces, data link(s), and, to the extent possible, the forwarding engines themselves, with potentially very low latency. BFD can provide low-overhead failure detection on any kind of path between systems, including direct physical links, virtual circuits, tunnels, MPLS LSPs, multihop routed paths, and unidirectional links where there exists a return path as well. It is often implemented in some component of the forwarding engine of a system, in cases where the forwarding and control engines are separated. With respect to , a BFD agent can be implemented as a control-plane service or application that would use the CPSI towards the forwarding plane to send/receive BFD packets. However, a BFD agent is usually implemented as an application on the device and uses the forwarding plane to send/receive BFD packets and update the operational-plane resources accordingly. Services and applications of the control and management planes that monitor or have subscribed to changes of resources can learn about these changes through their respective interfaces and take any actions as necessary.

This document has been developed after a thorough and detailed analysis of related peer-reviewed literature, the RFC series, and documents produced by other relevant standards organizations. It has been reviewed publicly by the wider SDN community, and we hope that it can serve as a handy tool for network researchers, engineers, and practitioners in the years to come. We conclude this document with a brief summary of the terminology of the SDN layer architecture. In general, we consider a network element as a composition of resources. Each network element has a forwarding plane (FP) that is responsible for handling packets in the data path and an operational plane (OP) that is responsible for managing the operational state of the device. Resources in the network element are abstracted by the Device and resource Abstraction Layer (DAL) to be controlled and managed by services or applications that belong to the control or management plane. The control plane (CP) is responsible for making decisions on how packets should be forwarded. The management plane (MP) is responsible for monitoring, configuring, and maintaining network devices. Service interfaces are abstracted by the Network Services Abstraction Layer (NSAL), where other network applications or services may use them. The taxonomy introduced in this document defines distinct SDN planes, abstraction layers, and interfaces; it aims to clarify SDN terminology and establish commonly accepted reference definitions across the SDN community, irrespective of specific implementation choices.

This document does not propose a new network architecture or protocol and therefore does not have any impact on the security of the Internet. That said, security is paramount in networking; thus, it should be given full consideration when designing a network architecture or operational deployment. Security in SDN is discussed in the literature, for example, in , , and . Security considerations regarding specific interfaces (such as, for example, ForCES, I2RS, SNMP, or NETCONF) are addressed in their respective documents as well as in .

Open vSwitch is an open-source software switch designed to be used as a vswitch (virtual switch) in virtualized server environments. A vswitch forwards traffic between different virtual machines (VMs) on the same physical host and also forwards traffic between VMs and the physical network. Open vSwitch is open to programmatic extension and control using OpenFlow and the OVSDB (Open vSwitch Database) management protocol. This document defines the OVSDB management protocol. The Open vSwitch project includes open-source OVSDB client and server implementations. Software-Defined Networking (SDN) has been one of the major buzz words of the networking industry for the past couple of years. And yet, no clear definition of what SDN actually covers has been broadly admitted so far. This document aims to clarify the SDN landscape by providing a perspective on requirements, issues, and other considerations about SDN, as seen from within a service provider environment.</t><t> It is not meant to endlessly discuss what SDN truly means but rather to suggest a functional taxonomy of the techniques that can be used under an SDN umbrella and to elaborate on the various pending issues the combined activation of such techniques inevitably raises. As such, a definition of SDN is only mentioned for the sake of clarification. This document gives an overview of the IETF network management standards and summarizes existing and ongoing development of IETF Standards Track network management protocols and data models. The document refers to other overview documents, where they exist and classifies the standards for easy orientation. The purpose of this document is, on the one hand, to help system developers and users to select appropriate standard management protocols and data models to address relevant management needs. On the other hand, the document can be used as an overview and guideline by other Standard Development Organizations or bodies planning to use IETF management technologies and data models. This document does not cover Operations, Administration, and Maintenance (OAM) technologies on the data-path, e.g., OAM of tunnels, MPLS Transport Profile (MPLS-TP) OAM, and pseudowire as well as the corresponding management models. This document is not an Internet Standards Track specification; it is published for informational purposes. This document provides an overview of a workshop held by the Internet Architecture Board (IAB) on Network Management. The workshop was hosted by CNRI in Reston, VA, USA on June 4 thru June 6, 2002. The goal of the workshop was to continue the important dialog started between network operators and protocol developers, and to guide the IETFs focus on future work regarding network management. This report summarizes the discussions and lists the conclusions and recommendations to the Internet Engineering Task Force (IETF) community. This memo provides information for the Internet community. This memo defines the publication stream for RFCs from the Internet Research Task Force. Most documents undergoing this process will come from IRTF Research Groups, and it is expected that they will be published as Informational or Experimental RFCs by the RFC Editor. This document is not an Internet Standards Track specification; it is published for informational purposes. Symbolics, Inc.

243 Vassar Street Cambridge MA 02139 US DCP@MIT-MC

The implementation of protocol P on a sending host S decides, through protocol P's routing mechanism, that it wants to transmit to a target host T located some place on a connected piece of 10Mbit Ethernet cable. To actually transmit the Ethernet packet a 48.bit Ethernet address must be generated. The addresses of hosts within protocol P are not always compatible with the corresponding Ethernet address (being different lengths or values). Presented here is a protocol that allows dynamic distribution of the information needed to build tables to translate an address A in protocol P's address space into a 48.bit Ethernet address. Generalizations have been made which allow the protocol to be used for non-10Mbit Ethernet hardware. Some packet radio networks are examples of such hardware. The protocol proposed here is the result of a great deal of discussion with several other people, most notably J. Noel Chiappa, Yogen Dalal, and James E. Kulp, and helpful comments from David Moon. The purpose of this RFC is to present a method of Converting Protocol Addresses (e.g., IP addresses) to Local Network Addresses (e.g., Ethernet addresses). This is a issue of general concern in the ARPA Internet community at this time. The method proposed here is presented for your consideration and comment. This is not the specification of a Internet Standard. Ipsilon Networks, Inc.

2191 East Bayshore Road Suite 100 Palo Alto CA 94303 US +1 415 846 4603 pn@ipsilon.com

Sprint Technology Services - Long Distance Division

9300 Metcalf Avenue Mailstop KSOPKB0802 Overland Park KS 66212-6333 US +1 913 534 5334 texas@sprintcorp.com

Ipsilon Networks, Inc.

2191 East Bayshore Road Suite 100 Palo Alto CA 94303 US +1 415 846 4604 hinden@ipsilon.com

Ipsilon Networks, Inc.

2191 East Bayshore Road Suite 100 Palo Alto CA 94303 US +1 415 846 4610 hoffman@ipsilon.com

Ipsilon Networks, Inc.

2191 East Bayshore Road Suite 100 Palo Alto CA 94303 US +1 415 846 4607 fong@ipsilon.com

Ipsilon Networks, Inc.

2191 East Bayshore Road Suite 100 Palo Alto CA 94303 US +1 415 846 4601 pugs@ipsilon.com

Ipsilon Networks, Inc.

2191 East Bayshore Road Suite 100 Palo Alto CA 94303 US +1 415 846 4605 minshall@ipsilon.com

The Ipsilon Flow Management Protocol (IFMP), is a protocol for allowing a node to instruct an adjacent node to attach a layer 2 label to a specified IP flow. The label allows more efficient access to cached routing information for that flow. The label can also enable a node to switch further packets belonging to the specified flow at layer 2 rather than forwarding them at layer 3. Nokia

232 Java Drive Sunnyvale CA 94089 US +1 408 990 2003 pn@ipsilon.com

Sprint Technology Services - Long Distance Division

9300 Metcalf Avenue Mailstop KSOPKB0802 Overland Park KS 66212-6333 US +1 913 534 5334 texas@sprintcorp.com

Nokia

232 Java Drive Sunnyvale CA 94089 US +1 408 990 2004 hinden@ipsilon.com

Nokia

232 Java Drive Sunnyvale CA 94089 US +1 408 990 2010 hoffman@ipsilon.com

Coppercom

+1 408 873 2688 fong@coppercom.com

Nokia

232 Java Drive Sunnyvale CA 94089 US +1 408 990 2001 pugs@ipsilon.com

Fiberlane Communications

1399 Charleston Road Mountain View CA 94043 US +1 650 237 3164 minshall@fiberlane.com

This memo specifies enhancements to the General Switch Management Protocol (GSMP). The major enhancement is the addition of Quality of Service (QoS) messages. Other improvements have been made to the protocol resulting from operational experience. GSMP is a general purpose protocol to control an ATM switch. It allows a controller to establish and release connections across the switch; add and delete leaves on a multicast connection; manage switch ports; request configuration information; and request statistics. Cisco Systems, Inc.

170 West Tasman Drive San Jose CA 95134-1706 US +1 408 526 5260 kzm@cisco.com

SNMPinfo

3763 Benton Street Santa Clara CA 95051 US +1 408 221 8702 dperkins@snmpinfo.com

TU Braunschweig

Bueltenweg 74/75 38106 Braunschweig DE +49 531 3913283 schoenw@ibr.cs.tu-bs.de

Constraint-based path computation is a fundamental building block for traffic engineering systems such as Multiprotocol Label Switching (MPLS) and Generalized Multiprotocol Label Switching (GMPLS) networks. Path computation in large, multi-domain, multi-region, or multi-layer networks is complex and may require special computational components and cooperation between the different network domains.</t><t> This document specifies the architecture for a Path Computation Element (PCE)-based model to address this problem space. This document does not attempt to provide a detailed description of all the architectural components, but rather it describes a set of building blocks for the PCE architecture from which solutions may be constructed. This memo provides information for the Internet community. Operations, Administration, and Maintenance (OAM) is a general term that refers to a toolset for fault detection and isolation, and for performance measurement. Over the years, various OAM tools have been defined for various layers in the protocol stack.</t><t> This document summarizes some of the OAM tools defined in the IETF in the context of IP unicast, MPLS, MPLS Transport Profile (MPLS-TP), pseudowires, and Transparent Interconnection of Lots of Links (TRILL). This document focuses on tools for detecting and isolating failures in networks and for performance monitoring. Control and management aspects of OAM are outside the scope of this document. Network repair functions such as Fast Reroute (FRR) and protection switching, which are often triggered by OAM protocols, are also out of the scope of this document.</t><t> The target audience of this document includes network equipment vendors, network operators, and standards development organizations. This document can be used as an index to some of the main OAM tools defined in the IETF. At the end of the document, a list of the OAM toolsets and a list of the OAM functions are presented as a summary. This document specifies the IP Flow Information Export (IPFIX) protocol, which serves as a means for transmitting Traffic Flow information over the network. In order to transmit Traffic Flow information from an Exporting Process to a Collecting Process, a common representation of flow data and a standard means of communicating them are required. This document describes how the IPFIX Data and Template Records are carried over a number of transport protocols from an IPFIX Exporting Process to an IPFIX Collecting Process. This document obsoletes RFC 5101. ITU Open Networking Foundation Fielding, Roy Object Management Group ITU ITU ITU ITU ITU Open Networking Foundation Open Networking Foundation ETSI

The authors would like to acknowledge Salvatore Loreto and Sudhir Modali for their contributions in the initial discussion on the SDNRG mailing list as well as their document-specific comments; they helped put this document in a better shape. Additionally, we would like to thank (in alphabetical order) Shivleela Arlimatti, Roland Bless, Scott Brim, Alan Clark, Luis Miguel Contreras Murillo, Tim Copley, Linda Dunbar, Ken Gray, Deniz Gurkan, Dave Hood, Georgios Karagiannis, Bhumip Khasnabish, Sriganesh Kini, Ramki Krishnan, Dirk Kutscher, Diego Lopez, Scott Mansfield, Pedro Martinez-Julia, David E. Mcdysan, Erik Nordmark, Carlos Pignataro, Robert Raszuk, Bless Roland, Francisco Javier Ros Munoz, Dimitri Staessens, Yaakov Stein, Eve Varma, Stuart Venters, Russ White, and Lee Young for their critical comments and discussions at IETF 88, IETF 89, and IETF 90 and on the SDNRG mailing list, which we took into consideration while revising this document. We would also like to thank (in alphabetical order) Spencer Dawkins and Eliot Lear for their IRSG reviews, which further refined this document. Finally, we thank Nobo Akiya for his review of the section on BFD, Julien Meuric for his review of the section on PCE, and Adrian Farrel and Benoit Claise for their IESG reviews of this document. Kostas Pentikousis is supported by , a research project partially funded by the European Community under the Seventh Framework Program (grant agreement no. 317880). The views expressed here are those of the author only. The European Commission is not liable for any use that may be made of the information in this document.

The authors would like to acknowledge (in alphabetical order) the following persons as contributors to this document. They all provided text, pointers, and comments that made this document more complete: Daniel King for providing text related to PCEP. Scott Mansfield for information regarding current ITU work on SDN. Yaakov Stein for providing text related to the CAP theorem and SDO-related information. Russ White for text suggestions on the definitions of control, management, and application.