Terminology for Benchmarking Session Initiation Protocol (SIP) Devices: Basic session setup and registration

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14, RFC2119 . RFC 2119 defines the use of these key words to help make the intent of standards track documents as clear as possible. While this document uses these keywords, this document is not a standards track document. The term Throughput is defined in RFC2544 . For the sake of clarity and continuity, this document adopts the template for definitions set out in Section 2 of RFC 1242 . The term Device Under Test (DUT) is defined in the following BMWG documents: Device Under Test (DUT) (c.f., Section 3.1.1 RFC 2285 ). Many commonly used SIP terms in this document are defined in RFC 3261 . For convenience the most important of these are reproduced below. Use of these terms in this document is consistent with their corresponding definition in the base SIP specification as amended by , and . Call Stateful: A proxy is call stateful if it retains state for a dialog from the initiating INVITE to the terminating BYE request. A call stateful proxy is always transaction stateful, but the converse is not necessarily true. Stateful Proxy: A logical entity that maintains the client and server transaction state machines defined by this specification during the processing of a request, also known as a transaction stateful proxy. The behavior of a stateful proxy is further defined in Section 16 of RFC 3261 . A transaction stateful proxy is not the same as a call stateful proxy. Stateless Proxy: A logical entity that does not maintain the client or server transaction state machines defined in this specification when it processes requests. A stateless proxy forwards every request it receives downstream and every response it receives upstream. Back-to-back User Agent: A back-to-back user agent (B2BUA) is a logical entity that receives a request and processes it as a user agent server (UAS). In order to determine how the request should be answered, it acts as a user agent client (UAC) and generates requests. Unlike a proxy server, it maintains dialog state and must participate in all requests sent on the dialogues it has established. Since it is a concatenation of a UAC and a UAS, no explicit definitions are needed for its behavior. | | | | | | | Attempting | | | 2. | 200 OK | | | ---------+<--------------| | | | ACK | | | |-------------->| | | | | | | | | | | | | | Media | Established | |<=====>| | | | | 3. | BYE | | | --------+--------------> | | | | | | | Disconnecting | | | 4. | 200 OK | | | --------|<-------------- | | | | | | | ]]> -->

Service Providers and IT Organizations deliver Voice Over IP (VoIP) and Multimedia network services based on the IETF Session Initiation Protocol (SIP) . SIP is a signaling protocol originally intended to be used to dynamically establish, disconnect and modify streams of media between end users. As it has evolved it has been adopted for use in a growing number of services and applications. Many of these result in the creation of a media session, but some do not. Examples of this latter group include text messaging and subscription services. The set of benchmarking terms provided in this document is intended for use with any SIP-enabled device performing SIP functions in the interior of the network, whether or not these result in the creation of media sessions. The performance of end-user devices is outside the scope of this document. A number of networking devices have been developed to support SIP-based VoIP services. These include SIP Servers, Session Border Controllers (SBC) and Back-to-back User Agents (B2BUA). These devices contain a mix of voice and IP functions whose performance may be reported using metrics defined by the equipment manufacturer or vendor. The Service Provider or IT Organization seeking to compare the performance of such devices will not be able to do so using these vendor-specific metrics, whose conditions of test and algorithms for collection are often unspecified. SIP functional elements and the devices that include them can be configured many different ways and can be organized into various topologies. These configuration and topological choices impact the value of any chosen signaling benchmark. Unless these conditions-of-test are defined, a true comparison of performance metrics across multiple vendor implementations will not be possible. Some SIP-enabled devices terminate or relay media as well as signaling. The processing of media by the device impacts the signaling performance. As a result, the conditions-of-test must include information as to whether or not the device under test processes media and if the device does process media, a description of the media handled and the manner in which it is handled. This document and its companion methodology document provide a set of black-box benchmarks for describing and comparing the performance of devices that incorporate the SIP User Agent Client and Server functions and that operate in the network's core. The definition of SIP performance benchmarks necessarily includes definitions of Test Setup Parameters and a test methodology. These enable the Tester to perform benchmarking tests on different devices and to achieve comparable results. This document provides a common set of definitions for Test Components, Test Setup Parameters, and Benchmarks. All the benchmarks defined are black-box measurements of the SIP signaling plane. The Test Setup Parameters and Benchmarks defined in this document are intended for use with the companion Methodology document.

The scope of this work item is summarized as follows: This terminology document describes SIP signaling performance benchmarks for black-box measurements of SIP networking devices. Stress and debug scenarios are not addressed in this work item. The DUT must be an RFC 3261 capable network equipment. This may be a Registrar, Redirect Server, Stateless Proxy or Stateful Proxy. A DUT MAY also include a B2BUA, SBC functionality. The DUT MUST NOT be end user equipment, such as personal digital assistant, a computer-based client, or a user terminal. The Tester acts as multiple "Emulated Agents" (EA) that initiate (or respond to) SIP messages as session endpoints and source (or receive) associated media for established connections. SIP Signaling in presence of Media The media performance is not benchmarked in this work item. Some tests require media, but the use of media is limited to observing the performance of SIP signaling. Tests that require media will annotate the media characteristics as a condition of test. The type of DUT dictates whether the associated media streams traverse the DUT. Both scenarios are within the scope of this work item. SIP is frequently used to create media streams; the signaling plane and media plane are treated as orthogonal to each other in this document. While many devices support the creation of media streams, benchmarks that measure the performance of these streams are outside the scope of this document and its companion methodology document . Tests may be performed with or without the creation of media streams. The presence or absence of media streams MUST be noted as a condition of the test as the performance of SIP devices may vary accordingly. Even if the media is used during benchmarking, only the SIP performance will be benchmarked, not the media performance or quality. Both INVITE and non-INVITE scenarios (registrations) are addressed in this document. However, benchmarking SIP presence or subscribe-notify extensions is not a part of this work item. Different transport -- such as UDP, TCP, SCTP, or TLS -- may be used. The specific transport mechanism MUST be noted as a condition of the test as the performance of SIP devices may vary accordingly. REGISTER and INVITE requests may be challenged or remain unchallenged for authentication purpose. Whether or not the REGISTER and INVITE requests are challenged is a condition of test which will be recorded along with other such parameters which may impact the SIP performance of the device or system under test. Re-INVITE requests are not considered in scope of this work item since the benchmarks for INVITEs are based on the dialog created by the INVITE and not on the transactions that take place within that dialog. Only session establishment is considered for the performance benchmarks. Session disconnect is not considered in the scope of this work item. This is because our goal is to determine the maximum capacity of the device or system under test, that is the number of simultaneous SIP sessions that the device or system can support. It is true that there are BYE requests being created during the test process. These transactions do contribute to the load on the device or system under test and thus are accounted for in the metric we derive. We do not seek a separate metric for the number of BYE transactions a device or system can support. IMS-specific scenarios are not considered, but test cases can be applied with 3GPP-specific SIP signaling and the P-CSCF as a DUT.

The combination of signaling and media messages and processes that support a SIP-based service. SIP messages are used to create and manage services for end users. Often, these services include the creation of media streams that are defined in the SDP body of a SIP message and carried in RTP protocol data units. However, SIP messages can also be used to create Instant Message services and subscription services, and such services are not associated with media streams. SIP reserves the term "session" to describe services that are analogous to telephone calls on a circuit switched network. SIP reserves the term "dialog" to refer to a signaling-only relationship between User Agent peers. SIP reserves the term "transaction" to refer to the brief communication between a client and a server that lasts only until the final response to the SIP request. None of these terms describes the entity whose performance we want to benchmark. For example, the MESSAGE request does not create a dialog and can be sent either within or outside of a dialog. It is not associated with media, but it resembles a phone call in its dependence on human rather than machine initiated responses. The SUBSCRIBE method does create a dialog between the originating end-user and the subscription service. It, too, is not associated with a media session. In light of the above observations we have extended the term "session" to include SIP-based services that are not initiated by INVITE requests and that do not have associated media. In this extended definition, a session always has a signaling component and may also have a media component. Thus, a session can be defined as signaling-only or a combination of signaling and media. We define the term "Associated Media", see , to describe the situation in which media is associated with a SIP dialog. The terminology "Invite-Initiated Session" (IS) and "Non-invite-Initiated Session" (NS) are used to distinguish between these two types of session. An Invite-Initiated Session is a session as defined in SIP. The performance of a device or system that supports Invite-Initiated Sessions that do not create media sessions, "Invite-Initiated Sessions without Associated Media", can be measured and is of interest for comparison and as a limiting case. The REGISTER request can be considered to be a "Non-invite-Initiated Session without Associated Media." A separate set of benchmarks is provided for REGISTER requests since most implementations of SIP-based services require this request and since a registrar may be a device under test. A Session in the context of this document, can be considered to be a vector with three components: A component in the signaling plane (SIP messages), sess.sig; A media component in the media plane (RTP and SRTP streams for example), sess.med (which may be null); A control component in the media plane (RTCP messages for example), sess.medc (which may be null). An IS is expected to have non-null sess.sig and sess.med components. The use of control protocols in the media component is media dependent, thus the expected presence or absence of sess.medc is media dependent and test-case dependent. An NS is expected to have a non-null sess.sig component, but null sess.med and sess.medc components. Packets in the Signaling Plane and Media Plane will be handled by different processes within the DUT. They will take different paths within a network. These different processes and paths may produce variations in performance. The terminology and benchmarks defined in this document and the methodology for their use are designed to enable us to compare performance of the DUT with reference to the type of SIP-supported application it is handling. Note that one or more sessions can simultaneously exist between any participants. This can be the case, for example, when the EA sets up both an IM and a voice call through the DUT. These sessions are represented as an array session[x]. Sessions will be represented as a vector array with three components, as follows: session-> session[x].sig, the signaling component session[x].medc[y], the media control component (e.g. RTCP) session[x].med[y], an array of associated media streams (e.g. RTP, SRTP, RTSP, MSRP). This media component may consist of zero or more media streams. models the vectors of the session. N/A. None. Media Plane Signaling Plane Associated Media Invite-Initiated Session (IS) Non-Invite-Initiated Session (NS)

The plane in which SIP messages are exchanged between SIP Agents . SIP messages are used to establish sessions in several ways: directly between two User Agents , through a Proxy Server , or through a series of Proxy Servers. The Session Description Protocol (SDP) is included in the Signaling Plane. The Signaling Plane for a single Session is represented by session.sig. N/A. None. Media Plane EAs

The data plane in which one or more media streams and their associated media control protocols are exchanged between User Agents after a media connection has been created by the exchange of signaling messages in the Signaling Plane. Media may also be known as the "bearer channel". The Media Plane MUST include the media control protocol, if one is used, and the media stream(s). Examples of media are audio and video. The media streams are described in the SDP of the Signaling Plane. The media for a single Session is represented by session.med. The media control protocol for a single media description is represented by session.medc. N/A. None. Signaling Plane

Media that corresponds to an 'm' line in the SDP payload of the Signaling Plane. Any media protocol MAY be used. For any session's signaling component, session.sig, there may be zero, one, or multiple associated media streams. When there are multiple media streams, these are represented be a vector array session.med[y]. When there are multiple media streams there will be multiple media control protocol descriptions as well. They are represented by a vector array session.medc[y]. N/A. None.

Overload is defined as the state where a SIP server does not have sufficient resources to process all incoming SIP messages . The distinction between an overload condition and other failure scenarios is outside the scope of black box testing and of this document. Under overload conditions, all or a percentage of Session Attempts will fail due to lack of resources. In black box testing the cause of the failure is not explored. The fact that a failure occurred for whatever reason, will trigger the tester to reduce the offered load, as described in the companion methodology document, . SIP server resources may include CPU processing capacity, network bandwidth, input/output queues, or disk resources. Any combination of resources may be fully utilized when a SIP server (the DUT) is in the overload condition. For proxy-only (or intermediary) devices, it is expected that the proxy will be driven into overload based on the delivery rate of signaling requests. N/A.

A SIP INVITE or REGISTER request sent by the EA that has not received a final response. The attempted session may be Invite Initiated or Non-invite-Initiated. When counting the number of session attempts we include all INVITEs that are rejected for lack of authentication information. The EA needs to record the total number of session attempts including those attempts that are routinely rejected by a proxy that requires the UA to authenticate itself. The EA is provisioned to deliver a specific number of session attempts per second. But the EA must also count the actual number of session attempts per given tie interval. N/A. None. Session Session Attempt Rate Invite-Initiated Session Non-Invite-Initiated Session

A SIP session for which the EA acting as the UE/UA has received a 200 OK message. An Established Session MAY be Invite Initiated or Non-Invite-Initiated. Early dialogues for INVITE requests are out of scope for this work. N/A. None. Invite-Initiated Session

A Session that is created by an exchange of messages in the Signaling Plane, the first of which is a SIP INVITE request. When an IS becomes an Established Session its signaling component is identified by the SIP dialog parameter values, Call-ID, To-tag, and From-tag (RFC3261 ). An IS may have zero, one or multiple Associated Media descriptions in the SDP body. The inclusion of media is test case dependent. An IS is successfully established if the following two conditions are met: Sess.sig is established by the end of Establishment Threshold Time (c.f. ), and If the DUT is a device, such as an SBC or B2BUA, that may receive media from a calling or called party before a signaling dialog is established or confirmed , this fact needs to be taken into account when the EA is built. Any additional parameters that need to be configured in the EA must be added to the list of test setup parameters in Section 5.1 of N/A. None. Session Non-Invite-Initiated Session Associated Media

A session that is created by an exchange of SIP messages in the Signaling Plane the first of which is not a SIP INVITE message. An NS is successfully established if the Session Attempt via a non- INVITE request results in the EA receiving a 2xx reply before the expiration of the Establishment Threshold timer (c.f., ). For the purpose of this document, a NS is a session created only by the REGISTER request and no other request. N/A. None. Session Invite-Initiated Session

A session attempt that does not result in an Established Session. The session attempt failure may be indicated by the following observations at the EA: Receipt of a SIP 3xx-, 4xx-, 5xx-, or 6xx-class response to a Session Attempt. The lack of any received SIP response to a Session Attempt within the Establishment Threshold Time (c.f. ). N/A. None. Session Attempt

A device in the test topology that initiates/responds to SIP messages as one or more session endpoints and, wherever applicable, sources/receives Associated Media for Established Sessions. The EA functions in the Signaling and Media Planes. The Tester may act as multiple EAs. N/A None. Media Plane Signaling Plane Established Session Associated Media

Device in the test topology that facilitates the creation of sessions between EAs. This device is the DUT. The DUT is a RFC3261-capable network intermediary such as a Registrar, Redirect Server, User Agent Server, Stateless Proxy, Stateful Proxy, B2BUA or SBC. NA None. Signaling Plane

The protocol used for transport of the Signaling Plane messages. Performance benchmarks may vary for the same SIP networking device depending upon whether TCP, UDP, TLS, SCTP, websockets or any future transport layer protocol is used. For this reason it is necessary to measure the SIP Performance Benchmarks using these various transport protocols. Performance Benchmarks MUST report the SIP Transport Protocol used to obtain the benchmark results. While these are not units of measure, they are attributes that are one of many factors that will contribute to the value of the measurements to be taken. TCP, UDP, SCTP, TLS over TCP, TLS over UDP, TLS over SCTP, and websockets are among the possible values to be recorded as part of the test. None.

Configuration of the EA for the number of sessions per second that the EA attempts to establish using the services of the DUT. The Session Attempt Rate is the number of sessions per second that the EA sends toward the DUT. Some of the sessions attempted may not result in a session being established. A session in this case may be either an IS or an NS. Session attempts per second None. Session Session Attempt

Configuration of the EA that represents the amount of time that an EA client will wait for a response from an EA server before declaring a Session Attempt Failure. This time duration is test dependent. It is RECOMMENDED that the Establishment Threshold Time value be set to Timer B (for ISs) or Timer F (for NSs) as specified in RFC 3261, Table 4 . Following the default value of T1 (500ms) specified in the table and a constant multiplier of 64 gives a value of 32 seconds for this timer (i.e., 500ms * 64 = 32s). Seconds None.

Configuration of the EA that represents the amount of time that the SIP dialog is intended to exist between the two EAs associated with the test. The time at which the BYE is sent will control the Session Duration. seconds None.

Configuration on the EA for a fixed number of frames or samples to be sent in each RTP packet of the media session. For a single benchmark test, media sessions use a defined number of samples or frames per RTP packet. If two SBCs, for example, used the same codec but one puts more frames into the RTP packet, this might cause variation in the performance benchmark results. An integer number of frames or samples, depending on whether hybrid- or sample-based codec are used, respectively. None.

The name of the codec used to generate the media session. For a single benchmark text, all sessions use the same size packet for media streams. The size of packets can cause a variation in the performance benchmark measurements. This is a textual name (alphanumeric) assigned to uniquely identify the codec. None.

The maximum value of the Session Attempt Rate that the DUT can handle for an extended, pre-defined, period with zero failures. This benchmark is obtained with zero failure in which 100% of the sessions attempted by the Emulated Agent are successfully completed by the DUT. The session attempt rate provisioned on the EA is raised and lowered as described in the algorithm in the accompanying methodology document, until a traffic load at the given attempt rate over the sustained period of time identified by T in the algorithm completes without any failed session attempts. Sessions may be IS or NS or a mix of both and will be defined in the particular test. sessions per second (sps) None. Invite-Initiated Sessions Non-Invite-Initiated Sessions Session Attempt Rate

The maximum value of the Registration Attempt Rate that the DUT can handle for an extended, pre-defined, period with zero failures. This benchmark is obtained with zero failures in which 100% of the registrations attempted by the EA are successfully completed by the DUT. The registration rate provisioned on the Emulated Agent is raised and lowered as described in the algorithm in the companion methodology draft until a traffic load consisting of registration attempts at the given attempt rate over the period of time necessary to attempt N registrations completes without failure, where N is a parameter specified in the algorithm and recorded in the Test Setup Report. This benchmark is described separately from the Session Establishment Rate (), although it could be considered a special case of that benchmark, since a REGISTER request is a request for a Non-Invite-Initiated session. It is defined separately because it is a very important benchmark for most SIP installations. An example demonstrating its use is an avalanche restart, where hundreds of thousands of end points register simultaneously following a power outage. In such a case, an authoritative measurement of the capacity of the device to register endpoints is useful to the network designer. Finally, in certain controlled networks, there appears to be a difference in the registration rate of new endpoints registering versus existing endpoints refreshing their registrations. This benchmark can capture these differences as well. registrations per second (rps) None.

Configuration of the EA for the number of registrations per second that the EA attempts to send to the DUT. The Registration Attempt Rate is the number of registration requests per second that the EA sends toward the DUT. Registrations per second (rps) None. Non-Invite-Initiated Session

This document requires no IANA considerations.

Documents of this type do not directly affect the security of Internet or corporate networks as long as benchmarking is not performed on devices or systems connected to production networks. Security threats and how to counter these in SIP and the media layer is discussed in RFC3261 , RFC 3550 , RFC3711 and various other drafts. This document attempts to formalize a set of common terminology for benchmarking SIP networks. Packets with unintended and/or unauthorized DSCP or IP precedence values may present security issues. Determining the security consequences of such packets is out of scope for this document.

The authors would like to thank Keith Drage, Cullen Jennings, Daryl Malas, Al Morton, and Henning Schulzrinne for invaluable contributions to this document. Dale Worley provided an extensive review that lead to improvements in the documents. We are grateful to Barry Constantine for providing valuable comments during the document's WGLC.