]?> Illinois Institute of Technology

201 East Loop Road Wheaton IL 60187 USA +1 630 682 6024 davids@iit.edu

Bell Laboratories, Alcatel-Lucent

1960 Lucent Lane Rm 9C-533 Naperville IL 60566 USA +1 630 224 0216 vkg@bell-labs.com

Allot Communications

300 TradeCenter, Suite 4680 Woburn MA 08101 USA +1 508 309 2179 sporetsky@allot.com

Operations and Management Area Benchmarking Methodology Working Group This document provides a methodology for benchmarking the Session Initiation Protocol (SIP) performance of devices. Terminology related to benchmarking SIP devices is described in the companion terminology document. Using these two documents, benchmarks can be obtained and compared for different types of devices such as SIP Proxy Servers, Registrars and Session Border Controllers. The term "performance" in this context means the capacity of the device-under-test (DUT) to process SIP messages. Media streams are used only to study how they impact the signaling behavior. The intent of the two documents is to provide a normalized set of tests that will enable an objective comparison of the capacity of SIP devices. Test setup parameters and a methodology are necessary because SIP allows a wide range of configuration and operational conditions that can influence performance benchmark measurements.

In this document, the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" are to be interpreted as described in BCP 14, conforming to and indicate requirement levels for compliant implementations. RFC 2119 defines the use of these key words to help make the intent of standards track documents as clear as possible. While this document uses these keywords, this document is not a standards track document. The term Throughput is defined in . Terms specific to SIP performance benchmarking are defined in .

This document describes the methodology for benchmarking Session Initiation Protocol (SIP) performance as described in Terminology document . The methodology and terminology are to be used for benchmarking signaling plane performance with varying signaling and media load. Media streams, when used, are used only to study how they impact the signaling behavior. This document concentrates on benchmarking SIP session setup and SIP registrations only. The device-under-test (DUT) is a SIP server, which may be any SIP conforming device. Benchmarks can be obtained and compared for different types of devices such as a SIP proxy server, Session Border Controllers (SBC), SIP registrars and a SIP proxy server paired with a media relay. The test cases provide metrics for benchmarking the maximum 'SIP Registration Rate' and maximum 'SIP Session Establishment Rate' that the DUT can sustain over an extended period of time without failures (extended period of time is defined in the algorithm in ). Some cases are included to cover encrypted SIP. The test topologies that can be used are described in the Test Setup section. Topologies in which the DUT handles media as well as those in which the DUT does not handle media are both considered. The measurement of the performance characteristics of the media itself is outside the scope of these documents. SIP permits a wide range of configuration options that are explained in Section 4 and Section 2 of . Benchmark values could possibly be impacted by Associated Media. The selected values for Session Duration and Media Streams per Session enable benchmark metrics to be evaluated without Associated Media. Session Establishment Rate could possibly be impacted by the selected value for Maximum Sessions Attempted. The benchmark for Session Establishment Rate is measured with a fixed value for maximum Session Attempts. SIP permits a wide range of configuration options that are explained in and Section 2 of . Benchmark metrics could possibly be impacted by Associated Media. The selected values for Session Duration and Media Streams per Session enable benchmark metrics to be benchmarked without Associated Media. Session Setup Rate could possibly be impacted by the selected value for Maximum Sessions Attempted. The benchmark for Session Establishment Rate is measured with a fixed value for maximum Session Attempts. Finally, the overall value of these tests is to serve as a comparison function between multiple SIP implementations. One way to use these tests is to derive benchmarks with SIP devices from Vendor-A, derive a new set of benchmarks with similar SIP devices from Vendor-B and perform a comparison on the results of Vendor-A and Vendor-B. This document does not make any claims on the interpretation of such results.

There are two test topologies; one in which the DUT does not process the media () and the other in which it does process media (). In both cases, the tester or EA sends traffic into the DUT and absorbs traffic from the DUT. The diagrams in and represent the logical flow of information and do not dictate a particular physical arrangements of the entities. Test organizations need to be aware that these tests generate large volumes of data and consequently ensure that networking devices like hubs, switches or routers are able to handle the generated volume. depicts a layout in which the DUT is an intermediary between the two interfaces of the EA. If the test case requires the exchange of media, the media does not flow through the DUT but rather passes directly between the two endpoints. shows the DUT as an intermediary between the two interfaces of the EA. If the test case requires the exchange of media, the media flows through the DUT between the endpoints.

+ |------------>+ | | | | | | | | | Response | | Response | | | Tester +<------------| DUT +<------------| Tester | | (EA) | | | | (EA) | | | | | | | +--------+ +--------+ +--------+ /|\ /|\ | Media (optional) | +==============================================+ ]]>

+ |------------>+ | | | | | | | | | Response | | Response | | | Tester +<------------| DUT +<------------| Tester | | (EA) | | | | (EA) | | |<===========>| |<===========>| | +--------+ Media +--------+ Media +--------+ (Optional) (Optional) ]]>

Test cases may be performed with any transport protocol supported by SIP. This includes, but is not limited to, TCP, UDP, TLS and websockets. The protocol used for the SIP transport protocol must be reported with benchmarking results.

The Signaling Server is defined in the companion terminology document, (, Section 3.2.2). The Signaling Server is a DUT.

Some tests require Associated Media to be present for each SIP session. The test topologies to be used when benchmarking DUT performance for Associated Media are shown in and .

The test cases specified in this document provide SIP performance independent of the protocol used for the media stream. Any media protocol supported by SIP may be used. This includes, but is not limited to, RTP, RTSP, and SRTP. The protocol used for Associated Media MUST be reported with benchmarking results.

Benchmarking results may vary with the number of media streams per SIP session. When benchmarking a DUT for voice, a single media stream is used. When benchmarking a DUT for voice and video, two media streams are used. The number of Associated Media Streams MUST be reported with benchmarking results.

The value of the DUT's performance benchmarks may vary with the duration of SIP sessions. Session Duration MUST be reported with benchmarking results. A Session Duration of zero seconds indicates transmission of a BYE immediately following a successful SIP establishment. Setting this parameter to the value '0' indicates that a BYE will be sent by the EA immediately after the EA receives a 200 OK to the INVITE. Setting this parameter to a time value greater than the duration of the test indicates that a BYE is never sent.

The value of the DUT's performance benchmarks may vary with the Session Attempt Rate offered by the tester. Session Attempt Rate MUST be reported with the benchmarking results.

In order to benchmark the test cases uniformly in , the algorithm described in this section should be used. A prosaic description of the algorithm and a pseudo-code description are provided below, and a simulation written in the R statistical language is provided in Appendix A. The goal is to find the largest value, R, a SIP Session Attempt Rate, measured in sessions-per-second (sps), which the DUT can process with zero errors over a defined, extended period. This period is defined as the amount of time needed to attempt N SIP sessions, where N is a parameter of test, at the attempt rate, R. An iterative process is used to find this rate. The algorithm corresponding to this process converges to R. If the DUT vendor provides a value for R, the tester can use this value. Alternatively, in cases where the DUT vendor does not provide a value for R, or in cases where the tester wants to ascertain a vendor provided value using local media characteristics, the algorithm could be run by setting "r = R" and observing the value at convergence. The algorithm proceeds by initializing "r = 100"; "r" is the session attempt rate. The algorithm dynamically increases and decreases "r" as it converges to the a maximum sps value for R. The dynamic increase and decrease rate is controlled by the weights "w" and "d", respectively. If the DUT vendor provides a value for R, the tester can use that value; however, because the requirements and media characteristics are a function of the test environment, it is best that the tester reflect these requirements during testing and allow the algorithm to converge to R. The pseudo-code corresponding to the description above follows, and a simulation written in the R statistical language is provided in Appendix A.

= 10) while (TRUE) { s := send_traffic(r, m, n) ; Send r sps, with m media ; characteristics until n sessions established. if (s == n) { if (r > old_r) { old_r = r } else { count = count + 1 if (count >= 10) { # We've converged. h := max(r, old_r) break } } r := floor(r + (w * r)) } else { r := floor(r - (d * r)) d := max(0.10, d / 2) w := max(0.10, w / 2) } } return h end proc ]]>

SIP Transport Protocol = ___________________________ (valid values: TCP|UDP|TLS|SCTP|websockets|specify-other) Session Attempt Rate = _____________________________ (session attempts/sec) Total Sessions Attempted = _________________________ (total sessions to be created over duration of test) Media Streams Per Session = _______________________ (number of streams per session) Associated Media Protocol = _______________________ (RTP|RTSP|specify-other) Media Packet Size = _______________________________ (bytes) Establishment Threshold time = ____________________ (seconds) TLS ciphersuite used (for tests involving TLS) = ________________________ (e.g., TLS_RSA_WITH_AES_128_CBC_SHA) IPSec profile used (for tests involving IPSEC) = _____________________

Session Establishment Rate = ______________________ (sessions per second) Is DUT acting as a media relay (yes/no) = _________

Registration Rate = ____________________________ (registrations per second) Re-registration Rate = ____________________________ (registrations per second)

To benchmark the Session Establishment Rate of the Emulated Agent (EA) with zero failures. Configure the DUT in the test topology shown in . Set media streams per session to 0. Execute benchmarking algorithm as defined in to get the baseline session establishment rate. This rate MUST be recorded using any pertinent parameters as shown in the reporting format of . This is the scenario to obtain the maximum Session Establishment Rate of the EA and the test bed when no DUT is present. The results of this test might be used to normalize test results performed on different test beds or simply to better understand the impact of the DUT on the test bed in question.

To benchmark the Session Establishment Rate of the DUT with no associated media and zero failures. Configure a DUT according to the test topology shown in or . Set media streams per session to 0. Execute benchmarking algorithm as defined in to get the session establishment rate. This rate MUST be recorded using any pertinent parameters as shown in the reporting format of . Find the Session Establishment Rate of the DUT when the EA is not sending media streams.

To benchmark the Session Establishment Rate of the DUT with zero failures when Associated Media is included in the benchmark test but the media is not running through the DUT. Configure a DUT according to the test topology shown in . Set media streams per session to 1. Execute benchmarking algorithm as defined in to get the session establishment rate with media. This rate MUST be recorded using any pertinent parameters as shown in the reporting format of . Session Establishment Rate results obtained with Associated Media with any number of media streams per SIP session are expected to be identical to the Session Establishment Rate results obtained without media in the case where the DUT is running on a platform separate from the Media Relay.

To benchmark the Session Establishment Rate of the DUT with zero failures when Associated Media is included in the benchmark test and the media is running through the DUT. Configure a DUT according to the test topology shown in . Set media streams per session to 1. Execute benchmarking algorithm as defined in to get the session establishment rate with media. This rate MUST be recorded using any pertinent parameters as shown in the reporting format of . Session Establishment Rate results obtained with Associated Media may be lower than those obtained without media in the case where the DUT and the Media Relay are running on the same platform.

To benchmark the Session Establishment Rate of the DUT with zero failures when using TLS encrypted SIP signaling. If the DUT is being benchmarked as a proxy or B2BUA, then configure the DUT in the test topology shown in or . Configure the tester to enable TLS over the transport being used during benchmarking. Note the ciphersuite being used for TLS and record it in . Set media streams per session to 0 (media is not used in this test). Execute benchmarking algorithm as defined in to get the session establishment rate with TLS encryption. Session Establishment Rate results obtained with TLS Encrypted SIP may be lower than those obtained with plaintext SIP.

To benchmark the Session Establishment Rate of the DUT with zero failures when using IPsec Encrypted SIP signaling. Configure a DUT according to the test topology shown in or . Set media streams per session to 0 (media is not used in this test). Configure tester for IPSec. Note the IPSec profile being used for and record it in . Execute benchmarking algorithm as defined in to get the session establishment rate with encryption. Session Establishment Rate results obtained with IPSec Encrypted SIP may be lower than those obtained with plaintext SIP.

To benchmark the maximum registration rate the DUT can handle over an extended time period with zero failures. Configure a DUT according to the test topology shown in or . Set the registration timeout value to at least 3600 seconds. Execute benchmarking algorithm as defined in to get the maximum registration rate. This rate MUST be recorded using any pertinent parameters as shown in the reporting format of . For example, the use of TLS or IPSec during registration must be noted in the reporting format. Provides a maximum registration rate.

To benchmark the re-registration rate of the DUT with zero failures. Configure a DUT according to the test topology shown in or . First, execute test detailed in to register the endpoints with the registrar and obtain the registration rate. After at least 5 minutes of Step 2, but no more than 10 minutes after Step 2 has been performed, execute Step 3 of the test in . This will count as a re-registration because the SIP address of records have not yet expired. The rate should be at least equal to but not more than the result of .

This document does not requires any IANA considerations.

Documents of this type do not directly affect the security of Internet or corporate networks as long as benchmarking is not performed on devices or systems connected to production networks. Security threats and how to counter these in SIP and the media layer is discussed in RFC3261, RFC3550, and RFC3711 and various other drafts. This document attempts to formalize a set of common methodology for benchmarking performance of SIP devices in a lab environment.

The authors would like to thank Keith Drage and Daryl Malas for their contributions to this document. Dale Worley provided an extensive review that lead to improvements in the documents. We are grateful to Barry Constantine for providing valuable comments during the document's WGLC.

&rfc2119; &rfc2544; &rfc3261;

source("simulate.r") # > find_R(100) # ... debug output omitted ... # [1] 458 # # Thus, the max sps that the DUT can handle is 458 sps, which is # close to the absolute maximum of 460 sps the DUT is specified to # do. find_R <- function(r) { s = 0 old_r = 0 h = 0 count = 0 # Note that if w is small (say, 0.10) and r is small # (say, <= 9), the algorithm will not converge since it # uses floor() to increment r dynamically. It is best # off to start with the defaults (w = 0.10 and # r >= 10) cat("r old_r w d \n") while (TRUE) { cat(r, ' ', old_r, ' ', w, ' ', d, '\n') s = send_traffic(r) if (s == TRUE) { # All sessions succeeded if (r > old_r) { old_r = r } else { count = count + 1 if (count >= 10) { # We've converged. h = max(r, old_r) break } } r = floor(r + (w * r)) } else { r = floor(r - (d * r)) d = max(0.10, d / 2) w = max(0.10, w / 2) } } h } send_traffic <- function(r) { n = TRUE if (r > DUT_max_sps) { n = FALSE } n } ]]>