To specify a standard for measuring IPsec forwarding performance, the Network Processing Forum has developed the IPsec Forwarding Application Level Benchmark. The benchmark is specified in text, not code, to enable benchmarking of widely divergent network processor (NP) architectures.
In this article, we'll take a detailed look at the benchmark, examining the key technical elements that must be examined to determine IPsec performance. During the discussion, the article will look at forwarding rate, throughput, and latency tests. It will also examine related work being conducted by other standards bodies. To start off the discussion, however, let's provide a quick overview of the IPsec benchmark spec.
Benchmark Details
The NPF IPsec Forwarding Application Level Benchmark is designed to enable standard testing of dataplane-level performance of an IPsec implementation. Thus, the benchmark attempts to measure raw encryption and decryption under different realistic usage scenarios in the presence IPv4 packet forwarding. This benchmark, however, does not attempt to measure IPsec functions like key exchange and their associated negotiations.
Current IPsec devices are capable of much higher performance than current IPsec test hardware. To avoid testing the tester rather than the IPsec device, test setups require cleartext only traffic generation and analysis by the network tester.
The algorithm used to generate the IPsec selectors is included in the benchmark. This algorithm specified scales to support multiple port combinations and varying numbers of selectors. Vendors may use a selector setup that works for their reference hardware while still generating comparable numbers
Flexibility in encryption protocols is another goal of the benchmark. To this end the benchmark does not mandate a particular IPsec encryption protocol. The benchmark does require vendor disclosure of the IPsec protocol used.
The IPsec Benchmark Implementation is made up of two documents, the IPsec Benchmark Implementation Agreement and the IPsec Benchmark Reporting Template. A third component also available is the IPsec Benchmarking toolkit.
The IPsec Benchmark Implementation Agreement is a text description of the benchmark itself. Document structure and terminology follow the current IETF RFCs 12425 and 25447 and the draft IPsec benchmarking terminology.4 Benchmark sections include terminology, test configuration, test description, test parameters, and measurement criterion.
Disclosure of the results is standardized by the IPsec Benchmark Reporting Template. This template specifies the data points and graphs to be reported. The standard reporting format specified here enables an easy comparison of one report to another.
This benchmarking toolkit provides a sample implementation for a commercially available network tester. The scripts included in this toolkit drive the tester to generate the necessary packets and make the necessary measurements. The implementation kit is distributed for reference and is not considered an official part of the benchmark
Test Setup
The IPsec benchmark specifies a packet forwarding system containing two identical IPsec devices under test (DUTs) and one or more cleartext traffic testers (traffic generators/analyzers). The interfaces of each DUT are divided into two sets.
- M Clear Interfaces: That receive and transmit unencrypted IP traffic.
- N IPsec Interfaces: That receive and transmit IPsec traffic.
The M cleartext interfaces of each DUT are connected to the traffic tester. The N IPsec encrypted traffic interfaces connect the two DUTs. The traffic generator sends in cleartext traffic on the 2M clear traffic interfaces. The DUT receiving cleartext packets from the tester encrypts the packet and forwards it to the second DUT on one of the N interfaces inter-connecting the two DUTs. The receiving DUT then decrypts the packet and sends it on to the tester through a cleartext interface. With this setup, DUT IPsec performance can be tested to the very high IP cleartext packet rates delivered by testers (Figure 1).
Figure 1: Diagram showing an IPSec benchmark test setup.
A drawback of the two DUT setup is the inability to separately test DUT encryption and decryption performance. Test results represent only the sum of the two values.
The Forwarding Rate
The forwarding rate is defined as the maximum rate at which frames can be sent through the DUTs. Forwarding rate is measured in IP output frames per second for two different traffic pattern mixes:
- 100 Percent IPsec — all of the packets sent by the tester are encrypted before being sent from one DUT to the other.
- 50 Percent IPsec — Half of the tester generated packets are encrypted. The other half is IP forwarded from one DUT to the other as cleartext.
Testing both mixes shows the impact of cleartext traffic on encryption rates and visa versa. Zero percent IPsec traffic is not included since other NPF benchmarks provide comparable tests for this case.
The 100- and 50-percent traffic patterns mixes specifed above are tested for 1, 16, 512, and 1024 security associations (SAs) per output port. Individual tests for text packet sizes of 40, 576, 1400, and 9100 bytes are required, as are tests for an Internet mix of packet sizes consisting of 56-percent 40-byte, 20-percent 576-byte, and 24-percent 1400-byte IPv4 cleartext packets. Each traffic pattern is tested for either transport mode or tunnel mode or for both at a vendor's discretion. This set of parameters was selected to provide a broad measure of IPsec performance within a manageable number of individual tests.
As specified in the reporting template results from these tests are plotted against the theoretical maximum for the given set of interfaces. An example graph of the results for a fictitious DUT appears in the Figure 2.
Figure 2: Forwarding rate in packets per second for an example DUT.
Like all the IPsec benchmark tests, the forwarding rate test does not measure IPsec key exchange and tunnel setup performance. Before starting the benchmark tests the IPsec selectors on the DUTs are preset (manually or through DUT key exchanges). Selectors are specified through a selector setup algorithm described in section 5.1 of the IPsec benchmark.
Throughput and Latency
NPF's IPsec benchmark specifications defines throughput is defined as the maximum forwarding rate through the DUTs at which no offered frame is dropped. Like forwarding rate, throughput is measured in IP output frames per second.
This test is useful in cases where the forwarding rate achieved on the system is not the same as the theoretical maximum possible. This test shows the upper bound of the forwarding performance obtainable on the device without packet loss. Throughput is measured for the same set of packet sizes, SAs, modes, and traffic mixes used to measure forwarding rate
As specified in the RFCs, the IPsec IA defines latency differently for cut-through and for store-and-forward devices. Vendors are required to state which kind of device they have measured. The specified two DUT test setup means, latency can only be measured for full IPsec tunnels including both encrypting and decrypting DUTs. A limitation imposed by our use of cleartext only testers.
Latency for store-and-forward devices is defined as the time interval starting when the last bit of the input cleartext frame reaches the input port of the encrypting DUT and ending when the first bit of the output cleartext frame is seen on the output port of the decrypting DUT. For cut-through or bit-forwarding devices, latency is defined as the time interval starting when the first bit of the input cleartext frame reaches the input port of the encrypting DUT and ending when the first bit of the cleartext output frame is seen on the output port of the decrypting DUT. In both cut-through and store-and-forward devices, latency is measured in seconds
The benchmark specifies the measure of average, minimum and maximum latencies seen per packet on the device. Latencies are plotted for the same packet sizes and traffic patterns described in the forwarding rate section. Additionally these latencies must be measured at 100, 95, 90 and 50 percent of throughput rate of the DUTs. Latencies measured in this way provide a measure of the performance degradation seen when offered load increases. An ideal device would show little or no variation in latency as the load increases. A sample result for fictitious DUTs is shown in Figure 3.
Figure 3: Diagram showing latency measurements for an example DUT.
Along with latency, throughput, and forwarding rates, the IPsec Reporting Template specification mandates the disclosure of important IPsec and DUT characteristics. These characteristics include:
- IPsec Algorithm(s) used (E.g, 3DES / HMAC SHA-1 / CBC)
- Total memory requirements per SA
- Total memory requirement
- Interface details
- The number, type and speed of each cleartext interface
- The number, type and speed of each IPsec interface
Related Work
The NPF IPsec Forwarding Application Level Benchmark provides a standard set of metrics, a standard yet flexible test setup, and a standard set of benchmark tests. The benchmark enables industry standards based measurement of the forwarding rate, throughput and latency of IPsec packet forwarding devices.
Complimenting the NPF's efforts, a number of benchmarking terminology standards are already in place within the industry. The IETF defines a useful and very broad set of IPsec specific terminology in "Terminology for Benchmarking IPsec Devices".4 This IETF work builds on previous RFCs which also provide a vocabulary for benchmarking: RFC1242, RFC2285, RRC2544.5,6,7 The NPF used the terminology described in these documents in developing the NPF IPsec benchmark.
IPsec performance measurement is clearly of interest within the literature. NetBench presents a suite of nine benchmarks that include general purpose CPU code for Diffie-Helman public key encryption and MD5 signature generation.8 Commbench focuses on eight small program kernels including block cipher.9,10 NPBench includes a broad set of IPsec related benchmarks covering: AES, MD5 and Diffie-Hellman.11 All of these efforts are general purpose processor focused. All are also task level benchmarks (kernel focused). None includes the other functions needed to build an IPsec forwarding device.
Author's Note: A full description of the actual benchmark can be found at www.npforum.org
References
- IPv4 Benchmarking: Fairly Comparing NPU Beasts. Manohar Castelino, Frank Hady. CommsDesign.com. http://www.commsdesign.com/showArticle.jhtml?articleID=16501245.
- P. Chandra, F. Hady, R. Yavatkar, T. Bock, M. Cabot, P. Mathew, "Benchmarking Network Processors" HPCA8 — 2002.
- Standard Performance Evaluation Corporation [1995], SPEC CPU2000 V1.2, http://www.spec.org/cpu2000/.
- M. Bustos, T. VanHerck, M. Kaeo, "Terminology for Benchmarking IPsec Devices," IETF internet draft, draft=ietf-bmwg-IPsec-term-01.
- Benchmarking Terminology for Network Interconnect Devices, IETF RFC 1242
- Benchmarking Methodology for Network Interconnect Devices, IETF RFC 2285
- Benchmarking Methodology for Network Interconnect Devices, IETF RFC 2544
- Gohkham Memik, William Mangione-Smith, Wendong Hu, "NetBench: A Benchmarking Suite for Network Processors", International Conference on Computer Aided Design 2001 , Pages 39-42.
- Tilman Wolf, Jonathon Turner, "Design Issues for High Performance Active Routers," IEEE Journal on Selected Areas of Communications, Vol 19 No. 3, March 2001.
- Tilman Wolf, Mark Franklin, "Commbench — A Telecommunications Benchmark for Network Processors." IEEE International Symposium on Performance Analysis of Systems and Software," Austin, Texas, 2000.
- B. Lee, L. John. "NPBench: A Benchmark Suite for Control plane and Data plane Applications on Network Processors," IEEE International Conference on Computer Design (ICCD 2003), October 2003.
About the Authors
Manohar Ruben Castelino is a senior network software engineer in Intel's Infrastructure Processor Group. He works on developing high performance applications using the Intel IXP Family of network processors. Manohar has a B.E. degree from KREC India and can be reached at manohar.r.castelino@intel.com.
Frank Hady is a senior principal engineer at Intel where he leads a small research group focused on providing high performance platforms for heavy compute networking applications. He served as the NPF, NP Benchmarking Task Group chair during the creation of the IPsec benchmark. Frank earned a PhD in Electrical Engineering from the University of Maryland, as well as MS and BS degrees in EE from the University of Virginia. He can be reached at frank.hady@intel.com.
