Chapter 4: Test Methodology
www.spirentcampus.com
Topic Overview
Types of Tests
What is Performance Testing?
BMWG – Benchmarking Methodology Working Group
Core RFCs (1242, 2544, 2285, 2889)
Evolution of Switching Technology
Ethernet, VLAN, IPv4, QoS Test Methodologies
Test Objectives
Test Parameters
Test Procedures
Reporting Formats
2
Types of Tests
In exercising and validating all the characteristics of a DUT,
there are four different types of tests to run:
A Conformance test consists of verifying the device’s
behavior in correspondence with the standards, conventions,
rules.
A Functional test consists of verifying that the device does
everything it is supposed to do (i.e., protocol support, filters,
management, etc.).
A Performance test consists of “stress testing” the device to
see how it behaves under loaded conditions.
A Passive test consists of Protocol analysis. This can be
accomplished either intrusively or non-intrusively.
3
Conformance Testing
Validating adherence to a standard
Tests all facets of the RFCs
Concepts developed by Telcordia – Bell Standard
Hundreds of individual tests
Identifies the sections of the RFC that are being tested
Assures correct handling of error conditions
Should reveal how the “SHALLs” and “MAYs” are
implemented (i.e., optional elements)
Subset of these tests can be used as manufacturing test
(e.g., does a specific board meet a physical interface
standard?)
4
Functional Testing
Validates “system” components
Internal management functions
Control plan and data plane functions
Proprietary functions between systems (e.g. encapsulation,
trunking protocols)
Testing of the correct implementation and operation of a
communications protocol
Typically less rigorous than a conformance test
Verifies interaction between multiple functions
Negative Testing (e.g., introduce impairments to a stable
device or network)
5
Functional Testing – Negative Testing
Introduce impairments to a stable device or network:
Route or link flapping
Invalid routes
Malformed packets
Invalid policies
Lost keep-alives
6
Performance Testing
What is performance testing?
Setting an expectation of the device’ functionality
It could be said, “there are 3 levels of performance testing.”
Performance testing (set a benchmark under low load conditions)
Load testing (how it functions under normal operating conditions)
Stress testing (diagnose problems under heavy loads)
Keep it realistic.
A broad distribution of source and destination addresses.
A mix of packet sizes and types.
The bandwidth characteristics of realistic traffic.
Different ratios of unicast and multicast traffic.
Many different streams and conditions that force prioritization (for QoS)
7
Who Needs Performance Testing?
Test setups and methodologies were derived from the
following RFCs:
RFC 1242 “Benchmarking Terminology for Network Interconnection Devices”
RFC 2544 “Benchmarking Methodology for Network Interconnection Devices”
RFC 2285 “Benchmarking Terminology for LAN Switching Devices”
RFC 2889 “Benchmarking Methodology for LAN Switching Devices”
Performance Tester
Traffic
Generation:
X Frames
Transmitted
Device Under Test (DUT)
Traffic
Analysis:
Y Frames
Received
8