octoScope

Wireless personal testbeds


MENU

Test suite for Broadband Forum TR-398

Documentation

TR-398 is an industry standard test plan created by the Broadband Forum. The primary goal of the specification is to provide a standard set of test cases and framework to measure aspects of the performance between Access Point (AP) and one or more reference Stations (STA) under controlled laboratory conditions. The test cases are defined for a Device Under Test (DUT – AP only) tested against a STA or a set of STA.

Spirent’s implementation of TR-398 runs on a set of standard octoBox testbeds as outlined in this datasheet. The implementation features a web UI to execute each of the test case in TR-398 individually, in groups, or all of them at once. A printable HTML report can be generated at the end of the execution of the test cases. TR-398 test suite requires Spirent’s scriptMachine to run (see separate datasheet).

Features

  • Performance tests of Access Points according to TR-398 issue 2
  • Works on multiple octoScope testbeds.
  • Web UI available for tests execution
  • HTML reporting available for test cases.

Benefits

  • Measures the performance of access points against an industry standard test plan
  • Provides 100% coverage for TR-398 Issue2 test plan.
  • Complete isolation from outside interference to produce reliable and repeatable results.

TR398 Issue 2 test cases covered by octoScope’s testbed.

Test Case# TESTS STACK-MAX STACK-MID STACK-MIN
6.1.1 Receiver sensitivity
6.2.1 Maximum Connection
6.2.2 Maximum Throughput
6.2.3 Airtime Fairness  
6.2.4 Dual-band Throughput
6.2.5 Bidirectional Throughput
6.3.1 Rate Vs Range
6.3.2 Spatial Consistency
6.3.3 802.11ax Peak Performance
6.3.4 Repeated Wi-Fi throughput  
6.3.5 Basic Roaming Performance  
6.4.1 Multiple STAs Performance
6.4.2 Multiple Association Disassociation Stability
6.4.3 Downlink MU-MIMO
6.5.1 Long Term Stability
6.5.2 AP Coexistence  
6.5.3 Automatic Channel Selection  

Testbed Details

6.1.1 Receiver Sensitivity Test

TR-398

Fig 1. Receiver Sensitivity

This test provides a simplified measurement of the receiver’s sensitivity, relative to the total attenuation inserted between the DUT and the STA. As that attenuation is increased, the STA is limited to a single coding scheme, eventually causing the connection to degrade. The attenuation at which the connection degrades represents the receiver’s approximate sensitivity.

Fig 1 highlights the paths used to implement the Receiver Sensitivity Test in an octoBox STACK-MID testbed. The Device Under Test is placed in chamber A. Pal-6 test instrument from chamber C is used as the station. An RvR (Rate Vs Range) test is run at each orientation ranging from 0 to 360 degrees in steps of 45 degrees. The attenuation level at which the PER is greater than 10% is recorded at each step. This average of these values is recorded as the receiver’s approximate sensitivity.

6.2.1 Maximum Connection Test

TR-398

Fig 2. Maximum Connection

The Maximum Connection test intends to verify that the Wi-Fi AP can support 32 STAs simultaneously connected with minimal packet loss and no disassociations taking place.

Fig 2 highlights the paths used to implement the Maximum Connection Test in an octoBox STACK-MID testbed. The Device Under Test is placed in chamber A. Pal-6 in chamber C is configured as 32 stations to connect to the DUT in 2.4GHz and 5GHz band. A downlink UDP connection is established between the STAs and the DUT. The PER is recorded. To pass this test case, the PER for each STA shall achieve less than 1%.

6.2.2 Maximum Throughput Test

TR-398

Fig 3. Maximum Throughput

Maximum throughput test intends to measure the maximum throughput performance of the DUT. The test is conducted with connection by air interface in short distance (by considering the actual utilization of Wi-Fi).

Fig 3 highlights the paths used to implement the Maximum Throughput Test in an octoBox STACK-MID testbed. The device under test is placed in chamber A. Pal-6 in chamber C is used as the STA in the test. A downlink TCP connection is established between Pal-6 and the DUT. The measured throughput should meet the performance requirement specified in the TR398 test plan for different configuration modes. The throughput results are reported by octoScope TR398 test suite along with other statistics including MCS, rate, RSSI, bandwidth etc. And it can be saved as HTML/PDF reports.

6.2.3 Airtime Fairness Test

TR-398

Fig 4. Airtime Fairness

Wi-Fi signal transmission can be seen as a multicast process since the STAs involved share the transmission medium. Air interface becomes a rare resource when dense connections or high throughput requests exist. Channel condition determines the MCS selection, therefore affecting the data throughput. In general, long distance to travel or obstacle penetration leads to larger attenuation, which makes the data rate in a low level. Occupying excessive airtime of STA with small MCS will be unfair to the STAs with large MCS (here, assuming the QoS requirement is similar) when the air resources have already run out. Airtime Fairness Test intends to verify the capability of Wi-Fi device to guarantee the fairness of airtime usage.

Fig 4 highlights the paths used to implement the Airtime Fairness Test in an octoBox STACK-MID testbed. Device under Test is placed in chamber A. Pal-6 in chamber B and STApal in chamber E are used as STAs. The test has 3 scenarios. The first is a baseline test which assumes both the stations at a same distance from the DUT. In the second scenario, STA2 is moved at a certain distance, and in the third the second STA is converted to a legacy device to test the DUT capability to guarantee airtime fairness. In this test Pal-6 and STApal are configured as STAs and the attenuators in the path are used to simulate the range. A downlink TCP connection is established with the two STAs associated simultaneously to the DUT for the 3 mentioned scenarios. The throughput limit and throughput variation are recorded for each scenario and saved as HTML/PDF reports from the TR398 test suite.

6.2.4 Dual-band Throughput Test

TR-398

Fig 5. Dual-band Throughput

The Dual-band Throughput Test is intended to measure the throughput the DUT can support when concurrently connected to multiple stations on both the 2.4 and 5 GHz bands, each operating with two spatial streams.

Fig 5 highlights the paths used to implement the Dual-band Throughput Test in an octoBox STACK-MID testbed. The Device Under test is placed in chamber A. Pal-6 STAs each operating in 2.4GHz and 5GHz band respectively are used in chamber C. A TCP connection is established between the STAs and the DUT, and various iterations of downlink and uplink throughput are recorded for specified attenuation steps.

6.2.5 Bidirectional Throughput Test

TR-398

Fig 6. Bidirectional Throughput

The Bidirectional Throughput Test measures the uplink and downlink simultaneously achieved through the DUT and a station.

Fig 6 highlights the paths used to implement the Bi-directional Throughput Test in an octoBox STACK-MID testbed. Device Under Test is placed in chamber A. Pal-6 in chamber C is used as a STA. The Pal-6 is configured a STA in 2.4GHz and 5GHz, respectively. The MPE-B is set to Line-of-Sight mode and is used to simulate range between the STA and the DUT. A downlink TCP connection is established between the Pal-6 and DUT and throughput is recorded at specified attenuation steps.

6.3.1 802.11ax Peak Performance Test

TR-398

Fig 7. 802.11ax Peak Performance

This test case examines the maximum performance expected from a DUT supporting 802.11ax with the following additional capabilities beyond the default configuration: 4 spatial streams, 160 MHz Channels.

Fig 7 highlights the paths used to implement the Bi-directional Throughput Test in an octoBox STACK-MID testbed. Device Under Test is placed in chamber A. Pal-6 in chamber C is used as a STA. The Pal-6 is configured in different ax modes as specified in the TR-398 test plan. A full 8x8 path is provided between the Pal-6 and the DUT through MPE-A and MPE-B. MPEs are configured in LOS mode for this test and the attenuation is set to 0dB. A downlink and uplink TCP connection are established between the client and the AP and the throughput is recorded in the results.

6.3.2 Rate Vs Range Test

TR-398

Fig 8. Rate vs Range

Range versus rate test intends to measure the baseband and RF chain performance of Wi-Fi device. The attenuation of signals due to range increase is achieved by using attenuator in the STA side.

Fig 8 highlights the paths used to implement the Rate Vs Range Test in an octoBox STACK-MID testbed. Device Under Test is placed in chamber A. Pal-6 in chamber C is used as the STA. MPE-B is set to Line-of-Sight mode and is used to simulate the range. A TCP connection is established between the STA and the DUT. Throughput is recorded both in uplink and in downlink directions between the Pal-6 and DUT at specified attenuation steps.

6.3.3 Spatial Consistency Test

TR-398

Fig 9. Spatial Consistency

Spatial consistency test intends to verify the Wi-Fi signal consistency in spatial domain.

Fig 9 highlights the paths used to implement the Spatial Consistency Test in an octoBox STACK-MID testbed. Device Under Test is placed in chamber A. Pal-6 in chamber C is configured as STA in specified configuration modes included in the test plan. A TCP connection is established between the STA and the DUT. Throughput in downlink and in uplink is recorded at different orientations using the turntable in chamber A ranging from 0 degrees to 360 degrees. The attenuation is increased using the attenuator connecting the STA and the DUT and throughput is recorded at strong, medium, and weak signal levels.

6.3.4 Repeated Wi-Fi Throughput Performance Test

TR-398

Fig 10. Repeated Wi-Fi Throughput Performance

This test case measures the throughput performance of a Wi-Fi system, where the DUT is the combination of a “base AP” and a “Wi-Fi repeater.” The “base AP” is defined as the AP devices connected to the wired network connection of the traffic generator and analyzer, while the “Wi-Fi repeater” has only RF connections to the both the “base AP” and the STA. During this test, the STA is only connected to the “Wi-Fi Repeater.” The Wi-Fi connection between the “base AP” and “Wi-Fi repeater” is configured to enable all supported radios and protocols, allowing the two devices to adapt the Wi-Fi connection according to their internal logic. The STA’s RF connection is configured to a specific operating mode. The back-haul link between “base AP” and “Wi-Fi Repeater” is expected to be at least 802.11ac NSS=2 or better in performance for this test.

Fig 10 highlights the paths used to implement the Repeated Wi-Fi Throughput Performance Test in an octoBox STACK-MID testbed. The base AP under test is placed in chamber B. The repeater under test is placed in chamber A Linux STApal in chamber E is configured as a STA in specified configuration modes included in the test plan. A TCP connection is established between the STA and the repeater under test. The attenuator connecting chamber B and A simulates the range between the client and the repeater. Throughput is recorded in both uplink and in downlink directions at specified attenuations.

6.3.5 Basic Roaming Performance Test

TR-398

Fig 11. Basic Roaming Performance

This test case attempts to measure the roaming performance of the STA roaming between Wi-Fi connections of the “base AP” and the “Wi-Fi repeater.” The “base AP” is defined as the AP devices connected to the wired network connection of the traffic generator and analyzer, while the “Wi-Fi repeater” has only RF connections to the both the “base AP” and the STA. The roaming performance is defined as the duration of time the STA is disconnected from the network (not able to send or receive a packet from the Ethernet traffic generator / analyzer).

Fig 11 highlights the paths used to implement the Basic Roaming Performance Test in an octoBox STACK-MID testbed. The base AP under test is placed in chamber A. The repeater under test is placed in chamber B. The Linux STApal in chamber E is configured as a STA in specified configuration modes included in the test plan. QA-G and QA-E are used to simulate range between the client and the DUTs. A downlink UDP connection is established between the STApal and the base AP under test in chamber A. The attenuation on both QA-G and QA-E is modified so that the STApal roams from the base AP in chamber A to the repeater in chamber B. The lost packets are recorded as PER (Packet Error Rate) in TR-398 test suite.

6.4.1 Multiple STAs Performance Test

TR-398

Fig 12. Multiple STAs Performance

Multiple STAs performance test intends to measure the performance of Wi-Fi device connected with multiple STAs simultaneously. To simulate a circumstance of real environment, various levels of signals reflecting various distance between Wi-Fi device and STA are considered in the test.

Fig 12 highlights the paths used to implement the Multiple STA Performance Test in an octoBox STACK-MID testbed. Device Under Test is placed in chamber A. The test requires a total of 9 STAs in short, medium, and long range. The STApals in chamber C are configured as 9 stations and are associated with the DUT in chamber A. Individual channels of MPE-A and MPE-B are used to simulate short, medium, and long range between DUT and STApals. A TCP connection is established between the DUT and 9 STAs. Throughput is recorded in both uplink and in downlink direction by the TR-398 test suite.

6.4.2 Multiple Association/Dissociation Stability Test

TR-398

Fig 13. Multiple Association/Dissociation Stability

Multiple association/disassociation stability test intends to measure stability of Wi-Fi device under a dynamic environment with frequent change of connection status.

Fig 13 highlights the paths used to implement the Multiple STA Performance Test in an octoBox STACK-MID testbed. Device Under Test is placed in chamber A. Pal-6s in chambers B and D are configured as STA. The tests require a total of 16 STAs, 8 STAs are picked for sending/receiving packet while the other 8 STAs are picked to do association/re-association process during the test. Eight STApals in chamber C is configured are stations and are associated with the DUT in chamber A. A downlink UDP connection is established between STApals in chamber B and the DUT in chamber A. Eight Pal-6 virtual stations in chamber C are used for association/re-association process. Downlink UDP flow is enabled from DUT to STApals in chamber C. Association/Re-association occurs between Pal-6 in chamber C and the DUT every 30 seconds. The PER is recorded and saved in HTML/PDF reports by the TR-398 test suite.

6.4.3 Downlink MU-MIMO Performance Test

TR-398

Fig 14. Downlink MU-MIMO Performance

Downlink MU-MIMO Performance Test intends to verify the performance of Wi-Fi device when Downlink MU-MIMO is applied. This best represents a typical deployment, where stations may only support 1x1 or 2x2 RF chain configurations. The test is only applicable to the Wi-Fi device supporting the 802.11ac. Downlink MU-MIMO capability, also referred to as 802.11ac Wave 2. The DUT SHALL support 802.11ac MU-MIMO and at least 4 spatial streams.

Fig 14 highlights the paths used to implement the Multiple STA Performance Test in an octoBox STACK-MID testbed. Device Under Test is placed in chamber A. STApals in chamber C are configured as STAs. The test requires 3 STAs to establish a downlink TCP connection between the client and the AP. The throughput is recorded for individual STApals in chamber C with MU-MIMO enabled and then disabled. The sum of the throughput of all 3 STAs with MU-MIMO enabled should be 45% more than the sum of the throughput of all 3 STAs with MU-MIMO disabled.

6.5.1 Long Term Stability Test

TR-398

Fig 15. Long Term Stability

Long term stability test intends to measure the stability performance of Wi-Fi device under stresses that would typically been seen under heavy user load, such as watching multiple 4k video streams. Throughput and connection availability are continuously monitored over a period of 4 hours, during which time, the performance must remain steady. Testing is conducted in multiple bands (2.4 GHz and 5 GHz) simultaneously.

Fig 15 highlights the paths used to implement the Multiple STA Performance Test in an octoBox STACK-MID testbed. Device Under Test is placed in chamber A. STApals in chamber C are configured as STAs. The test requires a total of 4 STAs, two in each 2.4 GHz and 5 GHz band. Two STApals are configured as two stations. Two downlink UDP connections are established to STApals in chamber C and traffic measurements are taken for 15 minutes. While taking the throughput measurements the state of the other STA in each band is toggled after 5 minutes i.e., associate/re-associate to the DUT. Lost UDP packets are recorded in the form of PER by the TR-398 test suite. These measurements are taken for 16 intervals and lost packets are recorded for each measurement interval. The packet loss ratio should be less than or equal to 1E-4.s

6.5.2 AP Coexistence Test

TR-398

Fig 16. AP Coexistence Test

AP coexistence test intends to verify Wi-Fi device performance with existence of alien AP. The alien AP in the test SHALL support the same Wi-Fi standard (802.11n/802.11ac).

Fig 16 highlights the paths used to implement the Multiple STA Performance Test in an octoBox STACK-MID testbed. Device Under Tests is placed in chamber A. Pal-6 in chamber B is configured as a STA. AP in chamber B is configured as an alien AP associated with STApal in chamber E. A downlink TCP connection is established between the APs and the Pal-6. Downlink TPT is recorded with alien network off, idle, traffic running on same channel, overlapping channel, and an adjacent channel. The measured average TPT in the above 5 scenarios should be less than the allowed reduction specified in the test plan.

6.5.3 Automatic Channel Selection Test

TR-398

Fig 17. Automatic Channel Selection Test

When operating in environments with multiple Wi-Fi based networks the AP may be able to select the channel more reliably with the lowest level of impairments from the adjacent networks compared to a less knowledgeable end user. The test verifies the DUT (AP) will select a Wi-Fi channel that is not presently utilized by other networks and will alter this channel on subsequent reboots, if the environment has changed (i.e., the channels utilized by the adjacent network has changed). Note, this test is not intended to verify avoidance of radar or other non-Wi-Fi signals or usage of the same frequencies by other technology.

Fig 17 highlights the paths used to implement the Automatic Channel Selection Test in an octoBox STACK-MID testbed. Device Under Test is placed in chamber A. The Pal-6 in chamber B is used as the STA for the tests. The Access Point in chamber B is associated with STApal in chamber E as the interference network. ACS is enabled on the DUT in chamber A. STApal in chamber E is associated to the additional AP in chamber B and a TCP connection is established between the two. The DUT in chamber A is powered on and the channel used by the DUT is recorded for both 2.4GHz and 5GHz.