Categories
field notes troubleshooting

Non-wifi interferers

I was analyzing a set of surveys and found some interesting interference sources. Except for the first three, they are all above 5.725 GHz which is home to the U-NII-3 (5.725-5.850 GHz) and the 5.8 GHz ISM band (5.725-5.875 GHz).

Feel free to comment if you have seen it in the wild.

This looks like Godzilla, consistent interference in a wide area, source unknown.

Possibly some video surveillance system, there are two in the area, see next.
Possibly video surveillance systems

Movement sensors for lighting fixtures
Movement sensors for lighting fixtures
Movement sensors for lighting fixtures

Strong interference spilling over from U-NII-3 up to channel 136. Source unknown.
This interferer is spilling up to channel 132. Unknown source.
Different site, similar interference pattern.
Yet another site, same pattern with harmonics reaching channel 140.

Possibly a non-wi-fi security camera on channels 60-64.

Categories
field notes troubleshooting

Bruker NMR interference?

I was called to discuss the installation of an access point in a chemistry research facility where a Nuclear Magnetic Resonance (NMR) spectrometer by Bruker is operating. It’s a model slightly smaller than the one pictured below.

Credits: Mike25. Source: Wikipedia

Researchers operating the laboratory were concerned about the signal from a wi-fi access point in the same room possibly interfering with the instrument. My concern was in turn that the NMR would interfere with the wi-fi operations, given the much higher energy in play in an NMR spectrometer.

Upon discussion with the lead researcher, it was ascertained that the spectrometer works with RF below 1 Ghz, mainly in the 400 Mhz range. According to Wikipedia, it operates in the 60-1000 Mhz range.

Interestingly, a spectrum analysis with Metageek Chanalizer in the lab showed a distinctive interference on 2.4 Ghz centered on channel 2 and a lesser signal on channel 1. The measure was taken from a fixed position at 3m from the spectrometer, while it was operating.

The signal on channel 2 has a high duty cycle with 95% at -50dBm in a 20 minute interval.

At -70 dBm the duty cycle on channel 1 was above 80%.

Moving away from the NMR into adjacent rooms, the interference weakened and wasno longer detectable. I was not able to identify the exact source of the interference, besides it being related to the Bruker device operations. The access point was moved out of the room into the corridor.

Categories
troubleshooting

Disable AP status light in Extreme CloudIQ

This is so stupid and still not so easy to find out: to turn off the LED of an AP in Extreme Networks CloudIQ, go to Manage > Devices, select your APs, click on Utilities > Tools > Locate your device.

You will be presented a dialog to switch-off the light or start a blinking pattern. https://extremeportal.force.com/ExtrArticleDetail?an=000080072

This feature is meant for troubleshooting and device location, when you are on-site and don’t know where a specific AP is. Just configure a blinking pattern on XcloudIQ and see which AP blinks.

If you place APs in hotel rooms, student dorms or home bedrooms, you MUST turn the LED light off. Otherwise, people will cover it with chewing gum, stickers, tampons or anything at hand.

Categories
troubleshooting

Validating DFS in Extreme AP410C

Following my previous post on DFS, I’m testing a modern Extreme Networks AP410C with the Wifimetrix tool.

AP410C is a new cloud managed wi-fi 6 access point evolved from the Aerohive product line, now acquired by Extreme. Tested model: AP410C Rev: F, OS 10.1.3.0 (10.1r3).

As usual, the AP is on a DFS channel, Wifimetrix sends a simulated radar wave, and the packet capture is analyzed.

Test: channel switch

After detecting a simulated radar signal, the AP does not send a channel Switch Announcement action frame. It sends a beacon frame containing a Channel Switch Announcement information element with channel switch mode 1 (clients are not allowed to transmit during the switch), destination channel, count 8 (AP will switch after the 8th beacon is transmitted), last transmitted beacon count 1 (AP switching immediately before the next TBTT). This behavior is consistent, either with and without client stations associated to the AP.

AP410C on channel 52 detects a radar and sends CSA in beacons. It then switches to channel 64. See pcap for details.
Channel Switch Announcement IE in the beacon.

Test: channel switch-back

The switch back to the original DFS channel happens after about 30 minutes and does not use any Channel Switch Announcement to inform the associated stations of the new channel.

AP410C on channel 64 switches back to channel 52. See pcap for details.
AP410C on channel 48 switches back to channel 52. There are no CSA action frames nor CSA IE in beacons. See pcap for details.

I observed that when stations are associated to the AP, they keep transmitting on the same channel after the switchback, sending probes, retries, and eventually disassociating. I had a Windows10 laptop and an iPhone associated and streaming video (Youtube), both lost connectivity and did not reconnect to the original channel. Zero-wait DFS was enabled on the AP.

Impact assessment of channel switch

With AP410C it takes about 918ms (9 TBTT) from an initial radar detection to the move to a new channel, during which clients are not allowed to transmit (mode 1). This is way more that the maximum 150ms roaming hand-off time required for VoWiFi. I can figure out 2 scenarios:

  • The client station remains associated to the AP, the voice call will suffer degradation during the channel switch.
  • The client station decides to roam away to another AP, therefore voice calls and videoconferencing are not disrupted (provided FT is enabled or PSK is used).

Which of the two scenarios is more likely depends on the roaming decisions made by the client (if roaming is feasible at all, that is).

The disruption from a radar-induced DFS event is noticeable, but very limited and quickly recovered.

Impact of channel switch-back

After a defined amount of time on the new channel, the AP switches back to the original DFS channel without any announcement to the associated stations. There are two scenarios that come to my mind:

  • The AP implements Zero-wait DFS or a similar mechanism: a client station on the channel fails to receive frames from the AP (beacons, ACKs) and either:
    • somehow it discovers the AP on the new channel and moves, keeping the original association;
    • or it roams to another AP.
  • The AP performs a Channel Availability Check (CAC) before transmitting, the client station roams to another AP.

It is unclear to me how long the roaming will take, considering the time spent by the station sending retry frames, probes, listening to beacons on different channels. I tested an SSID on a single AP and experienced complete loss of connectivity.

My personal impression is that the channel switch back is even more disruptive to service than the initial radar-triggered channel switch. Where CAC is performed (without Zero-wait DFS mechanisms), service from the original AP is disrupted for 30 seconds or more.

Final thoughts

Mitigating the impact of DFS events seems to focus on roaming. Careful roaming design is called for, including secondary converage, 802.11k (which should be enabled anyway if DFS channels are used), Fast Transition and 802.11v.

Categories
troubleshooting

Validating DFS in Extreme AP 3900 series

I tested the DFS implementation in Extreme Networks legacy AP 3800 and 3900 series. Unfortunately, it does not implement channel switch announcement.

Upon detecting a radar signal, a legacy Extreme AP 3900 complies to domain regulation and leaves the 5Ghz channel where radar was received. It does so by sending deauthentication frames to the BSS and moving straight to a new channel. There is no Channel Switch Announcement in action frames or in beacons. The new channel number however can be defined in the controller configuration.

Affected devices

This issue affects legacy, controller-based 3900 and 3800 APs from the ExtremeWireless/Identify product line. It does not affect the recent ExtremeWireless cloud AP line following the Aerohive acquisition.

The following models were tested:

  • WSAP3935i-ROW firmware 10.51.170006
  • WSAP3915i-ROW firmware 10.51.170006
  • WSAP3912i-ROW firmware 10.51.170006
  • AP3825 firmware 10.51.170006

3900 are 802.11ac Wave2 APs that went out of sale in December 2020. They are supported through 2025 on the latest Extreme Campus Controllers (ECC).

3800s are an older 802.11ac Wave1 AP line, still supported until 2023 on legacy Extreme controllers.

I have not tested the 3900s from the WING product line. As far as I know it’s exactly the same chipset. If you are a WING user and use DFS channels, you should test.

Background

During a CWNA course in 2019, Devin Akin mentioned DFS validation and the wide differences in DFS implementation by vendor and by AP model. The DFS Project website has a good introduction to the issue and several example packet captures you can check out. The DFS Project lists a set of questions that the validation process should answer:

  • Does my AP detect DFS events of various types?
  • If a DFS event is detected, does my AP move to a new channel? If so, how long does that take?
  • Does my AP announce the Channel Switch Announcement (CSA) in beacons, probe responses, and action frames? If so, for how long?
  • If a DFS event happens, how does it affect application performance on my devices?
  • When my channel changes, where does it go?
  • Does my AP return to the original channel? If so, after how long?
  • After a channel change, what happens to the channels on the other APs in the area?

The Channel Switch Announcement information element is found in action frames and beacons. It provides a graceful transition mechanism to a new channel, while keeping the client stations associated. The clients are informed of the channel number and the move time. If the destination channel is U-NII-1 U-NII-3 (non DFS) the service disruption is minimal.

Tools

I used the Wifimetrix handheld wi-fi tester, which I finally purchased this year. It does many useful WLAN things, my favorite being the FCC radar wave simulation. I am under the ETSI regulatory domain, so the Wifimetrix manufacturers kindly provided me this FCC-ETSI equivalence table:

FCC ETSI
0   1,2
1   1,2
3   3
4   3

Test results

I configured Extreme APs 3935, 3915, 3912 and 3825 on a DFS channel on 5Ghz, and started a packet capture. I simulated a radar event on that channel with the Wifimetrix and analyzed AP behavior in the capture.

Leaving the channel

Upon detecting a radar signal, the AP sends broadcast and unicast deauthentication frames with Reason code: Deauthenticated because sending STA is leaving (or has left) IBSS or ESS (0x0003). It then leaves the channel for good.

The destination channel can be defined in the AP configuration as custom channel plan (legacy Identifi controllers) or DFS fallback channel (Extreme Campus Controller). It should be a non-DFS chanel so that the AP does not have to perform a Channel Availability Check (CAC) and can start transmitting immediately.

A 3935 AP on a legacy Identifi controller on channel 60 detects radar and sends a broadcast deauthentication frame. See pcap for full details.
The above deauthentication, with reason code
A 3915 AP on a legacy Identifi controller on channel 56 detects a radar and sends broadcast and unicast deauthentication frames to each of the connected clients. See pcap for details.
A 3912 AP on a new Enterprise Campus Controller, on channel 100. A radar event triggers unicast deauthentication frames. Note the beacon transmitted after the deauthentication. See pcap for details.
A 3935 AP on a new Enterprise Campus Controller, on channel 104. A radar event triggers unicast deauthentication frames. See pcap for details.
A legacy 3825 AP in channel 52 detects a radar signal and sends broadcast and unicast deauthentication frames. See pcap for details.

Switching back to the original channel

I have not tested this behavior fully, this are just my preliminary results with a 3912 on ECC. After about 1 hour the AP simply leaves the channel without any deauthentication of CSA frames.

A 3912 AP on channel 48 leaves and returns to channel 100. See pcap for details.

It is very likely that the AP will do a channel availability check before resuming transmission in the original DFS channel, but I have not tested it yet.

Impact assessment

The DFS mechanism implemented in this APs effectively disrupts service to the client stations in the basic service set.

Leaving the channel

When the AP detects a radar signal it send deauthentication frames to the BSS. After being deauthenticated (and therefore disassociated), clients cannot use roaming mechanisms to move to another AP. The client will choose to associate to the original BSSID, to the 2.4 Ghz BSSID from the same AP, or to another AP.

The clients must discover the new channel (or a new BSSID) by themselves, either listening to beacons, sending probes, or using information from the neighbor report if 802.11k is implemented. Clients must then do authentication, association, PSK or 802.1X authentication.

This will break voice calls and realtime audio/video conferencing. Buffered multimedia streaming may or may not be impacted, asynchronous traffic such as web and email may not notice any interruption.

Since the fallback channel can be defined, there is some room to ensure that the destination channel is U-NII-1 in a contingency channel plan.

Returning back to the original channel

The switchback to the original channel seems to be totally silent. Client stations won’t receive ACKs to their frames, beacons will be missed, and the clients will eventually move to another channel or BSSID. In addition, CAC will be performed by the AP on the original DFS channel.

This means service disruption for all kind of traffic. The only upside is that clients will keep their association, so that after learning AP information via beacons, probes or neighbor reports, the client can roam to another BSSID. If 802.11r is enabled, roaming may be fast and almost seamless.

Impact on channel plans

Deployments with low AP and user density, in domains where U-NII-1 and U-NII-3 channels are available, should exclude DFS channels from the channel plan.

High density deployments with potential high co-channel interference should carefully evaluate the requirements and choose a trade-off between service availability and airtime:

  • sticking to non-DFS channels will prevent occasional service disruption caused by radar events, but due to CCI the airtime % use will be high and the service will be severely degraded permanently.
  • including DFS channels in the channel plan will minimize CCI, at the price of the occasional disruption caused by radar evens (real or false positives).

Whichever you choose, it is important to state the issue clearly to the organization management. Users should be informed of the service level they can expect from the wi-fi network.

Mitigation

Extreme Networks in their support documentation recommends to monitor the occurrences of DFS events, and exclude the frequently offending channels from the custom channel plan on an AP by AP base. There is no dynamic RRM for DFS issues in 3900 APs, only an initial auto channel setup called ACS (automatic channel selection).

Other techniques that I have found helpful are:

  • Enable 802.11k to help clients finding their next AP. This is important even if there are no radar events, as clients cannot do active discovery on DFS channels.
  • Plan 2.4 Ghz wisely. Clients will probably associate to the same AP on 2.4, if available.
  • Design good secondary coverage. If there are 2 APs in a lecture hall, configure one AP on an U-NII-1 channel and another on a DFS channel.

Long term solution

Unfortunately, the only permanent solution to this issue is replacing 3900 and 3800 APs with more modern models that implement Channel Switch Announcement correctly. This may be expensive, given the high number of 3900s installed and the longevity of the product. Management should be informed of the issue and involved in the assessment of its impact.

Final considerations

DFS events in the wild are difficult to capture and analyze. Vendor datasheets and documentation may not go in such details. You should always validate the DFS implementation of your APs, you could be in for a surprise. Buy a Wifimetrix and go testing, or have a WLAN professional do it for you.