Real World Troubleshooting Scenario #276,460

I was sent onsite a few years back to troubleshoot an issue that our support team had been working and it was evident that we needed more data than what we were being given by the facility. The issues were that the proprietary devices were having issues with choppy audio and the iPhone devices used an app that would work intermittently as well.

When I arrived on site, I start trying to observe the issues and gather as much data as I can. The facility appeared to have enough access points to provide coverage, the initial transmit power setting were the Cisco default of -10dBm and 30dBm, which is a usual suspect in any troubleshooting issue. Meanwhile, I'm trying to capture pcap data to see what the devices are doing. One of the very first things I noticed is that the proprietary devices are roaming entirely too much, even while stationary. Everything was pointing to low coverage in an area with good, visible coverage. This led me to ask for the entire sh running-config output from the Cisco controller. Sure enough the transmit power for the access points was mostly between 2 and 5dBm, which I thought to be odd since usually, the default power settings tend to error on the too high side rather than the too low side. Here is the I/O graph from that first observation:

Figure 1. Example of Excessive Roaming Due to Low Coverage

This graph shows 60 seconds where the device roamed nine times in an effort to find a strong signal and could not. Most of those roams were into access points with signals as weak as the ones they roamed from. This is due to the low coverage that was occurring because all the transmit power was so low. 

I recommended that they increase the power levels and this is what the same area looked like afterwards:

                                                                                                     Figure 2. Example of Proper Roaming

Now that the transmit power had been increased the devices roamed correctly and the audio issues vanished.

There was still the matter of the iPhone devices with the app on them. This actually was fixed as well, but I discovered that app had been set to be silent on cellular. The reason was that the caregivers went home with their devices and if the device was on cellular, that meant they weren't at work, so the app didn't work. The problem was that inside the building the access point signal strength was so bad, the phones would revert to the cell signal that might come through the windows of the facility, which caused the app to go dark. Increasing the transmit power kept the iPhones on the Wi-Fi network and resolved the issue.

There were some other issues because the access points were in the middle of the hallways and not using the existing walls to attenuate signal, so there were still some sticky roaming issues to overcome, but the functionality had at least been restored and a proper RF survey would need to be conducted to show where the access point locations should be. 

Meanwhile, I was able to resolve the issue and send the support team the update that allowed them to close their ticket, it wasn't a total win since there still needed to be some re-design that needed to be implemented but by being onsite and able to observe the devices, I was able to discover and resolve the main issue.