Cease and Desist!

My ISP (Speakeasy) sent me a nice letter recently informing me the Eurpopean Union's copyright infringement division was displeased with me. The said that based on these allegations, I would be in violation of the Speakeasy Acceptable Use Policy. "How can that be?", thought I. I buy my music on iTunes, I do not partake in bittorrent, limewire or any other version of the now dead Napster (old school version not the new subscription based system) music/file-trading system Hell, I pay for stuff!. I have encoded all my purchased CD's and boxed them away but I keep them to myself. In fact I am a true supporter of "legitimate" digital music use via iTunes or any other service that, in some way, supports the artisits that create the music I love. This includes freely distributed music a la Radiohead.

So why was the European Union comin after me? Speakeasy's Tech Support and Security groups were very helpful in pointing out to me that they could track streams of file sharing originating at my IP address. So I thought deeply about this (for 2 seconds) and arrived at the most logical conclusion. My neighbors were connecting to me via Free The Net, the Meraki based San Francisco free wifi network and uploading/downloading music to their hearts content. I have 2 repeaters on my roof and 4 others in houses nearby providing firewalled access to the Internet. This made me sad. I was very pleased to provide an un-asked for service to my neighbors who may not have - or may not be able to afford - Internet access. I wrote to Meraki explaining my dilemma and asked of there was someway to restrict my neighbors from conducting file trading on my network.

People in my hood are sharing music over my wireless network and
abusing my speakeasy acceptable use policy. Speakeasy.net has warned
me that any continued abuse will result in disconnection of my
service. Therefore I must inform you that unless you can lock it
down so only port 80 is being used I will have to disconnect the
Meraki repeaters and access points from my network.

I am very sorry. This seems like a real shame. I was very eager to
participate in "Free the Net" but now I am a bit saddened that folks
are abusing it.

Please get back to me and let me know if there is anything you can do
on your end.

They replied back with...

Hey Bruce,

grr. that's really irritating. but actually what's surprising is that
we haven't had to address this issue so far. as far as blocking
everything but port 80: I don't think any of us would be happy with a web-only Internet connection, so that doesn't seem like a good answer. to me it seems the real solution here would be to figure out who the culprit is and block them.

I looked on your gateway and didn't see anyone transferring an
inordinate amount of traffic. do you happen to have any idea who it is? do you know if it is bittorrent they are using? maybe they are using a different gateway at least part of the time (probably mine, hehe).

next week I guess we can figure out how to set up the right counters on your gateway so that we can figure out who it is (any insight or additional info you can provide would obviously be super helpful). hopefully Speakeasy can wait that long. if you need to unplug, we understand, but leaving your repeater plugged into power would at least soften the blow.

ugh,

So far they have found no way to track or stop the activity and I love my Speakeasy service. So I have no choice. Until such a time as I can trust my neighbors not to conduct activity that the European Union deems as illegal or until Meraki finds a way to filter this traffic out, I must disconnect my network from "Free the Net". I still have repeaters on my roof but they are no longer connected to my network, file traders now siphon off some other guys pipe or tube or truck that backs up and unloads Internet.

Comments and suggestions, as always, are very welcome.

Al Gore: New thinking on the climate crisis

Probably somewhat off topic, but I am a fan of Al and I aggree we are at a crisis crossroads. Time to get busy.

WiFi Enabled Bag!

OK, Admission. The previously lauded WiFi enabled Shirt SUCKED! It had this big plasticky velcro'y patch on the front. I just couldn't wear it. I thought it would be more integrated. So...

My wife the genius (BTW, she is a geek, too. She really wanted me to order her one. So I did). She takes one look at her shirt and says, " I am not going to wear this but I have an idea."

She tears the shirt apart and buys herself a new bag/backpack/purse. She cuts a hole in the front of the bag for the cable. Sews the felt backing used to secure the velcro to the front of the bag. Puts the battery pack and cable into an internal pocket and away we go. WiFi enabled bag!!!

This is way more cool than the lame shirt. Now she carries this to work proudly, letting all around her know if they can surf via WiFi or not.

WiFi enabled shirt?!!!

As if some geeky clothing designer and an Uber-RF-Nerd had a dorky lovechild, ThinkGeek has launched the first WiFi enabled SHIRT!!!

I am not kidding. this a a wearable WiFi detector that illustrates the strength of the WiFi signal with glowing bars and antenna and the words, "802.11" underneath.

I cannot wait to wear this to the next, "I can't get a date with nobody/Star Trek/D&D/comic book/video game/networking convention"

Look, Drool and count the days till it is released.

WLAN IDS and the bizarre world of security exploits

If you make security software (or any software, for that matter) sooner or later you will create what I technically refer to as a booboo. A security vulnerability in your software that raises the ire of your customers and make you feel foolish and sad. Not to worry, mateys, this happens to all software manufacturers. The important thing to remember here is how you handle it. Are you going to be a Pro or a shmuck? Recently, AirDefense (why no dot com?), a WLAN IDS manufacturer had just such and incident. Is this uncommon? Relatively so. Is it dire? Not really. Are you just sniping at your competitor? Kind of, but in the interest of disclosure, we had an incident a long time ago as well so, dear friends, I feel their pain.

Let's talk about what happened first. The vulnerability as explained here happens when you send a specially crafted HTTPS request, which will cause the HTTPS service on the system to crash. It appears from my quick glance as if you need to authenticate first and also be on the segment from which you can administer the system. So what is this? Granted it can bring down the sensor but actually it appears to be a "tempest in a teacup". You need to be the admin or snarf the admin login in order to cause a denial of service to one of probably many tens or hundereds of sensors. Unlikely at best.

So how was this handled? Professionally, in my humble opinion. AirDefense contacted the people who reported the exploit and directed them to a patch for it as reported here, "Solution: Update to the latest firmware version"

AirMagnet had a similar experience Last October. And we handled it the same way. Here is our official response to the problem from back then:


Re: Airmagnet management interfaces multiple vulnerabilities
AirMagnet vendor response below -

(1) The vulnerabilities are tested against an over-a-year old AirMagnet Enterprise product,
(2) Some of these vulnerabilities have been patched and fixed in AirMagnet Enterprise version 7.0.x,
(3) All vulnerabilities are now completely fixed by AirMagnet Enterprise version 7.5 build 6307 and later.
(4) AirMagnet customers can download patches from MyAirMagnet support web site (http://www.airmagnet.com/my_airmagnet/index.php)

So to summarize, there are a lot of security professionals out there who are trying to make a name for themselves and do it in an industry, like the WLAN industry, that is going places. They spend all their time looking for these exploits and I, for one, am glad they do. They keep us honest and ensure that we are doing our very best to protect our customers. Are their motives pure? Debatable but mostly. Do they sit down afterwards and talk amongst themselves about what l@m3rz those software guys are? You bet! Should I take it personally? Nah.

The Myth of the Self-Monitoring WLAN

Recently, as you all probably know by now, Duke University had a WLAN meltdown. The CIO, Tracy Futhey (Comment here) and the assistant IT director, Kevin Miller (Comment here) have put to rest the notion that the Apple iPhone caused it. Cisco has issued an advisory to the effect and Apple assisted in the effort.

I am not going to go into the details of what happened or why. Suffice it to say that mobile handhelds of all types, not just iPhones, send a lot of ARP traffic and the Cisco infrastructure was not ready for it. The quote at Network World explains that, "The advisory finally makes it clear that the iPhone simply triggered the ARP storms that were made possible by the controller vulnerabilities. Any other wireless client device, moving from one subnet to another apparently could have done the same thing."

What I will point out, however, is the problem we in the Wi-Fi community have today with the following simple delusion, "Your WLAN infrastructure as a cohesive, integrated, single-vendor solution is all anybody needs. It is self monitoring and self healing." I talk to a lot of people about which WLAN solution they are going to purchase and implement and I am always surprised by how many believe that the AP and controller vendor has all the answers. Don't get me wrong, I am a huge fan of this type of solution. Central management is critical for even medium sized organizations of 50 or more APs, much less larger ones that may a few hundred or even thousands. Manually changing the configuration of each AP is not a viable solution in these cases. The Admin needs assistance. And the story sounds so great, "Implement our solution and it will fix itself when it breaks and protect itself when security policies are breached." Who wouldn't want that?

But the truth is a little more complicated. As we have seen from previous posts, sometimes the solution doesn't behave the way your business practices need. Similarly, sometimes there are security problems within the infrastructure itself. So what to do?

This will sound like an advertisement for the company I work for and I apologize ahead of time but there is a very good reason I continue to work there. Mainly, I believe in the message.

When the Duke network went down and the Assistant IT director looked at his WLAN infrastructure dashboard, what did he see? I have not spoken with him directly but my guess would be it said, "hey man, it ain't me. Everything looks good from my end" So what did he do? he pulled out a sniffer and got to work. With packet traces in hand and assistance from Cisco and Apple he solved the problem. Did the infrastructure fix itself? Did it correctly identify the problem and solution? No. A patch is now needed to keep this from happening again.

One should not blame the infrastructure for not getting this right at the outset nor should one blame Mr. Miller. He was correctly reading what the controllers were telling him. But it shows how important it is to have a separate, 3rd party solution also available to get down to the bits and bytes or even spectrum analysis (if the problem should be something other than 802.11 protocol madness.)

There are a few great WLAN security vendors out there and they make 3rd party, best of breed solutions for monitoring the security of your WLAN (one of which recently got snatched up pennies on the dollar and will probably be rolled into another integrated, self-healing, self-monitoring role; against my better judgment.) There are an even smaller number who both monitor your security and your connectivity and performance and give you great troubleshooting tools built-in (insert shameless plug here). These should be your trusted advisor's when things go wrong. I am in no way suggesting that they would have identified the problem and cause and given a solution at Duke either (although I think they at least would have shown alerts for denial of service and strange traffic behavior.) What I am suggesting is that with them in place you now have a set of tools to assist in solving the problem. Remote packet and/or spectrum analysis. Alarm thresholds that can be set by the admin and will continue surveillance. Reports. System-to-system notifications. Graphs of speed and traffic type. Lists of who is connected to what and how. All the things you would need to get to the bottom of any problem in that invisible Luminiferous Ether.

Cisco Ripples - DCA and RRM - Help is on the way

Since I first published " The Ripple Effect" back in February I have heard from many folks who have validated the effect but to my chagrin, I have had no solution to offer. Well thankfully there are smarter people than me out there and solutions have started to appear.

I was alerted to the fact that Medical Connectivity consulting recently put Cisco in their sights and quoted my blog with regard to Dynamic Channel Assignment and RRM causing issues. The Web, being the great time waster that it is, lead me on a journey. As I read the article I clicked here and there and next thing I knew I was looking at a forum at Cisco that was talking about this exact phenomena.

One of the forum posters had some great suggestions to eliminate this problem in the future. Bruce Johnson at Partners Healthcare offered this solution,

"We saw the majority of DCA events were triggered by Interference from Rogue APs. After we disabled Foreign AP Avoidance the number of channel changes dropped by an entire order of magnitude (1000s to 100s). We disabled Cisco AP Load Avoidance and this reduced the number of DCAs within an order of magnitude (100s less).

DTPC will power-up APs to max levels to provide a 3-neighbor -65 RSSI coverage "grid" and 7921s will power up to follow suit (up to their max Tx Power). Other clients with higher Tx power may send the APs to max power causing a mismatch with IP phones.

You can decrease the tx-power-threshold so the "grid" won't be as hot (default is -65, change to -71 or -74):

config advanced 802.11a tx-power-control-thresh <-50 to -80>
config advanced 802.11b tx-power-control-thresh <-50 to -80>

and reduce the coverage hole detection threshold (reduce Min SNR level in RRM Thresholds) to suppress the power-up activity."

Bruce seemed on track with this fix. the problem is that it isn't a fix. It shuts off the RRM and DCA so that the WLAN would remain stable. So where is the benefit of a controller based system?

He does note that a fix is forthcoming from Cisco, "They are revamping the behavior of RRM in the WLC 4.1 Maintenance release." Which is later confirmed by a Cisco employee, Saurabh Bhasin a TME,

"With the 4.1 Maintenance Release(MR) due out on cisco.com shorly, many improvements based on such feedback have been brought into RRM's algorithms ? improvements aimed at allowing administrators to fine-tune their RRM-run WLANs where desired. These enhancements will allow for greater control over both the channel and power output selection algorithms, so administrators may assist RRM in being either more or less aggressive in such decisions, depending on application and network needs. Additionally, enhancements have been made to the management and reporting of all RRM information and configuration alterations to allow for better tracking of RF environmental fluctuations and to assist in keeping track of RRM activity. Further technical detail on the inner workings of these enhancements will be available very soon in an update to the above-mentioned RRM Whitepaper."

The paper he references is found here http://www.cisco.com/warp/public/114/rrm.html and explains a lot of what we are all seeing. (here is the PDF version)

(NOTE: Since publishing this post, Cisco has moved the link. Here is a more recent version. Please double-check with Cisco that you have their latest information)

So here is to hope that WLC 4.1 Maint. Rels. fixes it. As an aside, Bruce Johnson is skeptical,

"Its all well and good to make things work for Intel and the CCX/CCKM compliant crew, but if you have any of the other brands of WLAN NICs (like those made by medical device manufacturers, who won't subscribe to fast roaming features until they're adopted by the IEEE) you are best keeping RRM disabled until it delivers on its promise as stated in the following 802.11TGv Objectives draft:

Service and Function Objectives

Solutions shall define mechanisms to provide the service listed below.

[Req2000] TGv shall support Dynamic Channel Selection, to allow STAs to avoid interference. Solution shall be able to change the operating channel (and/or band) for the entire BSS during live system operation and be done seamlessly with no intermittent loss of connectivity from the perspective of an associated STA. Solution shall not define algorithm for channel selection."

Review of Nokia N800 for Wireless LANs

Bruce asked if I'd write up a review of my experiences with the Nokia N800 - here you go...

Earlier this year Nokia released an updated version of their popular Nokia N770 Internet Tablet. The N800 excels at being a small, light-weight, device capable of WiFi as well as Bluetooth access to the Internet.

I could go on an on about how this little device has changed how we access the Internet in our home. Instead of lugging around laptops, we have a couple of N800s that the kids access for Internet queries, (just what High School did ‘King James’ go to – while watching the playoffs), to googling, to streaming music, conducting mulit-user IM sessions, to just about anything you’d want to use the Internet for – but in a very small package.

The screen resolution is amazing! If the iPhone has this screen, I’ll be in line to buy one. Even though I’m so old now I have to use bi-focals to read the small stuff – I’m comfortable with the N800’s little screen – the resolution makes it possible to fit so much in a little package.

But for the interest of this Blog I’ll move on to the use of the N800 in a wireless analysis mode.

I saw an announcement for a new piece of wireless security gear – called a Silica. This software/hardware bundle puts the intelligent penetration attacks and exploits of Immunity’s Canvas software in a small Nokia. It looked WAY COOL and I *wanted* to have one. But the $3,600 cost felt a bit prohibitive.

I’ve since had a chance to play with a Silica – and was suitably impressed – I still haven’t parted with the $3,600 but here’s a bit of a review of the tool.

This is a customized version of the Canvas tool – shoehorned into this small form-factor Linux device (Nokia N800) It is VERY easy to use. Just turn it on and click the start scan…

It will run through a series of scans of the local wireless networks, then attempt to penetrate using a variety of currently known exploits to find and exploit holes in your wireless LAN. It’s like having a little team of hackers sitting in you hand.

I’ve found it to be easy to run with the Nokia in your pocket. Very unobtrusive! – but in reality it takes up to 20-25 minutes to do a full attack against a single AP. Not like while doing a real penetration test you’ll have an excuse to ‘hang around’ a specific area waiting for the attack to finish. (Though you could easily hide the device and come back to pick it up later – but that $3,600 cost will probably make you think twice about leaving it outside of your view)

The reports it gives are in HTML format – you can just e-mail them to yourself, or copy them off onto the SD cards used by the N800

It does a pretty cool thing when attacking a MAC-Filtered AP – it automatically finds an associated STA and spoofs it MAC address to get associated.

It doesn’t do ANY WEP Cracking or WPA cracking, or anything but the exploits that are in Canvas.

Ok, now for the less expensive, yet still fun stuff using a Nokia N800. As part of our Wireless LAN Security Assessment Toolkit course development. We came across the N800s, fell in love with them, then re-arranged and re-wrote many of our course lab exercises to specifically use the N800s. We added wVoIP, video over IP, as well as catching IM traffic, web browsing, and other conversations sent over wireless to use the N800 as our client of choice when ‘watching’ the open Wireless LANs and re-constructing conversations via packet capture.

Then we thought, “is there anything more we can use the N800s for”?

Since the N800 is just a little Linux computer… we added SSH, Terminal Shell, VNC, FTP, etc. to the system. Then once we got that running, it was a quick couple of steps to get Kismet and Metasploit running!

So just using Open Source software we were able to take the little $400 Nokia N800 and make it ‘like’ a Silica!

Just slip this little ‘bad boy’ in your pocket running kismet and go WarWalking to get all the APs in your area, including finding ‘hidden’ SSIDs. Or, start Metasploit and let ‘r rip – attempting whatever known exploits are available for Metasploit.

(By the way – this is NOT a fast device for Metasploit – but Way Cool to have running)

On the other hand – it makes a great Kismet platform!

We’ve got ours running the classroom with GoogleTalk and Gizmo Project for wVoIP and IM – but you have to have a Wifi access to use it, so this wont’ take the place of your cell phone. (It is possible to pair this device via Bluetooth to a cell phone running G3 speeds – but the easy way is with Wifi)

One more thing that is just *fun* to use the Nokia N800 and Wireless LANs – and I found this out by accident. I was testing in our offices a new access point – so I had the N800 associate to this new AP and started a ‘Hitcast’ session listening to some Internet Radio station. I kind of liked the station and so put the ‘radio’ in my pocket so I could have some tunes with me as I finished up writing up the analysis of this new AP. I checked my watch and realized the mail would have arrived, so I went out to the mailbox – down a long flight of stairs, outside the steel-sided building my office is in, and across the parking lot to the community mailbox to check the mail.

It wasn’t until I was heading back up the stairs and a co-worker commented on my choice of music that I noticed this little wonder continued to stream music the entire way. Thus was born the Audio Site Survey! Just associate, then walk till the music drops… Simple easy and leaves your hands free to work on other things while doing a fairly decent site survey. Cool!

There you go – a quick review of the Nokia N800 and how one might use it with Wireless LANs – if you want to see more about our Wireless LAN Security Assessment Toolkit class, check it out over at www.hotlabs.org/wlsat - as part of Bruce’s Blog you can use the discount code ‘Bruce1071’ to get $1,500 off the class.

Enjoy!

Keith Parsons - Managing Director

Institute for Network Professionals

Ripple Effect - Redux

This article is out of date.

Early in the year I posted an article about how the Cisco WLAN controller system may behave strangely in some conditions. I got some email from some folks that had major issues with it. One poster said that, "Before Cisco purchased the technology from Airspace, they had already put dampeners in the RRM so the hysteresis you describe wouldn't occur." This is just plain wrong. Cisco wants to sell more switches and routers and they found out if they purchased the Airespace system they would do just that but they did not make this significant change before releasing it with their name on it. And they are still changing the behavior of the WCS today because this problem still exists.


Did I lose you? As a refresher for those who did not see the original article it is posted HERE.


Since I published that comment back in early February I have spoken to quite a few people who have seen the same effect in their environments in recent months. One network engineer wrote, "I can vouch for having observed this recurrent DCA behavior, also in a hospital environment (12-24 channel changes per day across 10 floors of APs, as you depict in your example). The architecture is not alerting us to this being the result of interference or noise (no WLC or WCS events of either type), and the RSSI of rogue APs is above the threshold required for triggering DCA (neg 85dB)."


I was asked by the nay-sayers what Cisco told it's customers to do and here is what that same engineer said, "We have been told by Cisco that the 100mW AP neighbor beacons, used to determine the picture of the network, does not get input into DCA. Cisco claims these 100mW beacons are used only for dynamic power control, which we hold static -- do you think this voids the dynamic algorithms? Other docs say the RSSI of neighbor APs is the most important criterion in DCA behavior! In lieu of noise and interference alerts we can only surmise its the APs themselves that are the cause of their own DCA ripple effect."


This is just one example. I also have spoken to other folks who say that the Aruba system they are running does not do this. They say it is much more stable and after the original "learning" time it settles down and stays that way as long as the network is in use. I think this makes sense, why change the whole network because of one interferer? Better to be alerted to the fact and deal with it yourself.


I am collecting comments on this and would like to post more testimonials about this effect. If anyone wants to support this claim publicly, please feel free to drop me a line to bruce@hubbert.org or comment to this post. My goal here is not to raise hysteria but get things fixed and level the playing field. The infrastructure vendors tend to pitch the idea that they offer a panacea for all wifi woes and I feel that that is just a flavor of "Kool-Aid" I am unwilling to drink.

Meraki - Staunton, VA

Another great grass roots wifi project is being lead by a group of folks out in Staunton Virginia. With 19 nodes up as of today, the 3rd of May, it looks like it is doing well.

Again, I must say, citizens and grass roots efforts beat out commercial or governmental efforst all the time. Less bloat, less waste. These efforts are similar to the way we as world citizens take it upon ourselves to act responsibly on the road, or by taking the effort to recycle our bottles and cans. It is also parallel to the user created content wave sweeping the Internet. Not only do we want to get news and opinion our own way we want to get services our way too. And just like we don't mind spending some of our time to create that content we do not mind spending some our time and bandwidth to add wifi services.

Drop in on Staunton VA and take a look at http://www.stauntonwifi.org/ and let them know what a great job they are doing!