We have thousands of customers and a few dozen are collecting NetFlow or Cascade Flow from Riverbed Steelhead WAN optimization appliances. The balance is largely Cisco (e.g. Cisco WAAS) and a few other vendors like SonicWALL, Enterasys, Juniper, etc.

Latency is all the Rage

Latency is the new frontier in NetFlow collection and Riverbed is the 5th company that our IPFIX reporting tool includes latency trends for. The other vendors include:

• Citrix AppFlow / IPFIX
• Cisco Performance Monitoring / Medianet
• nProbe IPFIX
• SonicWALL IPFIX

Sadly, Cascade Flow is meant for Riverbed use only as they want their customers to buy their NetFlow reporting solution. What customers should be aware of is that Riverbed sets the NetFlow v9 header source ID to 0xbeef2002. They also use several element IDs including 100, 101 & 105 – 111 that Cisco may have already assigned to something else.

Riverbed Steelhead Problems

Since NetFlow is a Cisco technology, the above IDs Riverbed is using are not really available to be grabbed by a vendor other than Cisco.  Riverbed should be using IPFIX. When we asked Riverbed for the element ID translations we were told that they are proprietary and that they could not publicly disclose the format or the contents. We were a bit surprised to hear this as we had already signed an NDA and paid over $2000 to Riverbed to be an partner. On the other hand, I can understand that Riverbed wants their customers buying Mazu (aka Riverbed Cascade Network Performance Management).  Long term however, Riverbed Cascade problems might be on the horizon when it comes to working with other NetFlow collectors. This is the first vendor we’ve seen completely shut out other vendors from deciphering their flow exports.

We are Faced with a Question

Should we use the definitions for these IDs given to us by Cisco or label them with the names provided by Riverbed. Well, since Cisco owns NetFlow™ and Riverbed really should be using IPFIX for their unique flow exports like Sonicwall, Citrix, Plixer and nProbe plus, the fact that the large majority of our customers are Cisco focused, I’d have to say we are going with Cisco. But don’t worry, we are determined to figure out a way to create latency reports for Riverbed Cascade Flow exports. Contact me or plixer if you want to help us.

Riverbed Please Consider IPFIX

In the future, if Riverbed wants to continue using NetFlow v9 for Cascade Flow™, they should consider using values > 32767.  Even if they use IDs above 32767, they might still collide with another vendor but, it is less likely as there is a lot more room out beyond 32767.  Again, Riverbed could avoid all this by switching to IPFIX.

Riverbed Cascade Flow Provides Latency

Despite the above Riverbed issues, I’m sure everyone will appreciate these new Riverbed NetFlow exports especially when it comes to monitoring cloud services.  We commend Riverbed for jumping on board even if they refuse to share.

“To help admins know how their WAN is functioning, Steelheads can now export traffic flow information to NetFlow collectors.  I tested this using Scrutinizer from Plixer International and was impressed by how much information about my WAN was being captured and recorded.”

InfoWorld Media Group, Inc. (IDG Communications) – Keith Schultz

Riverbed® Cascade® http://www.riverbed.com/us/legal_notices.php Are the trademarks and property of Riverbed.com

IPFIX, NetStream, JFlow are all ‘NetFlow’ like technologies. These NetFlow technologies are truly ‘flow’ based.  On the other hand, sFlow is not.  It is a packet sampling technology. It has BIG benefits;  however, the benefits are very different from a flow-based protocol such as NetFlow and IPFIX. Lets take a look at the definition of a flow.

What is a Flow?

By definition: “A flow is a sequence of packets from a sending application to a receiving application.”

sFlow sampling does not sample packets in every flow.  It samples every X packet regardless of any flow. To the best of my knowledge, sFlow has no awareness of ‘flows’ at all. The way I understand it (which could be wrong) sFlow is not a flow technology at all, rather it is a packet sampling technology. In fact, many or often times most flows go completely unsampled with sFlow.  To be clear: sFlow doesn’t sample a packet within a flow, it samples every X (e.g. 1000th) packet on an interface.

For this reason, the customer jokingly suggested that sFlow be renamed ‘sPacket’. This is because sFlow is a technology where every X packet on an interface is sampled and exported off to a sFlow collector.  Multiple sFlow samples can be stuffed into a single sFlow datagram and sent off to the sFlow reporting tool.  The technology has its benefits however, lets clarify a few things.

sFlow Misconceptions:

• Misconception: sFlow is real time and NetFlow exports session information after a flow is complete.
  • Correction: NetFlow has an active timeout which exports a summary of active conversations every 60 seconds.  You don’t have to wait until the end of the flow.  However, with the Cisco ASA you do as it doesn’t support the active timeout. Everything else from Cisco and every other vendor I’m aware of supports the active timeout feature. BTW: sFlow is close to real time and likely won’t capture the packet you wanted nor was it intended to.
• Misconception: the most sFlow can sample is one in every 100 packets and it only samples the first X bytes.
  • Correction: with most vendors, this is configurable.  Foundry’s sFlow support can sample every other packet! If a single packet is large, it may be broken up and sent in a couple of sFlow datagrams.  So, a whole – single – very large packet can be sent in sFlow.  I think this is very cool about sFlow.
• Misconception: sFlow is hardware based.  NetFlow is software based and could severely impact the CPU.
  • Correction: This is partially true.  Many vendors (e.g. Cisco and Enterasys) have implemented NetFlow in hardware chips just like sFlow.  In my experience, 95% of NetFlow implementations have little to almost no impact on CPU utilization.
• Misconception: sFlow exports layer 2 information (e.g. MAC address) and NetFlow doesn’t.
  • Correction: NetFlow v9 and Flexible NetFlow export the MAC address as do several other vendors supporting NetFlow or IPFIX (e.g. Enterasys, Juniper, nProbe, SonicWALL and others). I should also mention that NetFlow can export VLAN information.  I wonder if sFlow export VLAN information?
• Misconception: NetFlow can’t sample traffic like sFlow.
  • Correction: NetFlow sampling is available and works fine.  It doesn’t however provide a counter export like sFlow which is a real bummer when trying to calculate total throughput.  NetFlow sampling could learn from sFlow here.

I deal with sFlow analysis a lot and I like the technology. The reason I wrote this blog is because often times people want their sFlow reports to be as accurate as their NetFlow reports and this isn’t ever going to happen. Here is a good blog on NetFlow vs sflow.  sFlow is what it is, a sampling technology and it is awesome to have around especially when there is no other option for gaining traffic insight.  sFlow can help save the day!

My sFlow wish

I wish the folks behind sFlow would improve the technology. What are the latest advancements in sFlow exports?  NetFlow is now exporting VoIP information such as Jitter, packet loss and Latency.

When to use sFlow

Enable sFlow on your switches and take advantage of this awesome technology. It can help you accurately determine top talkers, applications, etc. Be careful with the sample rate as it could overwhelm even the fastest sFlow collector.

When to use NetFlow

If you want to answer questions like the following with accurate results, I suggest NetFlow:

• Who is using DNS (i.e. port 53)?  What DNS servers are they connecting to?
• Who is using SMTP? What machines are spamming.

Thanks for reading.

New features in Scrutinizer version 8.5 include:

• Network Performance Monitoring (e.g. Jitter, Packet loss and round trip time) all with NetFlow!!!
• MAC Address and VLAN Reports
• SonicWALL NetFlow and IPFIX support (see videos)
• Juniper IPFIX Support
• Enterasys NetFlow Support for MAC addresses and VLANs
• nProbe IPFIX support: NetFlow probe and nBox
• Intelligent Template Recognition(TM) to display reports based on data available in the NetFlow Templates
• NetFlow Lite (NFlite) support
• PDF Reporting
• NetFlow billing module
• Many new NetFlow and sFlow collection features…

At one point, Cisco was demonstrating Scrutinizer’s support for the new NetFlow Lite (NFlite) technology. Can you imagine how excited we are to have Cisco demonstrating our software in their booth!

Below is a picture of Seth Cornwall (left) of Teneo with JimmyD (right). Teneo is the European distributor for plixer.

Several customers have already stopped by.  Please come by and tell us
what you are looking for in a NetFlow Analyzer.

A: Yes, depending on how you need to invoice.

Being a vendor that supports NetFlow and sFlow reporting, we deal with lots of flow questions.  Most recently, I was dealing with a customer that was trying to figure out how to do billing with sFlow.  Depending on how you want to invoice, sFlow may or may not be appropriate.  This document from Inmon sums it up nicely by saying, “sampling does not provide a 100% accurate result.”  The document goes on to state, “but it does provide a result in which the error can be accurately characterized.” Which is really a fancy way of saying that sampling allows you to be ‘fairly’ accurate.

In my experience, reactive traffic reporting with sFlow over a short period of time (e.g. last 10 minutes) when compared to NetFlow statistics is significantly different.  As much as half the entries in the TopN sampled by sFlow will be different from those accurately counted by NetFlow. If however you look at the data over time (i.e. not reactively) of lets say 24 hours, then sFlow starts to be more accurate.

One might argue to increase the sampling rate, however, this is often a poor choice.  Increasing the sampling rate causes more traffic over the network (i.e. often much more than NetFlow) and can overwhelm even the fastest collectors.

Q: Is sFlow better than NetFlow?

A: No

I agree with section 4 of the above PDF that overtime sFlow becomes more and more accurate, however, I take issue with the sentence, “Billing by lower bound of the confidence interval ensures that no customer is overcharged.”  My question is: Why not accurately charge? The answer is because you can’t with sFlow.

Also, in regard to the shortcomings of NetFlow for billing, the Inmon sFlow billing document states: “This method requires a variable, but significant amount of memory, especially under high load conditions, for example during a denial of service attack when every packet is a separate short-lived flow there may be 30,000 flows per second and the switch must export data rapidly to avoid flow cache overflow. In such a situation flow data will be lost.”

sFlow or NetFlow under Attack.

Depending on the attack, a 30,000 flow per second denial of service attack may end up being 1-20 flows exported in a single NetFlow packet. An over aggressive sampling rate (e.g. 1:100) on an sFlow switch would cause over 100 packets back to the collector.  This could cause a micro burst in traffic and lead to an interruption in my VoIP traffic.  My point is that this could be twisted in either direction and I feel the document is a bit biased.  Consider the author.

If you are trying to do accurate billing with NetFlow and are concerned about the volume of flows it will create, use Flexible NetFlow and export by subnet.  In my experience, smart implementations of NetFlow doesn’t  hammer the CPU.

Hardware has its Problems

Probably the biggest problem with sFlow could be it’s bragging point (all in hardware).  Cisco and Enterasys are both exporting NetFlow in hardware, however, we still need the CPU to export custom data types (e.g. interface names, counters like sFlow, etc.). By the way, NetFlow can be used to sample traffic!  Version 5 of sFlow has been available for years.  Where is the latest version?  Where is the NBAR like capabilities like NetFlow for deep packet inspection to identify applications (e.g. Skype, BitTorrent, PC Anywhere, etc.)?  Where is the ability to export things such as jitter, packet loss, round trip time, etc. for voice and video, like we see from Cisco Medianet and the nProbe?

Summary

I would limit sFlow billing to physical interfaces on the switch.  This would be accurate.  IMO: Trying to invoice based on IP addresses using sFlow is asking for trouble.  I don’t dislike sFlow.  I do dislike trying to hype it up to be accurate for something that NetFlow is far better suited for.  If your switches support sFlow, turn it on and take advantage of it, but don’t try to make it NetFlow by cranking up the sample rate.

I believe that sFlow sampling is awesome and it is great that so many switch vendors can implement it inexpensively and hence keep the cost down on their hardware. We have two sFlow switches on our network.

NetFlow is a thriving technology with the support of Cisco, Avaya, Alcatel, AT&T, IBM, Nokia, PSG, Qualcomm, and several others behind it and encouraging the emerging standard for NetFlow called IPFIX. The future of sFlow is guided by Inmon and that is about it.  Sure, lots of vendors implement sFlow chips, but how many are helping to guide the technology’s future?

These two technologies aren’t in the same league. SFlow is often a second thought feature on switches that implement it and it provides insight that SNMP simply can’t provide, it is easy for vendors to implement and its inexpensive.  Finally, everyone should know that none of these technologies are standards, not sFlow, not NetFlow and not even IPFIX.  However, look at the activity in the IETF IPFIX working group.   Clearly, IPFIX is the flow technology of the future for networking statistics and this includes sampling.

This month, I had the privilege of meeting with our largest UK reseller, Teneo.

I figured this was a great time to meet, considering the upcoming  release of Scrutinizer 8.0 and the new Mailinizer Exchange log monitor.  So off to England I went. I was staying in London, so I had to catch the tube to Reading. What a beautiful area, a quintessential British town.  The first part of the day at Teneo was the meet and greet. Being able to finally shake hands with Seth, Gemma, Lorraine and the rest of the energetic team was well worth the long plane ride and loss of sleep.

The rest of the day consisted of a sales meeting, support meeting, operations overview meeting, marking meeting, and setting up a time for more in-depth meetings when I get back to the States.

Don’t get me wrong, this many meetings in one day could put the strongest engineers to sleep.  But thanks to their energy and unique British sense of humor, the time seemed to pass by quickly.

As I mentioned before, having an intimate understanding of your reseller’s daily operations can be the key ingredient in solidifying a professional relationship. I have to admit, one of my favorite daily events is lunch. I was excited to see what this area had to offer.  So, for our lunch break, we went over to the nearby Fox & Hounds pub.  As a rite of passage I enjoyed the traditional fish-n-chips. It was all there, green peas, chips (French Fries)  and an unbelievably large piece of fish.  You can see from the picture we all had a great time.

After lunch we continued our meetings and worked on our game plan for the future.  After the close of the day Seth and the guys took me to a pub in downtown Reading.  At the pub we had the opportunity to discuss various British topics over an ale that was made by one of the teams’ friends. It was the perfect way to end the day.

After a quick ride to the train station, I was soon off to London.

With the words “Exterminate, Exterminate” ringing in my ear, I decided that the only way to save the human race was to find The Doctor. London is a big city and a bit foreign to me. I needed to find him and fast. I first stopped at Starbucks. A quick double espresso with an extra shot was what I needed. It could be the end of the world, no time to mess with tea.

The next stop was the BBC main office. I had worked with the BBC earlier this year. They used Scrutinizer as their sFlow collector. They might know where to find the Doctor. After a quick walk, I found their building. I ran into the door and with a polite but urgent voice I asked “Where is the Doctor?” The nice Indian gentlemen looked at me for a second or two and then said, “My friend, it is a television show. The Doctor isn’t real.”

I knew that this was a preprogrammed script that he was required to recite as a response to all the fair weather fans of the new Whovian era. I looked him in the eye and said, “I have followed The Doctor since his fourth regeneration. I need to see The Doctor.” The kind Indian gentlemen looked at me and whispered,  ”He has been seen at the Earl’s Court Station.” While running out the door I yelled “thank you”.

I grabbed a packet of Jelly Babies and went for a quick ride on the tube. Soon I was at Earl’s Court Station. I ran up the stairs and there it was. The famous blue police call box which is much larger on the inside than it is on the outside. It was the Tardis. I knocked on the door, but there was no reply. I looked around, but he wasn’t there. I decided that the best course of action was to leave him a note.  I quickly left my name, a brief message and a number to text me at.

I had tickets to ride “London’s Eye” that afternoon. Since the end of the world was on pause I decided that there was no better way to get your mind off of things than a ride on a really big Ferris wheel and have a look at London.

From The Eye you can see just about everything in London. The House of Parliament, Westminster Abbey and Big Ben are in clear view. In the distance I could see a unique arch. I asked one of the other passengers what it was and he replied “Wembly Stadium.” I chuckled to myself. The Football Association (FA), who is a co-owner, had chosen Scrutinizer for their NetFlow requirements.  I guess Walt Disney has it right, it is a small world.

All of a sudden, my phone started to vibrate. It was a text from the Doctor. “BBC let me know. All is Safe. Enjoy London -The Doctor”

I use the term “amateur horticulturist” loosely when describing one of my hobbies. I am still in the beginning steps of building a self-contained, hydroponic garden and fish farm powered by Arduino. By “beginning steps” I mean that I have a lot of bookmarks and interesting YouTube videos.

Today was different. Today I tossed away my geek gardener dreams and immersed myself in, what can only be called, the most beautiful garden on the planet.  I visited Kew Gardens, which also goes by the prestigious name of “The Royal Botanical Gardens”.

Early that morning I hopped on the tube and made my way to the Town of Kew. A quick walk down the street brought me to a large stone wall and entry gate.

There were so many interesting things to see.  From the Palm House (which experts consider the most important surviving Victorian iron and glass structure in the world) to the treetop walkway, visitors are left in awe of the 250 year old garden. My favorite exhibit was the “Evolution House” where you can walk through the evolution of our planet. Needless to say, this 300 acre garden took up my entire day.

Equally important, Kew functions as a botanical research center and maintains the largest plant collection in the world. This side of their organization is what brought “The Royal Botanical Gardens” to my attention. They picked Scrutinizer as their NetFlow solution and I had the pleasure of working with them.

Once I landed I needed to do all the things that an internationally traveling geek needs to do. The most important is getting connected! Between wireless at my lodging, cell phone and hot spots I have been able to get everything done that I needed. For most of my calls I use Skype and Skypeout. You can’t beat free calls to any US land line from here in Europe.  I was even able to video chat with my family from a local pub.

The rest of the day was spent walking around and seeing all the wonderful things the city has to offer. I passed by the Apple Store and took a picture for my Apple freak friend back at work. I stopped and listened  to a street performing  beatbox artist called mcxander that was truely amazing. I have never heard someone do a beatbox version of the didgeridoo. I think that we need to have his people talk to Mix Master Mitch’s people and see if they can make something happen!

Cisco or Google to buy Skype
With Skype being one of the largest players in the voice and video (aka VoIP) market, it is easy to understand why Cisco and Google are both trying to buy the company.

Internally, we use GoToMeeting to share desktops with customers, but we decided to test Skype out and we are fairly impressed. Below is a screen shot of the desktop sharing in Skype.  Notice it is a bit blurry, this could have been our network or my Skype settings.

Monitoring Skype with NetFlow NBAR
If you want to monitor Skype traffic using NetFlow you’ll need to switch to exporting Flexible NetFlow with NBAR to see the Skype traffic, otherwise the NetFlow collector will pick it up as HTTP. If you are asking yourself  how to configure Flexible NetFlow, we can help. Our NetFlow Analyzer is the best at NetFlow reporting and the leader in NetFlow Analysis.