A Deep Dive into Bit Counting
The question of “how many bits” were sent and received clearly is a matter of interest when measuring data rates or data quantities.
Mobile providers, consumer ISP's, consumers, and regulators talk a lot about their speed and (not so often) about their data caps. But rarely, if ever, are sufficient details provided:
You or I often can not tell which bits they are counting.
Nor can we tell which bits are not being counted.
How can we compare when we do not know what is being measured?
And it is not just a consumer issue: ISPs that exchange data with one another under contractual peering and transit arrangements need common ground when they discuss how much data each party is delivering to the other.
What's So Hard About Counting Bits?
You would think that counting bits would be clear and easy. But it is not.
Unfortunately there is a great lack of clarity about how to count the bits in a given Ethernet frame or IP packet.
Wikipedia has an article describing the Ethernet frame. Another Wikipedia page describes VLAN and QoS tagging.
How Does Counting Bits Vary Between Methods?
There are more than half a dozen commonly used ways to count the number of bits. And the terms used to refer to these counting methods are themselves sometimes overloaded and ambiguous.
To make matters worse there can be great differences in the numbers produced by the various counting methods.
Those who build hardware based network test and monitoring tools often count "hardware" bits that may be invisible to software but that nevertheless consume network bandwidth.
Software based tools and operating systems often count only the more easily seen parts of Ethernet frames of IP packets.
Standards bodies sometimes define extensions that add fields and bits. But these extensions may be so rarely used in practice that few tools bother to consider that they may be present.
Some tools are very selective; they measure (and count) only a portion of the traffic on a network. This is particularly true of tools that may be used by consumers, such as the ever increasing variety of "speed test" web tools or by the iperf family of tools. (A future article on this blog will discuss the shortcomings and biases of "speed test" tools.)
These differences tend to grow larger as packet sizes shrink. So for traffic types that use large packets - such as internet video - the bit counts produced by the various counting methods may diverge by only a few percent. But for other traffic types - such as voice-over-IP - one counting method could give a number that is 2x (or more) larger than the count produced by another method. The reason for this is that with small packets the ratio of overhead to payload increases. Some methods count some or all of that overhead, other methods do not.
Why You Can’t Trust Tools to Count Bits For You
When it comes to network testing and measurement tools, there's often a lack of transparency regarding their inner workings. These tools fail to give users comprehensive insights into what they count and how they do it. This information gap can leave users uncertain, relying on assumptions or attempting to reverse engineer the tool to uncover its counting methodologies.
This ambiguity can lead to skewed results and misinterpretations. Consequently, trusting these tools blindly can be risky, as their operations remain hidden behind a blanket of complexity. To ensure the accuracy and reliability of your network testing, it's essential to complement these tools with a deeper understanding of network dynamics and, in some cases, manual verification.
Tools that test and measure networks frequently do not adequately disclose the details of what and how they are counting. Users are thus forced into making presumptions or to reverse engineering the tool to discover what it is doing and measuring.
Understanding the Differences between Frames vs Packets in this Article
One common area of confusion in networking revolves around the terms "frames" and "packets." Let's break down these concepts to make them crystal clear:
Frames: Think of "frames" as the outermost layer in networking. These pertain to Ethernet-level components, encompassing the fundamental building blocks of data transmission. Frames serve as the containers that house all the data we want to send across a network.
Packets: Delving a layer deeper, we encounter "packets." Packets are the essential data units carried within the payload field of an Ethernet frame. They contain the actual information you're sending or receiving, whether it's a web page snippet, an email, or a video stream.
So, when we say "frame," visualize it as the outer shell encapsulating your data, and when we refer to a "packet," think of it as the precious cargo nestled within that frame.
This clear distinction helps us understand the hierarchy of data transmission and the roles these entities play in ensuring your information reaches its destination accurately and efficiently.
What are the Different Methods for Counting Bits?
Existing Terminology
There are a some counting method names that are in common use.
The most common of these are "L1", "L2", "L3", and "L4".
These refer to layers 1-4 of the OSI network model: L1: "Physical", L2: "Data Link", L3: "Network", and L4: "Transport".
These names, unfortunately, are often attached to slightly different counting algorithms. For instance the "L3" count used by one network test or measurement tool may differ from the "L3" count produced by another tool. In other words, the existing terminology can be ambiguous.
A New Naming System
To avoid the ambiguity of existing counting method names this article will create its own.
Our counting method names are based on the size of the data payload in an Ethernet frame.
It may be useful to look again at the structure of an Ethernet frame:
We will begin by looking at an Ethernet frame sans the source/destination MAC addresses, sans the Ethernet type/length, sans any VLAN tags/Qos, sans the CRC/FCS, sans the preamble, and sans the interframe gap.
So let's begin our enumeration.
EtherPayload
Count the bits in the Ethernet payload.
In our naming taxonomy this is our basic EtherPayload counting method. Only the payload is counted; the preamble, Ethernet header, CRC, and interframe gap are not counted.
VisibleEtherFrame
Count the bits beginning with the Ethernet header and ending at the end of the Ethernet payload. In other words, the preamble, CRC, and interframe gap are not counted.
In our naming taxonomy this is the VisibleEtherFrame counting method.
This view of the Ethernet frame is what is typically passed between network interface (NIC) hardware and an operating system. This is typically the lowest level of networking within an operating system and is often used by operating systems as the basis for network interface statistics.
This view of the Ethernet frame is also what is typically seen through software based packet (or frame) monitoring tools such as Wireshark.
This is why we refer to this as the "visible" portion of the Ethernet frame.
The relationship between EtherPayload and VisibleEtherFrame depends whether the Ethernet header contains an IEEE 802.1Q VLAN/QoS field or not. VisibleEtherFrame could be:
EtherData + 112 bits (14 bytes) if there is no VLAN/QoS field.
EtherData + 144 bits (18 bytes) if there is an IEEE 802.1Q field.
VisibleEtherFrame+4
Count the bits beginning with the Ethernet header and ending at the end of the Ethernet CRC/FCS. The preamble and interframe gap are not counted. This is similar to the VisibleEtherFrame count above, but is 4 bytes/32 bits larger.
In our naming taxonomy this is the VisibleEtherFrame+4 counting method.
This view of the Ethernet frame is sometimes used by network monitoring or testing tools built on specialized hardware.
And as above, the mathematical relationship between EtherPayload and VisiblEtherFrame+4 depends whether there is an IEEE 802.1Q field in the Ethernet header or not.
EtherData + 144 bits (18 bytes) if there is no VLAN/QoS field.
EtherData + 176 bits (22 bytes) if there is an IEEE 802.1Q field.
FullEtherFrame
Count all the bits in the full Ethernet frame. This would encompass the data in the visible Ethernet frame as well as the bits that form the Ethernet preamble, Ethernet CRC/FCS, and interframe gap.
In our naming taxonomy this is the FullEtherFrame counting method.
This view of the Ethernet frame is usually used only by specialized test and monitoring devices based on specialized hardware.
The FullEtherFrame is larger than the VisibleEtherFrame by 192 bits/24 bytes.
Because the size of the Ethernet header depends whether that header contains VLAN/QoS tags or not the relationship between FullEtherFrame and EtherPayload could be either:
EtherPayload + 304 bits (38 bytes) if there is no VLAN/QoS field.
EtherPayload + 336 bits (42 bytes) if there is an IEEE 802.1Q field.
IPPacket
Count the bits that comprise the IP (v4 or v6) header and IP data.
In our naming taxonomy this is the IPPacket counting method.
This method, of course, presumes that the Ethernet frame being counted contains an IP packet.
This count is often the same as EtherPayload but not always. There are cases in which the IP packet occupies less than the full space provided by the Ethernet data payload. The most that can be said is the EtherPayload >= IPPacket.
IPPayload
Count the bits that comprise IPv4 or IPv6 data.
In our naming taxonomy this is the IPPayload counting method.
This method depends on there being an IPv4 or IPv6 packet within the Ethernet frame.
Because the size of the IPv4 and IPv6 headers varies depending on what options are present the only thing that can be said is that FullEtherFrame > EtherPayload > IPPacket > IPPayload
IPTransportData
Count only the bits that comprise UDP or TCP data.
In our naming taxonomy this is the IPTransportData counting method.
This count is possible only for those Ethernet frames that contain an IP header and either a UDP or or TCP transport header.
For IPv4 packets (without IP options) carrying UDP with data that extends to the end of the Ethernet data area, the IPTransportData count is smaller than the EtherPayload count by 224 bits/28 bytes and is smaller than the IPPayload count by 64 bits/8 bytes.
How Does This Relate to Counts Produced by Various Test & Measurement Tools?
FullEtherFrame is often referred to as "L1"
VisibleEtherFrame and VisibleEtherFrame+4 are often referred to, ambiguously, as "L2"
EtherPayload is often referred to as "L3"
IPTransportData is sometimes called "L4". This is the counting method by iPerf
IWL's network testing and emulation tools (including KMAX and Maxwell Pro) can adapt to most counting methods by providing user access to the knobs and levers that control the way that bits in packets are counted.
Conclusion
There are good reasons why we need to know how much data, now many bits, are being carried on our networks, or could be carried on our networks.
However there is a lack of precision about how to count those bits. There are several counting algorithms. And there is some disagreement regarding which counting name is attached to what counting algorithm.
This article tries to bring some clarity to this situation by creating new names that are unambiguously attached to counting algorithms.
This article does not try to be all-inclusive. There still remain issues that come out of things such as VLAN and MPLS stacking. But those are rarely practiced and not usually of concern.