How do mobile carriers know video resolution over HTTPS connections?

Question

Verizon is modifying their "unlimited" data plans. Customers in the USA can stream video at 480p -or- pay to unlock higher resolutions (both 720p and +1080p). They are not the only mobile carrier to implement rules like this.

If I am on a site that implements HTTPS for video streaming, say YouTube or Facebook, how do carriers know what resolution I'm watching? If carriers are throttling bandwidth for all data, then talking about video resolutions seems like misdirection. If it's only video, that would seem to raise privacy concerns.

Answer 1

This is an active area of research. I happen to have done some work in this area, so I'll share what I can about the basic idea (this work was with industry partners and I can't share the secret details :) ).

The tl;dr is that it's often possible to identify an encrypted traffic stream as carrying video, and it's often possible to estimate its resolution - but it's complicated, and not always accurate. There are a lot of people working on ways to do this more consistently and more accurately.

Video traffic has some specific characteristics that can distinguish it from other kinds of traffic. Here I refer specifically to video on demand - not live streaming video. Video on demand doesn't often have those priority tags mentioned in this answer. Also I refer specifically to adaptive video, meaning that the video is divided into segments (each about 2-10 seconds long), and each segment of video is encoded at multiple quality levels (quality level meaning: long-term video bitrate, codec, and resolution). As you play the video, the quality level at which the next segment is downloaded depends on what data rate the application thinks your network can support. (That's the DASH protocol referred to in this answer.)

If your phone is playing a video, and you look at the (weighted moving average of) data rate of the traffic going to your phone over time, it might look something like this:

(this is captured from a YouTube session over Verizon. There's the moving average over 15 seconds and also short-term average.)

There are a few different parts to this session:

First, the video application (YouTube player) tries to fill the buffer up to the buffer capacity. During this time, it is pulling data at whatever rate the network can support. At this stage, it's basically indistinguishable from a large file download, unless you can infer that it's video traffic from the remote address (as mentioned in this answer).

Once the buffer is full, then you get "bursts" at sort-of-regular intervals. Suppose your buffer can hold 200 seconds of video. When the buffer has 200 seconds of video in it, the application stops downloading. Then after a segment of video has played back (say 5 seconds), there is room in the buffer again, so it'll download the next segment, then stop again. That's what causes this bursty pattern.

This pattern is very characteristic of video - traffic from other applications doesn't have this pattern - so a network service provider can pretty easily pick out flows that carry video traffic. In some cases, you might not ever observe this pattern - for example, if the video is so short that the entire thing is loaded into the buffer at once and then the client stops downloading. Under those circumstances, it's very difficult to distinguish video traffic from a file download (unless you can figure it out by remote address).

Anyway, once you have identified the flow as carrying video traffic - either by the remote address (not always possible, since major video providers use content distribution networks that are not exclusive to video) or by its traffic pattern (possible if the video session is long, much more difficult if it is so short that the whole video is loaded into the buffer all at once)...

Now, as Hector said, you can try to guess the resolution from the bitrate by looking at the size (in bytes) of each "burst" of data:

From the size per duration you could make a reasonable estimate of the resolution - especially if you keep a rolling average.

But, this can be difficult. Take the YouTube session in my example:

• Not all segments are the same duration - the duration of video requested at a time depends on several factors (the quality level, network status, what kind of device you are playing the video on, and others). So you can't necessarily look at a "burst" and say, "OK, this was X bytes representing 5 seconds of video, so I know the video data rate". Sometimes you can figure out the likely segment duration but other times it is tricky.
• For a given video quality level and segment duration, different segments will have different sizes (depending on things like how much motion takes place in that part of the video).
• Even for the same video resolution, the long-term data rate can vary - a 1080p video encoded with VP9 won't have the same long-term data rate as one encoded with H.264.
• The video quality level changes according to perceived network quality (which is visible to the network service provider) and buffer status (which is not). So you can look at long-term data rates over 30 seconds, but it's possible that the actual video quality level changed several times over that 30 seconds.
• During periods when the buffer is draining or filling as fast as possible (when you don't have those "bursts"), it's much harder to estimate what's going on in the video.
• To complicate things even further: sometimes a video flow will be "striped" across multiple lower-layer flows. Sometimes part of the video will be retrieved from one address, and then it will switch to retrieving the video from a different address.

That graph of data rate I showed you just above? Here's what the video resolution was over that time interval:

Here, the color indicates the video resolution. So... you can sort of estimate what's going on just from the traffic patterns. But it's a difficult problem! There are other markers in the traffic that you can look at. I can't say definitively how any one service provider is doing it. But at least as far as the academic state-of-the-art goes, there isn't any way to do this with perfect accuracy, all of the time (unless you have the cooperation of the video providers...)

If you're interested in learning more about the techniques used for this kind of problem, there's a lot of academic literature out there - see for example BUFFEST: Predicting Buffer Conditions and Real-time Requirements of HTTP(S) Adaptive Streaming Clients as a starting point. (Not my paper - just one I happen to have read recently.)

Answer 2

Nothing maxes out bandwidth at a consistent rate other than streaming video.

Also, in order to make sure that stream is handled with priority (and not like a big file download, for instance) streaming sources tag the packets in a way that tells the carriers that it is streaming video. The rest of the packet is encrypted, but the metadata that tells the ISP how to route it gets to see this part. If they did not do this, there would be a high chance that the stream would get interrupted or degraded as the ISP tried to balance all the needs of the network traffic at that time.

And here is how Verizon said they will do it:

Verizon apparently won't be converting videos to lower resolutions itself. Instead, it will set a bandwidth limit that video applications will have to adjust to. "We manage HD video throughput by setting speeds at no more than 10Mbps, which provides HD video at up to 1080p video," Verizon told Ars. The Mbps will presumably be lower than that in cases where Verizon limits video to 480p or 720p.

That means that both the subscriber and the fact that the traffic is shaped a certain way because it is a certain type of video means it is tagged.

How? Verizon has a video optimization system that has been shown to limit Netflix and YouTube to 10 Mbps even before the Aug 2017 announcement of the new caps.

Verizon acknowledged using a new video optimization system but said it is part of a temporary test and that it did not affect the actual quality of video. The video optimization appears to apply both to unlimited and limited mobile plans.

But some YouTube users are reporting degraded video, saying that using a VPN service can bypass the Verizon throttling.

This points to the ability for Verizon to identify video streams and limit the bandwidth accordingly, even if the content is delivered over HTTPS (but not VPNs).

Answer 3

Schroeder is almost certainly right in that its just a marketing way of saying they restrict bandwidth to certain sites IP addresses or look for priority markers on the packets.

It is worth noting however that theoretically there are ways they could make this work better if the sole aim was to force users to a certain resolution while video streaming and nothing else.

Much internet streaming these days uses a process called DASH (Dynamic Adaptive Streaming over HTTP). The way that this works is to request a small chunk of video, measure the bandwidth while this is downloaded and select the next chunk of video at a resolution / compression scheme that would allow it to be received in time for when the first chunk has finished playing.

This means there are hints in the requests as to what the user is doing. If your device sends a request to a website every 3 seconds requesting a file that takes just under 3 seconds to download then there is a very high chance that site is streaming video. From the size per duration you could make a reasonable estimate of the resolution - especially if you keep a rolling average. You can then just restrict bandwidth to that ip address.

By using known IP addresses for major video providers (googlevideo (youtube), Netflix etc) in decision weighting you could make the algorithm more aggressive without too many false positives.

Answer 4

The biggest thing is the address you connect to. HTTPS protects the data in flight, but doesn't protect the address you are talking to. If Verizon knows the IP addresses of the Netflix servers, they can place restrictions on streams of data from those IPs that are within a certain range. Netflix will then automatically adjust it's playback based on available bandwidth.

It's also broadly possible to notice based on having constant data rates of traffic that are less than the max possible. This likely means a media stream of some type, though it could also be a bandwidth limited file transfer, so there would be some false positives.

I believe that in the technical details they mentioned when announcing the changes, they were just planning on using the IP addresses to setup the limits, but I'm not sure if that will change over time or if I am even remembering correctly as it has been a little while since the announcement and I didn't keep notes on my original research about it.

Answer 5

If you use a (proper) VPN that values security/privacy over performance, it will employ a bunch of tricks to befuddle any attempts by your ISP to positively identify you as streaming video:

• The ISP will see you connecting to an IP address that isn't recognized as part of a video CDN (e.g. Youtube, etc.) - assuming your VPN isn't hosted by Youtube or any other major video provider. So at the least they won't be able to positively identify your remote IP endpoint as being part of a video CDN. Note: Of course, they could decide to throttle it because it's a VPN, but the major providers do not, as of now, explicitly state that they throttle VPNs (so, doing so would be extremely underhanded and would rank U.S. ISPs in their suppression of free speech similarly to the Great Firewall of China).
• A good VPN does not submit any metadata in the clear (e.g., QoS) that would hint at the nature of your traffic. This might result in weird priority inversions (such as a download causing lag in real-time VoIP or streaming video), but that's the price you pay for privacy.
• Depending on the application-layer video protocol you use, it would be difficult to impossible for the ISP to positively identify the traffic as video using traffic shape analysis. For example, if the protocol specifies that the server should burst send the video at maximum available bandwidth (line speed) to download most/all of the video at once instead of gradually pushing it at the video's bitrate, the ISP has no idea if you're downloading a Linux ISO, a streaming video from a well-known content provider like Youtube or CBS, or a big Excel document for your job.
• While the total volume of VPN traffic will roughly mirror the actual data being sent/received, good VPNs throw in junk data (and even introduce small latency in legitimate traffic) within the encrypted tunnel in order to make traffic analysis harder. The more your VPN introduces junk data and intentional latency, the slower your VPN will be, but on the flip side, it will improve security if your VPN's traffic is highly resistant to traffic shape analysis.

However, there are some limitations:

• The easiest type of streaming video to analyze via traffic shape analysis is a protocol that throttles your video download to the bitrate of the video. Based on bitrate, they can guesstimate what your likely resolution is.
• "Live" video (where the viewer values viewing the content very soon after it is generated) is trickiest to hide, because you have to transmit at close to the video bitrate to guarantee good responsiveness; buffering and bursting the data imposes an inherent delay to the stream that the user may not find acceptable (e.g. for video conferencing, Twitch.TV, etc.)

Of course, as a way of ensuring that videos can't be streamed at higher qualities, the ISP could choose to just throttle everything you download to an arbitrary bitrate lower than line speed - but there are many problems with that:

• There is no well-established "typical" bitrate of video at any particular resolution, so they could make assumptions, but they'd be just that: assumptions. Depending on the codec and encoding settings, a 1080p video could vary from low single-digit Mbit/s to 50 Mbps and higher.
• This would become an effective "everything throttle" and you'd never sniff maximum advertised speeds, leading to potential false advertising liability if you're getting far less than their advertised min/max speeds and haven't stepped on any landmines that they advertise as triggering a throttle (e.g. the typical 22 GB/month high speed data cap on LTE).
• If they throttle everyone heavily, they'd get a lot of backlash from legitimate users who never stream video but are, as previously mentioned, just downloading a big Excel file from work over a VPN or whatever.
• By inferring that traffic may be video and deciding to throttle it based on that (even if their inferences turn out to be incorrect), users are more likely to perceive their service as slow, and cancel / move to another provider, because the ISP doesn't advertise that they throttle other types of traffic besides video (unless you exceed a given cap per month or happen to be on a saturated tower; both of these situations are fairly easy to rule out, though, as a customer).

So overall, in the arms race of ISPs trying to detect users' specific content and discriminating based on what you are downloading, the user will ultimately win if you use a good VPN (and assuming that the ISP has no way of breaking your cryptography). The ISP won't be able to detect what your content is with any certainty, and guesses won't come with a very high probability of being accurate.

I think as users start to clue in to this and take measures to prevent ISPs from detecting their content, ISPs will respond by either decreasing their advertised speeds for everyone (to allow more subscribers to share the same available spectrum); raising prices; eliminating unlimited data plans (again); worsening throttling penalties for those who exceed an arbitrary (fairly low) data cap per month; or intentionally slowing down any traffic that they can't positively identify (for example, VPN traffic) and justifying it as the traffic being "suspicious".

They could argue, and convincingly so to regulators, politicians and the ignorant public, that most/all VPN traffic is illegal downloads, illicit commerce (drugs, etc.), and people trying to "break the rules" and bypass restrictions set by the ISP. While technical experts would cry foul, we have a tendency to get drowned out by those who are complicit in the system and happy to screw over the consumer to make a quick buck.

They could also do all of these things simultaneously. My experience with the wireless industry in the US is that, as bad as you think they are, they're worse. And rather than choosing only the most benign remedies to a problem, they have a tendency to favor ones that are maximally hostile to customers, which suppress free speech and violate net neutrality principles.

They also have a tendency to try and pit users against one another. They adopt policies that make users blame and call out other users as "data hogs", rather than working together as a group to pressure their provider to expand infrastructure to meet demand rather than trying to thin the herd to match currently-available resources.

It's just human nature, I suppose -- which is why the wireless industry needs to be regulated if we want it to be a consistent, valuable building block of society akin to electricity and running water. Until then, it's just another variation on the Byzantine proprietary PSTN of the Ma Bell days.

Answer 6

Verizon doesn't LITERALLY recognize and block encrypted HD video... but it has agreements with services like Netflix and Youtube that require THEM to recognize Verizon IP address ranges & respect any limits imposed by Verizon.

For smaller services, I believe Verizon has a second weapon at its disposal... I'm pretty sure that somewhere in the customer service agreement for "unlimited" plans is a clause that allows Verizon to count "opaque" encrypted traffic against the user's monthly "tethering" limit. So Verizon might not know (or care) that you're streaming HD video to your phone all day from your security cameras at home, but they CAN count the traffic against your monthly 5 or 10gb limit (or throttle it to 150kbps if your plan doesn't include tethering).

Answer 7

Not sure if this is true in your case but sometimes providers use a proxy. This could be automatically set when you use their network (OTA). The proxy has a valid certificate, as it's from your provider. If so they can just read all HTTPS traffic and use some software for traffic shaping. If this is the case a VPN would help.

So the provider automatically sets the APN on your mobile device. The internet traffic is send from your mobile device to the APN. The provider can use any commercial traffic shaping software to control video (from FB/YT/etc) streaming. Even if the connection is encrypted you can see from the way the buffering works what video resolution is used, videos download parts in chunks which create a destinct pattern. So if these chunks get bigger the video resolution is higher, with traffic shaping software you can probably regulate these download patterns. When using a VPN the provider cannot determine what kind of data is send over the VPN. He could still see the chunk pattern but limiting all data via VPN would cause your total download speed to be lower and I assume that is against the agreed speed mentioned in your contract.

Addition: Your phone connects to a APN. This APN can act like a proxy and it already has a valid certificate (so you don't install it). Your phone will accept the certificate as a valid one, as it is supposed to do. In case it acts as a proxy the provider is not stopped by your HTTPS encryption, the proxy is required to see all your request. In this case the provider can do anything with your request or with the received data. The provider could for example have software running to optimize/compress the video before it send from the proxy back to your phone to save bandwitdh.