I found out that my ISP does deep packet inspection. Can they see the contents of HTTPS connections? Wouldn't having HTTPS ensure that they can't see the contents being transferred?
And can having a VPN protect me against deep packet inspection by ISPs?
Deep Packet Inspection, also known as complete packet inspection, simply means they are analyzing all of your traffic as opposed to just grabbing connection information such as what IP's you are connecting to, what port number, what protocol and possibly a few other details about the network connection.
This is normally discussed in contrast to the gathering of NetFlow information which mainly collects the information listed above.
Deep packet inspection gives your provider a lot of information about your connections and habits of Internet usage. In some cases, the full content of things like SMTP e-mails will be captured.
HTTPS does encrypt the connections but your browser has to make DNS requests which are sent primarily via UDP so that data will be collected as will any unencrypted links or unencrypted cookies sent incorrectly without https. These additional bits which will be collected may be very telling about what type of content you are looking at.
The larger concern for most people is about data aggregation, by collecting this information a data scientist could create a fingerprint for your Internet usage and later associate with past activities or activities from other locations (when you are at work or are on vacation). Likewise, your service provider may choose to sell this to any number of organizations (possibly including criminal organizations) where it could then be used against you in ways. In many countries, people have an expectation that their communications are considered to be private and collecting this data very much goes against that privacy expectation.
Another interesting aspect of this is in the cases like the US where this data may soon be sold it allows International communications sent to people, or servers, in the US be sold as well. Likewise, this could potentially allow every agency from local law-enforcement, military, tax authorities, immigration authorities, politicians, etc. a way to bypass long-standing laws which have prevented them from accessing this type of information, or important informational subsets within this data otherwise.
A slightly different concern when this data can be sold is competitive intelligence / corporate espionage. In the scenario where a company does a lot of research-intensive work at their headquarters located in some small geographical location (think of pharmaceuticals or a defense contractor) selling that data makes it possible for anyone to buy all of the traffic from the local ISP where most of those researchers live and analyze what they are looking for when at home, possibly even directly from the ISP hosting the traffic for their corporate headquarters. If other countries aren't selling similar data it gives foreign companies and companies wise enough to try and buy this data a huge technical advantage. Likewise, it would also allow foreign governments to buy ISP traffic which includes the data from US (or other government) Officials homes.
Imagine companies monitoring their employee's behavior at home or on their mobile devices.
This will likely have a chilling effect on activists and whistle-blowers as well.
Likewise, if credit cards or PII are sent in the clear to a poorly secured remote site your ISP's data set now has a potential PCI or PII regulatory issue on their hands. So this amplifies data-leakage problems of all types by making additional copies of the data leaked.
With the examples I've just mentioned above, and there are hundreds of others, it should be easy to see why this type of data collection has a different level of importance to it than just metadata or basic connection information. Even if your ISP never sells this data they are collecting quite an interesting dataset.
It's a security issue that definitely has a lot of potential long-term security implications.
Trey Blalock's answer describes percisely what deep packet inspection (DPI) is. But I'd like to add three things to hopefully answer your specific questions:
As stated by Trey, DPI can see the entire content of your network traffic. All of it. If it is plain text, then they see everything that you do.
To add on to Miao's answer:
Things DPI can see, even when you use HTTPS:
With a VPN, they will still see 100% of the data. However, other than the connection to the VPN provider, they'll only see encrypted data. They will know that you downloaded 800GB from VPNco.com, but will know nothing of the data inside. Even things that are not encrypted via protocol will get encrypted since a lower level is encrypting. Now, the VPNco.com will then see your data.
With the (potential) change in US law about ISP and data privacy, combined with the (potential) loss of net neutrality, ISP's might be able to not only see 100% of your data, they could modify that data, slow or block sites they want, and might be able to sell any/all of your data to a 3rd party (as Trey states).
I'm not covering MITM (like Miao states in #1 above), since you stated ISP, I assume that you are talking about a home system and a DSL or cable modem.
https://stackoverflow.com/questions/499591/are-https-urls-encrypted
With deep packet inspection, the ISP can detect most VPN protocols (not the data encrypted in the VPN packets, just that there is VPN traffic) and block it. Some companies do this to ensure that they can decrypt all traffic (with the MITM attack and forged certificates to have DPI on SSL as well). The idea is to force you to use "insecure" communication channels by preventing everything else. Note that these "insecure" channels might be more secure , from the company's point if view, as they can do Data Leakage Prevention there.
In such a case, non-standard VPN techniques, like HTTP tunneling might be an option.
Note that the terms of use might disallow measures to circumvent DPI.
Edit: some ISPs use DPI for traffic shaping. They don't log all the transmitted data, they just check (for example) for BitTorrent traffic and assign it a lower priority or limited bandwidth. Now, they are not stealing your password, just the bandwidth you are paying for....
If you don't trust your ISP then your first priority should not be about packet inspection at all, but rather be to establish a trusted second channel of communication for which you can exchange information about circumventing such things.
As long as you rely solely on your ISP as the lone channel for all information exchange they can technically send you wrong login info to your VPN, even if they don't they could still take over any encryption handshakes attempted because they will always be in the middle.
They could have people in their employ who are bribed to do so or be required by law for any reason.
All the above is true. One more thought: Did your ISP give you a self-signed root certificate and is it in your browser? If they did, they can open your HTTPS traffic.
You have installed a vendor security certificate on your computer, otherwise there was no such option, and now there is a question how to bypass this test
