1

I've been messing around with Python to automatize downloading webpages and images when needed, and for that I'm using urllib.request, usually authenticated to the sites with my login. As someone still very ignorant about internet security, I heard HTTP connections are not recommended due to the fact that the information you send to the website (including usernames and passwords) gets exposed to your ISP and possibly hackers.

Would using urllib.request only on addresses starting with https:// automatically secure me, at least from the problems related to HTTP described above?

Or do I need to set up some stuff on my side too? Searching about this question online I found out it is recommended to at least use ssl.create_default_context() as the context for urllib methods which I'm gonna do from now on. So basically:

  1. Should I be worried about the passwords I used without the ssl library's context? Are they (and all the data I exchanged) exposed to my ISP? Or did connecting only to https URLs automatically protect me from this particular problem?
  2. If my passwords are indeed unsafe, will using the default context (after changing my passwords) help here? Or that's not enough?

2 Answers 2

2

Any data transmitted over HTTP is plain text and might be intercepted or read by anyone who has control of the devices/networks between the client and the server. HTTP should no longer be used for transmitting sensitive information.

Any request over HTTPS is today normally secure when using modern software and standard configuration. A modern server would refuse insecure connections, and a modern client would also refuse insecure connections.

However, a badly configured client (like older versions of Python) that don't do HTTPS certificate validation can lead to security problems. If anyone has full control over the network connection between your client and the server they might be intercepting and reading everything that is being transmitted. This is why you read that ssl.create_default_context() is recommended since it would configure older versions of Python to do certificate validation and set a recommended set of ciphers.

Not validating a certificate doesn't necessarily mean that your data is unsafe. If your connection ended up in the intended destination at the web server you wanted to reach, the connection is still securely encrypted (unless both the client and the server allows insecure protocols/ciphers). Even if someone like your ISP would try to dump the transmitted data, it would not be readable.

However, without validating the certificate you are vulnerable to man in the middle attacks if anyone in the network between your client and the web server is able to intercept your connection or by other means redirect your connection to a rogue server which can read the contents of your request (including any passwords), and pass it on to the real destination so you won't notice any thing.

  1. If you are using an older version of Python that does not validate SSL certificates by default, and you ALSO have a reason to not trust your ISP or any network between your client and the server, you should be worried. If you are using a newer version of Python it is secure and there is no need to worry. If you are using an older version that does not validate certificates by default, but you do trust the path between your client and the server, it should probably be OK was well.

  2. If you were using an old version of Python that does not validate SSL certificates by default, ssl.create_default_context() should be enough to validate certificates and enable a decent set of protocols and ciphers. Using a newer version of Python will automatically validate SSL certificates.

From release version 2.7.9/3.4.3, Python will validate certificates by default. This affects all relevant stdlib modules (urllib/urllib2, http, httplib).

Sources and documentation: https://www.python.org/dev/peps/pep-0476/ https://docs.python.org/2/library/httplib.html#httplib.HTTPSConnection

1
  • Thanks for clarifying the behaviour of Python wrt. its use by default of TLS across the different versions. Good point about MITM and certificate validation - this is somewhat difficult to do manually.
    – brynk
    Commented Mar 3, 2021 at 0:50
0

Would using urllib.request only on addresses starting with https:// automatically secure me, at least from the problems related to HTTP described above?

Yes, at least in Python3 (v3.8.2)

Should I be worried about the passwords I used without the ssl library's context? Are they (and all the data I exchanged) exposed to my ISP? Or did connecting only to https URLs automatically protect me from this particular problem?

No - you don't need to worry about your passwords being exposed on the wire if you use HTTPS (TLSv1.2+) .. at least at time of writing - if you do use http only, then you're credentials are exposed on the wire - see the subsequent screenshot.

If my passwords are indeed unsafe, will using the default context (after changing my passwords) help here?

Change your passwords if you've already used them over http, and switch to HTTPS from now on to ensure urllib.requests uses TLS. If you have sound advice to set the default ssl.create_default_context() then follow this also. In this post, I only reviewed the default behaviour against a web host that wants to ensure the connection is (more) secure. A malicious host might try to force a downgrade, or not force an upgrade to HTTPS.


I built the following test script, and had Wireshark running in the background at the same time as I ran the script. In the subsequent screenshot, you can clearly see the password is being transferred in the clear. Incidentally, infosec.SE processes the http transaction and forces an upgrade to HTTPS.

If, instead, you prefix the url with https:// then urllib.requests will ensure that the exchange occurs via TLS (in this case, TLSv1.2).

import urllib.requests
from urllib.parse import urlencode

data_string = urlencode( {
  'isLogin':  'true'  ,
  'isSignup': 'false'  ,
  'email':    '[email protected]'  ,
  'password': 'Fizz.Buzz.Obscene.Frankfurter'  ,
  } )

data_bytes = data_string.encode()

surl = '://security.stackexchange.com/users/login-or-signup/validation/track'

req = urllib.request.urlopen('http'+surl, data=data_bytes)
        # for python3.9+ you may need to add:  method='POST'
>>> data_string
'isLogin=true&isSignup=false&email=email%40email.com&password=Fizz.Buzz.Obscene.Frankfurter'
>>> req.read()
b'Login-OK'
>>> list( req.info().raw_items() )
[('Connection', 'close'), ('Content-Length', '8'), ('cache-control', 'no-cache'), ('pragma', 'no-cache'), ('content-type', 'text/plain'), ('server', 'Microsoft-IIS/10.0'), ('strict-transport-security', 'max-age=15552000'), ('x-route-name', 'Users/TrackLoginOrSignupValidation'), ('x-frame-options', 'SAMEORIGIN'), ('x-flags', 'AA'), ('x-aspnet-duration-ms', '0'), ('x-request-guid', '6e00e28c-9ced-4503-bef6-53213d971ca9'), ('x-is-crawler', '1'), ('x-providence-cookie', '25839c49-ee0d-05a4-0270-6121c53c7407'), ('content-security-policy', "upgrade-insecure-requests; frame-ancestors 'self' https://stackexchange.com"), ('Accept-Ranges', 'bytes'), ('Date', 'Tue, 02 Mar 2021 22:27:38 GMT'), ('Via', '1.1 varnish'), ('X-Served-By', 'cache-syd10150-SYD'), ('X-Cache', 'MISS'), ('X-Cache-Hits', '0'), ('X-Timer', 'S1614724058.110293,VS0,VE271'), ('Vary', 'Fastly-SSL'), ('X-DNS-Prefetch-Control', 'off'), ('Set-Cookie', 'prov=25839c49-ee0d-05a4-0270-6121c53c7407; domain=.stackexchange.com; expires=Fri, 01-Jan-2055 00:00:00 GMT; path=/; HttpOnly')]

Wireshark showing the HTTP POST exchange with unencrypted credentials

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .