To browse the internet on sensitive websites (government websites, banks, etc...), is it as safe to use a modern web browser as a programming language? Specifically, python + requests + beautifulSoup for example?
-
2What do you expect to gain by using an HTTP library versus a web browser?– Monica Apologists Get OutJun 12, 2018 at 14:32
-
Sounds more like you want to use a hand-crafted scraper, instead of a mainstream web browser. Is that what you mean?– S.L. Barth is on codidact.comJun 12, 2018 at 14:35
-
Yes, so for example, you might write your own script that goes to an important website, and handles a task for you. The benefit is that you don't have to do it yourself, clicking 20 things. But the question is: would that be as safe?– DevSharkJun 12, 2018 at 14:50
-
1Define "safe" as you see it: would your script result in credentials being stored in plain text on the source computer, for example? That would be less safe than using a browser and entering credentials from memory. Would it avoid execution of malicious scripts if the sensitive website got compromised? That would be safer for your computer than a browser which allowed them to run. Without knowing what you mean by "safe", we can't answer.– MatthewJun 12, 2018 at 14:56
-
1If you really want to be "safe" to accomplish small tasks, I'd look towards using an API for what you want to accomplish rather than trying to mimic a browser and screen-scrape. If you use an API it becomes MUCH less complicated, and your script has to do far less things.– Steve SetherJun 12, 2018 at 17:36
1 Answer
Modern browsers are massively complicated pieces of software: they can run at least one complicated interpreted language, handle a bunch of different types of media files, and connect to a wide range of services, which can use a whole range of different methods to transfer data in both clear text and encrypted forms. That tends to mean there are a lot of things that could go wrong resulting in data being exposed to an attacker.
However, they are also subject to a lot of scrutiny when it comes to the security aspects. They implement careful memory management for sensitive pages, isolate tab contents from other tabs, have vigorously tested encryption libraries, and in many cases get constantly fuzzed looking for flaws in processing routines. Chrome has (At time of writing) about 1500 CVE issues, but, and this is quite important, the majority of these are fixed quickly.
On the other hand, a script which loads a specific site and performs one specific actions is relatively simple, even if it uses pre-built libraries to do heavy lifting such as performing SSL connections. That makes it less likely to contain issues caused by unexpected interactions between components, but it is used by fewer people, so any issues less likely to be discovered. It probably doesn't make any attempt to ensure that passwords and other sensitive information isn't written to disk inadvertently, and may even contain hard coded credentials, which could be exposed if the system it is running on is compromised. It relies on the developer to keep encryption libraries up to date, and to handle any changes to the responses from sites which are required (e.g. if the site uses an API with an Authorization header added by JS, the script will either need to run the JS, or perform equivalent steps and add the header when required).
The script is also unlikely to handle things like HSTS headers or give alerts when certificates don't match expectations, meaning that if it is used on an insecure network, an attacker may be able to intercept the traffic generated.
The threat models for each are different: the script is likely to be vastly less secure if an attacker is able to compromise the system where the script is running, while an attacker who can place malicious code in an advert shown on a sensitive page is likely to be more successful against users of full browsers - a script is probably going to ignore the advertising code, or not be vulnerable to the methods used to try escape the browser sandbox.
Personally, I'd stick with the browser though - the developers are thinking of problems I've not even considered, and implementing fixes for them without me needing to lift a finger, and as long as I keep my browser up to date, these improvements just turn up like magic.
-
Thanks a lot for a very thoughtful answer. What's your view on using python+selenium (so using a browser) versus python+requests+BeautifulSoup. It sounds like you're in favor of python+selenium on balance?– DevSharkJun 12, 2018 at 16:46
-
I suspect that could combine the drawbacks of both: you've got a credential store issue, and the ability to run scripts. You would get the tried and tested SSL stack, and probably regular updates to protect against newly discovered issues, so it's potentially better than the raw script concept, assuming Selenium itself doesn't introduce any new issues.– MatthewJun 12, 2018 at 18:36