1

I have been tasked with some backend automation that needs to scrape data off an internal website using an employee's username/password to log in to the site.

There is a backend database server, but I do not have access to said database (bureaucracy is a stumbling block). Specifically, all employees have access to this data via a username/password login on the internal site, and my scraper needs to as well.

Since I have an automated scrape that needs to be done at regular intervals, I unfortunately require storage of a set of credentials in plaintext, such that they can be used to authenticate with the site, and then scrape the information for digesting. The host system for these scripts is locked down, behind key required SSH, and is probably stronger than the physical security in this office anyway (someone with access to the office could get this information much more easily than attacking my server).

My question: How can I make the stored plaintext passwords on the client as low a risk as possible, given that it is impossible to avoid altogether?

As an addenda for potential additional questions, the site is a simple HTTPS form, username/password are POSTed, and an auth token (session ID) is used for further request. Unfortunately the ID expires, so I need to redo the password auth every 2 hours.

EDIT:

Based on questions I have received so far, my current theoretical implementation is to keep the username/password stored, encrypted on disk, and require the decryption key at first run/reboot as well as at a set time period. Theoretically this would reduce the risk of the device being stolen physically, and compromising the authentication information.

An alternate option I'm considering is to have an intermediary service running on a separate or locked down instance/container, whose image would be encrypted on disk. This can be the middle man authentication service, and would have a single purpose, to fetch the session ID key from the site on request, and store the temporarily decrypted username/password information. This might serve to further separate the user/pass info from the potentially more exposed service.

|improve this question
  • As you say, the plaintext password needs to be accessible to the automated software, which means that any security you put on the key will be obfuscation at best and a skilled hacker with access to the system will find a way to extract the key from memory, disk, etc. I think the short answer is going to be "just store it and rely on the system being locked down", but maybe someone can suggest some obfuscation tricks. Since you mention SSH, I assume the system is linux? – Mike Ounsworth Sep 4 '18 at 21:31
  • 1
    Does the process have to start automatically, or can it be a long-running process that periodically scrapes but requires the password to be entered on startup? – AndrolGenhald Sep 4 '18 at 21:41
  • @MikeOunsworth the system is indeed linux. My current idea is to store the authentication info, encrypted on disk, and have a password to decrypt entered at boot, and required on a timed interval. That way, the plaintext is never kept between boots without some sort of oversight. – Baron_Von_Munchhausen Sep 5 '18 at 14:56
  • @AndrolGenhald That is kind of what I'm thinking about now for an implementation. The system can run for a few days, and theoretically require the password to be re-entered on a set timeout. – Baron_Von_Munchhausen Sep 5 '18 at 14:57
0

As usual with these types of questions, let's start by defining a Threat Model. The Electronic Frontier Foundation has a article called the Evaluating your threat model where they ask you to ponder three questions (italics mine):

  1. What are you protecting? (The data, communications, and other things that could cause problems for you if misused)
  2. Who are you protecting it from? (The people, organizations, and criminal actors who might seek access to that stuff)
  3. How many resources can you invest in protecting it? (The money, time, and convenience you're willing to dispense with protecting those things)

It sounds like:

  1. You are protecting A) data that is already public, at least within your organization, and B) a username/password for this content.
  2. You are protecting it from hackers who get onto your server, which is buried deep inside the local network, but excluding as a threat anybody with physical access to the office (and presumably also access to the network?). So which threat actors are you actually trying to protect against?
  3. It sounds like you're willing to invest some effort into protecting the username / password, but you can't have a human babysit it 24/7.

As pointed out in the question and in comments, the software needs access to the username / password, so the best you can do is to encrypt it on disk and require a human to enter the decryption password from time to time. There's a few options on how to do this depending on your threat model / how paranoid you want to be:

  1. Use openssl (or some other crypto lib) in your app to read the encrypted file from disk and prompt a human for the decryption password every time the scraper needs it, then wipe it from memory (this protects against malicious users that have compromised the server where it's running).
  2. Use openssl as above, but only prompt for a decryption password on process startup.
  3. Find some linux trick that protects it within the user's home directory (something equivalent to Microsoft's Data Protection API).
  4. If this is a single-user server or you're not worried about treating other users as malicious, then enable full disk encryption on the machine so that it prompts for a password during boot. This really only protects against physical theft of the server, but maybe that's appropriate for your threat model?

You should also create a separate user account for the scraper rather than giving it the credentials of a human user, this A) helps with auditing the logs for bizarre behaviour caused by the scraper, B) protects that person's password from leaking, especially if they reuse that password for other accounts, C) allows you to easily suspend the scraper's account if you do suspect compromise.

|improve this answer
  • I definitely agree with your recommendation for a threat model, and it is something that I have already done ( if not necessarily in name ) when considering the possible methods going forwards. – Baron_Von_Munchhausen Sep 5 '18 at 15:35
  • As an addendum, the box is already almost in line with your recommendation #4, single-user, no public access, SSH key, separate account for the script host, and I've actually physically removed the monitor/USB ports from the box, so should anyone be at the desk where it is locked, they would need time and hardware to reconnect them. Regarding assessment #2, essentially it is additional protection to prevent my or someone else's account login info from being easily accessible, assuming someone has network access. I'd rather my system not be the weak link, at lest for accountability's sake. – Baron_Von_Munchhausen Sep 5 '18 at 15:47
  • Last comment, I swear, The other main reason for needing to keep username/password as locked down as is reasonably possible, is because the script will be automating things that I would have to do by hand. The login info gives the holder the same rights as I would have on the network and to the protected services. Corporate policy says to never store the login info in insecurely, at least not publicly or outside of a password manager. The idea is that the implementation should make it harder to get at the stored password than to simply steal my personal laptop or hit me until I talk. – Baron_Von_Munchhausen Sep 5 '18 at 15:56
  • @Baron_Von_Munchhausen Read and acknowledged. I don't think I feel comfortable offering any deeper advice; you seem to understand the issues pretty well and need to make a decision that you can justify to your higher-ups. – Mike Ounsworth Sep 5 '18 at 16:09
  • Thank you for your advice so far, I think you've given me both a good starting point for further hardening, as well as a confirmation of what I already suspected. – Baron_Von_Munchhausen Sep 5 '18 at 16:13
3

The quick and easy (but wrong) solution is have a master password to encrypt the logins and passwords. You'll need to type this password each time you run the script. If you don't want to have to type it each time, you can see this question for other options.

The correct way to do this is go to the admin of your internal site and ask them for web scraping credentials that give you all the proper access rights to scrape what you need. You should never really have access to logins and passwords of others. It's dangerous for the company and a liability for yourself if something bad happens. Unfortunately, this may not be an option as it requires cooperation from someone else.

|improve this answer
  • 1
    I wholly agree with the vulnerability present in storing plaintext authentication information. I will eventually be able to get direct access, and bypass the web based method I'm stuck with currently. I do not store the password hard-coded, it is currently read in from an environment variable. Presently, it is entered each time it is run. As you mentioned, cooperation is possible, but not quite viable at the moment. – Baron_Von_Munchhausen Sep 5 '18 at 14:52
0

My question: How can I make the stored plaintext passwords on the client as low a risk as possible, given that it is impossible to avoid altogether?

If you really need to store it on disk maybe you could explain what is the reason? This might make it easier to give a precise answer. I might guess that you store it unencrypted because you have it hard-coded in the script? And the script is stored on disk? In this case, instead of hard-coding the password you could read it from an environmental variable. This is safer than hard-coding directly in the script.

|improve this answer
  • 1
    I would argue that "stored encrypted, but encryption key also stored on disk" is equivalent to plaintext. – Mike Ounsworth Sep 4 '18 at 21:52
  • Yeah, that is a good point. Edited. – hft Sep 4 '18 at 22:56
  • 1
    I do not keep the password hard coded. In fact, I have taken efforts to make sure that no authentication info is ever kept on disk, scripts run to redact login info from all config files committed to the working repo. I currently rely on environment variables at runtime to provide the authentication each time it is run. @MikeOunsworth I agree regarding encrypted passwords being the same as plaintext. – Baron_Von_Munchhausen Sep 5 '18 at 14:54
0

I suggest to use something like keepass, store username and password in it and lock it with master password (Or you can simply encrypt the username password using OpenSSL), now you need to protect your master password.

You can put master password in windows keystore or Linux keyring and associate it to service account on system which can run your automated script and release the master password to access username and password.

|improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.