I have been tasked with some backend automation that needs to scrape data off an internal website using an employee's username/password to log in to the site.
There is a backend database server, but I do not have access to said database (bureaucracy is a stumbling block). Specifically, all employees have access to this data via a username/password login on the internal site, and my scraper needs to as well.
Since I have an automated scrape that needs to be done at regular intervals, I unfortunately require storage of a set of credentials in plaintext, such that they can be used to authenticate with the site, and then scrape the information for digesting. The host system for these scripts is locked down, behind key required SSH, and is probably stronger than the physical security in this office anyway (someone with access to the office could get this information much more easily than attacking my server).
My question: How can I make the stored plaintext passwords on the client as low a risk as possible, given that it is impossible to avoid altogether?
As an addenda for potential additional questions, the site is a simple HTTPS form, username/password are POSTed, and an auth token (session ID) is used for further request. Unfortunately the ID expires, so I need to redo the password auth every 2 hours.
Based on questions I have received so far, my current theoretical implementation is to keep the username/password stored, encrypted on disk, and require the decryption key at first run/reboot as well as at a set time period. Theoretically this would reduce the risk of the device being stolen physically, and compromising the authentication information.
An alternate option I'm considering is to have an intermediary service running on a separate or locked down instance/container, whose image would be encrypted on disk. This can be the middle man authentication service, and would have a single purpose, to fetch the session ID key from the site on request, and store the temporarily decrypted username/password information. This might serve to further separate the user/pass info from the potentially more exposed service.