Say I want to build a public API to access a website that doesn't have one. The website is protected by a standard username/password scheme and the API implementation will use scraping to get the data from it.

What would be the best way to handle passwords in this scenario?

My current best idea is this. Get the username and password in an HTML form on my site (the same one that serves the API), and generated a URL to access the API that includes an encrypted blob of username+password. The encryption is symmetric AES256 based on a key stored only on the API server. When the user will access the API URL, the API server will decrypt the credentials and access the remote website.

As an additional level of protection, I'm thinking of running the API as CGI, so that only a short lived process is ever exposed to plaintext credentials.

Does that make any sense? Are there any standard solutions to this kind of problem?


Possible duplicate: How to store passphrase in this situation?

1 Answer 1


Your current solution would work, however, you could keep the information on your server by tracking a session. When a user sends their login credentials, you could store them in a database (possibly encrypted) and then issue them a unique session id (should be difficult to predict the session id, eg. crypto rand + hash).

Once the user has obtained the session key, they would send that id along with every request to your server. When you verify that this session key is valid, you get the username/password out of the database, and make the request on behalf of the user. After some amount of time, you'd invalidate this session key and delete it and the login information from your database. This would help guard against replay style attacks.

If you include the actual login information and don't have some kind of age off, an attacker could obtain the URL used to access the system, and possibly make requests on behalf of the user. Of course, this depends on how you set up the URL etc.

Typically though, this sort of situation is handled by storing the credentials locally in some sort of secure (e.g. encrypted) storage, and issuing some kind of temporal session identifier.

While there may be some benefit in using short running processes, it is likely more cost than benefit. If an attacker is to the point at which they can read memory in your processes, the fact that the time in memory is short lived won't help much. Furthermore, you are still probably storing the encrypted credentials somewhere on your server (e.g. http access logs), along with the encryption key (which I'm guessing is in a database). If the attacker has access to both, the credentials will be compromised. Depending on which language you're using to write application, there may be some way to zeroize memory once you're finished with the sensitive information.

If you use timestamps in the URL, you're going to need to make sure that the timestamp can't be tampered with (HMAC, AES-GCM etc). Keeping credential data, encrypted or not, also represents a problem controlling information. If you suspect your encryption key has been compromised, you can easily delete the credntials from your server. If the credentials are in URLs potentially scattered all over the internet in browser caches etc, you won't be able to mitigate the leakage.

If you store the credentials in the database, along with whatever material you use to derive the key, you'll have much more control over the data. Assuming you have your database application secured, it is arguably more secure. I would suggest using different keys for each set of credentials, and using some kind of KDF (http://en.wikipedia.org/wiki/Key_derivation_function) to derive the key, instead of storing the key in plaintext in the database. You could also include some portion of the key derivation material in the session id you issue to the user.

Of course, if someone has complete access to your server, including the database and/or the VA space of your CGI processes, the credentials will be comprimised, but that is true with nearly any solution.

If I were writing this application, I would use session ids to tie a user's requests to a set of credentials. I would use a database table containing fields for the id, creation time, last seen time, username, password, and key derivation seed (this part depends on how you derive keys). The session id I would issue to the user would be a large, hard to guess value (stored in the id field), combined with a portion of the material needed to generate the key (this material wouldn't be stored on the server, so both a valid session id and the material stored on the server are needed to decrypt the credentials).

For instance, a session id may look like this:


The first part would be the id portion, and the second half part of the data needed to derive the key.

When a request is made, the id portion of the session id would be verified, and the decryption key generated using the key material portion supplied by the user (via the session id) and the portion stored in the database. Once the key is generated, you'd decrypt and use the credneitals.

This way, if an attacker gains access to the database, they don't have everything they need to decrypt all of the credentials. They'd need a session id for every set of credentials they wish to decrypt.

In order to age off sessions you could have a cron job, or some other mechanism, deleting expired sessions from the database.

  • I could include a timestamp into the encrypted credentials and not handle the request if the bundle is too old. I expect the URLs to be used to import data into Google Sheets or something similar, so I can't use session management here. Thanks, you gave me some good pointers.
    – elifiner
    Dec 31, 2014 at 7:55
  • If I don't store the credentials anywhere on the server, will running the backend as CGI (i.e. short lived standalone processes) help in some way? My thinking is the less time the credentials spend on my server the better.
    – elifiner
    Dec 31, 2014 at 7:56
  • @gooli Updated my answer based on your comments Dec 31, 2014 at 17:26
  • This is absolutely priceless, thank you so much for ypur detailed and well thought out answer. Feels like a single upvote and an accepted answer isn't enough. Os there anything else I can do for you?
    – elifiner
    Jan 1, 2015 at 18:40
  • @gooli, Nothing else needed, glad I could help! Good luck with your application. Jan 1, 2015 at 19:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .