It turns out there is a need for our application to be able to poll data at a specific schedule from external API on behalf of clients. The API can be any rest API that the client would choose. We must be able to poll the API without client inputs after the initial setup. All components of the application will be hosted on cloud using cluster managing system.
Note: The client APIs may or may not support Oauth.
To support this, we have realized we have to store API credentials securely and in a reversible fashion.
To achieve this, we believe the following steps are required:
- When the API schedule is set up, client must provide credentials to the API
- This is transferred Securely using HTTPS to the servers
- The servers Encrypt the credentials using Symmetric Key Encryption
- When scheduler detects an API call has to be made, decrypt credentials -> Authenticate with the API -> Call the API
- Store the data
Steps 3 and 4 I believe require us to be really cautious.
We need to keep Encryption keys available to decrypt the data. We are thinking of using cloud KMS solutions (AWS or google or something else)
Storage of encrypted data we believe is important aspect well.
So the questions are:
- At a high level have we missed a security step / process? If so, where can I learn more about it.
- Where should we store the encrypted data? Can we keep it in a database (SQL, No-SQL)? Is there a better approach than database, such as credential manager in windows or something else?
- if we store encrypted data in the database what steps should be required to ensure the database is safe?
I have read the following links so my questions are more towards storage and understanding if there are missing items: