We have a web service logging/recording user data sent over https by our client application. Is there a way to securely ensure that the data is getting sent by the client application and not by a fake tool impersonating our app? Ideally, I wouldn't like to have to resort to things like hiding an obfuscated key in the client or other “security through secrecy” techniques…
You can't. A server fundamentally can't trust a client (there are some exceptions - if the client machine is hardened (hardware verifies firmware, firmware verifies OS, OS verifies every application running), it might be possible to do that (but not really - someone with a soldering iron could always modify the hardware verification), but if you don't 100% control the client, you're kinda out of luck.
You need to ask yourself: What are the risks of the client uploading bogus data? What are the consequences of not authenticating that it's your application uploading the data? One hard question you should ask: Does the client have an incentive to upload bogus data? If the user doesn't have an incentive to upload bogus data, they probably won't bother.
One thing that you can do: You can authenticate the user running the application (via any one of a number of authentication mechanisms) and then save the uploaded data along with the authenticated user ID (and IP address). That way if the user does upload bogus data, you can track back who made the change. Of course you've now made the data PII which may have privacy issues for your service.
You do need to authenticate the client, and that requires there to be some sort of secret. If a human user is controlling the client, you may just be able to register and authenticate the user, using some secret they know or have access to.
If the client is automated and you don't control the hardware and software environment that the client relies on, you're out of luck in the face of a determined attacker who has control of the client. But you can make it hard.
Your problem seems related to the general problem of licensing client applications, and a good overview of one approach to that (for android apps, in Java) is here:
You can adapt that to your environment or look for other open source libraries suited to your situation.
By the way, "security via secrecy" is fine - secrets like private keys are part of most good schemes. It is "Security thru obscurity" that people try to avoid, i.e. just making things complicated, or pretending that attackers won't understand your algorithms.
If you take a look at the 10 immutable laws of security, law #3 explains the problem at a fundamental level:
Law #3: If a bad guy has unrestricted physical access to your computer, it's not your computer anymore
You could use kerberos to perform mutual authentication between client and server. You would preferably have some out-of-band method for initially identifying the client, e.g. issuing the user with an identifier. of course, you now rely on that identifier not being compromised...
As the other guys said, you cannot trust a client, until the client authenticates with a cert YOU created. At this moment the whole problem boils down to a key/cert distribution problem:
- How does the client know that it's getting a good cert from a real server?
- How does the server know that it's not giving away good certs to malicious clients?
Until these questions are answered (correctly!) all other events following the authentication are moot.