Should we generate a strong password offline, keep it closely guarded secret, and use the same one across all copies of the device?
Absolutely not. There are two primary problems which make the risk (likelihood x impact) of an attack high:
- The first is that you are likely underestimating likelihood: It is very hard to keep something a secret in your own company. It is nearly impossible to do so in the field. For example, it is simple enough to dump the contents of eMMC using physical means. See 1.
- The second relates to the impact. If you used a different secret per device, an attacker would have to do an eMMC dump (for example) for each device, which is extraordinarily hard to scale. However, if only one secret is used, then every device can be compromised with no extra effort (i.e. the work effort for the attacker is constant, rather than linear w.r.t targets).
should I write code that on the first boot changes the password to salted hash of the device serial number
This makes things slightly better, but not by much.
If the salt is shared across all devices, you have a problem very similar to what I previously described: an attacker simply needs to do a memory dump one time and all devices become vulnerable (a serial number is obviously not cryptographically strong) so best practice is to assume an attacker will be able to take the (shared) password and the (shared) salt, guess the serial number and compute the correct password per target device with little effort.
Using a unique salt per device is a definite improvement assuming you don't store the actual password anywhere (only store the salted hash). This means you shouldn't "change the password [on first boot]" as suggested. Doing so provides some window of opportunity for an attacker to dump memory and get the password before first boot. At which point, every device is again vulnerable.
is there other well-known practices?
You can mitigate most of the above issues by relying on public-key cryptography. Generate a private/public key pair in your back office for each device, place the public key on the device file system and the private key in some back office "key management system".
To login, you essentially "sign" some piece of data (using the private key) and send this to the device. The device verifies it using the stored (unique) public key. SSH already supports this, see 2.
Analysis
Now let's look at the likelihood and impact of an attack:
Goal: Prevent malicious attacker from logging into to device over SSH.
Likelihood: An attacker can not log in without presenting a proper private certificate (which amounts to guessing a sufficiently long key: impractical) OR by tampering with the device's memory to replace the public certificate OR due to a software implementation bug in the login procedure.
Option 1 is impractical. Option 2 is possible, albeit somewhat time consuming and not scalable. Option 3 is perhaps the most likely, but hard to avoid unless disabling SSH is allowable.
Impact: The analysis here is easy. Since we have a unique key pair assigned to each device, an attacker who does any amount of work to crack or change the key pair for a target device will NOT be able to reuse the work on other targets. Let's call the impact of this "low"
Given this our risk is likelihood x impact = low/medium x low = ~ low risk
Of course, you really need to tune your risk model / tolerance to your organization and application. If, for example, these devices are used to control a safety-critical feature of an airplane or car, you may not tolerate even one-time, physical attacks.
Extra resources
It may be helpful to also review NIST's many guidelines for key management: https://csrc.nist.gov/Projects/Key-Management/Key-Management-Guidelines
Specifically, Chapter 10 of part 3.
References
1: https://www.blackhat.com/docs/us-17/wednesday/us-17-Etemadieh-Hacking-Hardware-With-A-$10-SD-Card-Reader-wp.pdf
2: https://www.ssh.com/ssh/key/
3: https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-57Pt3r1.pdf
