How to know whether a textfile has been edited or tampered with?

Question

Is it possible to know whether a textfile, e.g. in XML format, has been edited or tampered with over time?

The context to my question follows:

I am a scientist in industry using a technology called 'mass spectrometry (MS)'. MS is an analytical technique used, e.g. in forensic analysis to determine whether a particular compound is present in a sample (e.g. drug of abuse in blood or urine).

Mass spec. datafiles are usually stored in flat-file format to the instrument vendor's private binary specification - their software can process it, but nothing else can. However, open standards for MS data exist, and most vendors support export to at least one open specification. These open standards are mainly XML based these days (eg mzML) and allow processing with open source applications, and also allow long-term storage (> 10 years) of the data in a format that doesn't require that we maintain an archived computer and the OS (or VM) and the processing software for long periods.

The vendor binary format provides at least some security against data tampering, however the XML formats do not. Hence the issue - the open formats are very useful for providing access to data over archival timescales, but security is a problem.

Answer 1

The default solution would be to use cryptographic signatures. Have every technician generate a PGP keypair, publishing the public key and keeping the private key secure.

When a technician made an analysis, they sign the result file with their private key. Now anyone who wants to verify the file can check the signature using the public key of the technician. When anyone changes the file, the signature won't be correct anymore.

Security consideration: Should any private key of a technician get known to someone else, that person can change the files and also change the signature to one which will be valid. This problem can be mitigated by having multiple persons sign each result file. An attacker would require all keys to replace all signatures with valid ones.

Alternative low-tech solution: Print out each result file, have the technician sign it the old-school way (with a pen) and deposit the file in a physically secure archive.

By the way: Do not assume that the vendor-specific binary format provides any more security against tampering than XML does. Just because you can't read and edit it when you open it with a text editor doesn't mean nobody else can reverse-engineer the format and build an editor for it.

Answer 2

Any form of digital signature will do. Here are a few pointers:

• For XML data, there is a digital signature standard (XMLSign). Unfortunately, this standard is rather poor and has an important security loophole (documents needs to be normalized through an XML transform before they can be signed. This is extremely hard to do securely since the transform itself becomes an important part of the signature).

• You can also use PGP or S/MIME to digitally sign documents, These will produce new, text-based and mostly readable but still tampered-proof documents.

• Finally, you can use detached signatures. Basically, it's another file that contains the digital signature linked to another document and can be used to validated the original data (no matter what the original format).

Let me add a few extra info here:

• Picking the right properties for the signature (algorythm, key type and size, etc.) is very dependent on the condition you set: how long do you intend to have the data secure, against what type of adversary do you intend to protect them (what's the value of a forgery? what would be the value of an attack that would break all documents signed with the same key ?), is there any regulatory requirement? This means that you should consult a specialist who can translate these business requirement and tranlate them into technical ones.
• I strongly advise you to add a secure timestamp to your signature. This will not only allow you to prove that a document hasn't been tampered with but also allow you to prove when the signature occurred.

Answer 3

I will outline the three main options and pros/cons of each.

Store backups of the files in a secure location

Pretty self-explanatory. The "secure location" can be a read-only medium (like CDs), or a network drive that everyone can read but only the supervisor can write to, or an online storage service (e.g. Dropbox) that makes it reasonably hard to forge file modification dates.

Pros

• You should have a backup system anyway

Cons

• If files are large, downloading them for verification can be time-consuming
• If the forger breaks into the secure location, he can cover his tracks

 

Store hashes in a secure location

A hash is a fingerprint of a file that looks something like ^{8f2e3f53aa90b27bda31dea3c6fc72f6}; if two files are just slightly different they will have a different hash. Take a hash of the original file and store it securely, then to verify a file has not been modified, take a hash of it and compare it to the stored hash.

Pros

• You need to securely store/check a ~32 digit code instead of an entire file

Cons

• You still need to access an external resource to check the file
• If the forger breaks into the secure location, he can cover his tracks

 

Cryptographic signatures

In this case, one or more people can "sign" the file and if any changes are made these signatures will be invalidated. Of course, if everyone who needs to sign the file is willing to (or tricked into) sign a tampered file then you can get away with the tampered file.

Pros

• The security information can be kept within the file itself, or otherwise on the same drive, meaning easier verification.

Cons

• Everyone who signs files needs to be very careful to prevent someone stealing their private key.
• Everyone who signs files needs to be very careful they know what they are signing.

Answer 4

Take your xml file, and your favorite holiday photo. Concatenate the files and compute several hash values of the resulting file.

The holiday picture ensures that it is extremely hard to produce a collision, even if the holiday photo file is public. Also, if you use several hash algorithms, it is unlikely that all of these will be broken under 10 years span.

Answer 5

Addressing vendor file-format security, expanding on what @Philipp says in the comments.

I've had a poke around a vendor file format (not mass spec but near enough for these purposes). It was made a lot easier by having the software installed, but I'm no expert in these things. I could easily change metadata (extracting the metadata was my goal in the first place) real data would have been harder but by no means impossible to modify. As metadata includes things like sample ID and date of test, that's a big enough vulnerability for things like "whose sample was clean and when?" as seems relevant to you, or "who first discovered this drug?" in other fields.

Some software provides some anti-tamper features (e.g. internal use of -- not necessarily crypto-grade -- hashes; user permissions when editing using their software). Reverse engineering these would be little more than trivial for someone with a decent bit of skill in most cases. With the software installed even circumventing the built-in features could be as simple as writing a front-end to call the vendor's DLLs, as these anti-tamper features are normally optional add-ons (in many fields they're not required or deprecated).

(This could have been a sequence of comments, but as my goal was to make the vendor-file issue clearer, it seemed better to write it properly).

Answer 6

How about making the technicians post pairs of unique file ids and their hashs to Twitter using their own accounts?

This will prove that:

• Data file with said id and hash existed at the time of posting
• The person who has access to the account trusts the content of the file at that point
• The file is not modified after the fact as Twitter does not allow tweets to be edited

This method provides at least comparable security to many of the digital signature-based answers and benefits like:

• Much simpler to learn and use (no complicated private key generation, opening or back up procedures)
• High redundancy (through twitter's backups and third-party twitter scraping sites)
• Built-in timestamp (that will probably stand in a legal proceeding without much explanation)

I recommend using at least SHA256 as the hash algo.

Answer 7

One of the easiest ways is to create a hash of the file and store it elsewhere so you know if it gets changed. Intrusion detection programs use this technique all the time to verify the integrity (or at least indicate if some attacker has been fiddling with system files).

Look at a program such as AIDE, you could run this against the directory containing the files (and possibly run it on-demand when a file gets added) to update its database of hashes. Nightly, run it to check and email you a report showing all file changes.

If you need to know the original, then a versioned filesystem might be a good idea. Every change that is made to a file is recorded and old versions can be extracted. Alternatively a backup system that detects new files and backs them up to a secure location could be used (and keeps all the old versions - or an attacker could just modify the file repeatedly until the original is deleted).

Answer 8

the open formats are very useful for providing access to data over archival timescales, but security is a problem

Big question: how are the archives being accessed?

The issue with hashing a plain-text file is the hash is character-accurate. Change one character and the hash will be completely different. Works very well for binary files like executable programs (where one byte out of place is usually disastrous) but fails on things like markup files - normalizing (or compacting) the whitespace will change the hash but have no effect on the data.

If you are handing the files around by email or read-write network share you will have to have secure storage for the hash, or anyone with half a brain can edit the file and then update the hash. If you have secure storage for the hash, why not store the data file in the same place and forget about the hash?

This is going to sound strange at first, but look at uploading the file and description to a local installation of something like wordpress or mediawiki. Access can be as open or secure as you want, and the platforms have user-specific file upload controls. Once the IT department has set it up properly, the write access to the files can be locked up as tight as necessary.