Recently, I've become interested in applied cryptography and stumbled upon a link explaining how Linux estimates entropy.

At some point, we're told that entropy estimation is based on first, second and third differences of the timestamps of certain system events. Fair enough. What I didn't understand was the intuition behind this, explained in the link as follows:

This use of deltas is approximately the same as attempting to fit an n degree polynomial to the previous n+1 points, then looking to see how far the new point is from the best prediction based on the previous n points. The minimum of the deltas is used, which has the effect of taking the best fit of a 0th, 1st or 2nd degree polynomial, and using that one.

To clarify, here's what (I think) I've understood. Taking the example of mouse events in the link:

Mouse event times       1004   1012    1024    1025    1030    1041

1st differences              8      12       1       5      11

2nd differences                  4       11      4       6

3rd differences                      7       7       2

Fit an n degree polynomial to the previous n+1 points: I guess that would be taking the (i+1)-th diff, which are the 1st diffs of the i-th diffs. These could be used to predict the next values of the i-th diff, hence the 'fitting'. E.g. the 1st diff explains how consecutive values of the mouse event line (0th diff) change.

How far the new point is from the best prediction based on the previous n points: I guess this is given by the (i+2)-th diff? E.g. after the last mouse event time, 1041, the 2nd diff is 6, which is how far 1041 (the new point) is from 1035 (the best prediction). The prediction is obtained by taking the previous 0th diff value, 1030, added to the previous 1st diff value, 5.

Use of minimum delta: My best guess is that the entropy estimator chooses minimum value, because it is the best fit for the (i-1)-th diff (or (i-1)-th degree polynomial). I think I understand how this method picks the best fit, but I really didn't get the 'why' though.

My doubts/questions are:

  • I may be overlooking something obvious, but I still don't see the relation between my idea of n degree polynomial fitting (e.g. polynomial regression using a least squares method).
  • What's the actual rationale behind choosing the minimum delta? Why does it provide a good measure of how unexpected the next data point is? Is it because the minimum delta is the most conservative?

1 Answer 1


The estimator uses minimum third delta because the degree of the fitted polynomial of it is the entropy of the dataset. In the first example, the minimum delta was 2, its base 2 log is 1, therefore the data overall added 1 bit of entropy. Why the minimum delta? It is because using it allows to represent the entropy (in other words, unpredictability) of the entire dataset as a one-digit number. If you look at the entropy measurements in the first two examples (1 and 2 bits), you can see that they translate to 1st delta values of about 10 and 100, respectively. Accordingly, entropy of 3 would mean 1st deltas of about 1,000, 4 - 10,000, and so forth. In summary, in our delta representation we went from multiple to single digits by using the polynomial degree of the minimum third delta.

  • -1 Maybe I need to read this answer a few more times, but this just seems to be a re-stating of the question. After reading this, I'm no less confused about the intuition behind why this works. Can you expand this to be an explanation of the ideas behind the delta estimator? Mar 17, 2017 at 15:22
  • I was trying to answer the question directly - I will edit it to make it clearer, pls let me know if it helped any. Thanks for the feedback. Mar 17, 2017 at 15:27
  • @Mike Ounsworth I agree the first draft read awful - I need to get better at explaining the thinking about math formulas (which is rare anyways if you look around). Saved the edits - thanks again. Mar 17, 2017 at 15:49
  • I've spent a fair amount of time hacking around in the linux kernel's random.c and I'm not convinced that logarithms base 10 have anything to do with it, but I'll remove my downvote because your second draft is easier to read. Thanks. Mar 17, 2017 at 15:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.