I have started looking into Crytography, and the obvious starting point is frequency analysis. As a task for myself I thought it would be good to compile a script which could be used for perform the process of frequency analysis. One thing I started to look at, was to get the stats based on a dictionary - I have subsequently learned this is incorrect and that better figures can be gained from literature. SO my question is this ... are there any recommendations on open-source literature which can be used for this purpose?



closed as too broad by GdD, Adi, Xander, TildalWave, Rory Alsop Feb 18 '14 at 14:22

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • Welcome to Security.SE - unfortunately, questions like this are so broad they don't fit into our question and answer format - so we avoid recommendations. Specific questions are welcome, however if they are about cryptography itself, you might be better served at cryptography stack exchange. – Rory Alsop Feb 18 '14 at 14:23
  • Thanks @RoryAlsop, I didn't see the cyrptography stack exchange. Apologies, however I feel audioPhil has answered this. Apologies also for the format of the question. :-) – MHibbin Feb 18 '14 at 14:29

A classic example of open literature would be the Gutenberg project of freely available ebooks, but you could also stick to large email collections such as the Enron data set that might better reflect the actual frequency distributions in communications between humans, or even crawl all Wikipedia articles.

The first two mentioned resources can be found on the open science data cloud. Archives of Wikipedia articles from many different languages can also be found online.

If you decide to crawl the data yourself keep in mind that the frequencies differ based on the language the plain text is written in before encrypting it with one of the weak, frequency preserving crypto systems. Choose sample data in the correct language or your results may mislead you.

However, keep in mind that you do not need to reinvent everything from scratch. Frequency distribution is a well-researched topic and your script could use precomputed tables by other persons. This is especially of interest, as frequency analysis has only an educational purpose and can mostly not be used to analyse real-world crypto. The wikipedia article on letter frequencies is a good source for many different languages. Recheck their sources to ensure how they were built (for example, the english ones are taken from a dictionary, while the portuguese ones are taken from literature).

  • Thanks @audioPhil, I didn't think about Wikipedia, which would help with other languages. I understand the real-world usefulness of this mini project, however, I thought it would be kind of cool. – MHibbin Feb 18 '14 at 14:31

Not the answer you're looking for? Browse other questions tagged or ask your own question.