We have a situation where I am not finding any documentation on it. We are trying to scale across multiple machines/job by breaking up the keyspace by complexity. Its a 7 char completely random alphanum pwd.

We created a "wordlist" of 3844 "words" ie all combos of 2 alphanumeric chars (62 * 62 = 3844)


In CONFIG the plan was to set min/max len = 7

For scaling we were planning on creating 3844 rules, where each rule grabbed one line (Line N) from the wordlist followed by any alphanum for the remaining chars. Something like below.

[Line N word]?x?x?x?x?x = Rule N

We can then at the command line, call each different rule, as a distinct job, with all machines setup as clones of each other.

1) What would a rule like this look like (Rule N), I can only find references to Az or A0 (ie all wordlist items, or prepending in all the documentation and examples around the web)

2) Any other settings I may be missing to accomplish this?

2 Answers 2


In the past when I've sharded password cracking across many machines, I have just split my wordlist into sections and provided each machine a section of wordlist to try. This seems much more straightforward than manipulating rules per entry in the wordlist. (After all, rules are meant to be applied to the wordlist, not to a single word.)

  • This is basically the other option we entertained, except it requires 3844 wordlists updated for each job (instead of invoking a rules at the command line).
    – Ryan
    Jan 12, 2018 at 6:25

John's "--node" flag is designed to do this.

--node=MIN[-MAX]/TOTAL     this node's number range out of TOTAL count

Quoting the OPTIONS doc:

This option is intended to allow for some trivial manually-configured parallel and distributed processing. For example, to split the workload across two nodes (which could be machines, CPU cores, etc.), you'd specify "--node=1/2" on one invocation of John and "--node=2/2" on the other. (If you do this on just one machine and with the same build of John, you will also need to specify different "--session" names for the two simultaneous invocations.) The nodes are assumed to be same speed (if this is not the case, one will get ahead of the other and is likely to be done sooner, unless you're using a cracking mode and settings such that the session is not expected to ever "complete" - which is fine.) If your nodes are of very different speed, you may compensate for that by allocating ranges of node numbers to individual invocations. For example, if you use OpenMP-enabled builds of John on two machines, OpenMP is supported (with good scalability) for the hash type you're cracking, and one of the machines has twice more of similar speed CPU cores than the other, then you may use "--node=1-2/3" on the twice bigger machine (let it be nodes 1 and 2 out of 3 nodes total) and "--node=3/3" on the smaller one.

Efficiency of this approach to parallel processing, as currently implemented, varies by cracking mode and its settings (efficiency is higher for incremental mode and for wordlist mode with many rules, and lower for other cracking modes and for wordlist mode without rules or with few rules), hash type (efficiency is higher for slower to compute hashes), salt count (efficiency is higher for higher salt counts), and node count (efficiency is higher for lower node counts). Scalability may be limited. The highest node count you can reasonably use varies by cracking mode, its settings, hash type, and salt count. With incremental mode, efficiency in terms of c/s rate is nearly perfect (there's essentially no overhead), but some nodes may currently receive too little work - and this problem is exacerbated by high node counts (such as 100 or more) and/or restrictive settings (such as MinLen and MaxLen set to the same value or to a narrow range, and/or a charset file with few characters being used). With wordlist mode, for high efficiency the rule count (after preprocessor expansion) needs to be many times higher than node count, unless the p/s rate is low anyway (due to slow hash type and/or high salt count).

You'll also want to use "--fork=[cpu-count]" to take advantage of all local cores.

  • This is a large job that needs to scale across 1000s of cores/many machines. That method states its not good for > 100.
    – Ryan
    Jan 12, 2018 at 6:32
  • Ah, good to know. Might want to update the original question with that information. The "too little work" objection is related to fast hashes, and it sounds like you're dealing with a slow hash, so I suspect this would actually still work. Since it's easy to implement, you might give it a try and report back how it performs. Jan 12, 2018 at 17:08
  • The keyspace is actually in hundreds of trillions, (its larger than 7 alphanum) the hash rate is fast enough, but it will still take many weeks with 1000s of cores...hence needing to break it into jobs
    – Ryan
    Jan 12, 2018 at 21:14
  • Then I don't think that the 100s limit should impact you very much, as long as each node has a large enough chunk of work to come up to speed. You might consider posting the question to the john-users list as a cross-check. Jan 12, 2018 at 23:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.