There are multiple vendors of products that claim that their products perform what they have called vaultless tokenization or reversible tokenization, without storing the token and associated tokenized data. However, they also say that tokenization is not the same as encryption, and that they have a proprietary algorithm which tokenizes and de-tokenizes input data.

My question is - isn't this really just fancy encryption (and if it keeps the shape of the source data, possibly even less secure than typical encryption)?

I thought the original idea behind tokenization was that you generate a set of pseudo-random data, assign that pseudo-random data to the input data, and store the mapping between the two in a hardened vault. If you then store the tokens in your less secure systems, and data is stolen, it will be virtually impossible for an attacker to de-tokenize the data, unless they also have a copy of the mapping in the vault.

If the token is derived from the source data, doesn't that make it much easier to attack (assuming the mapping is not also stolen by the attacker), since it's really just an encrypted version of the source data?


The line between Encryption and Tokenization is blurry.

The answer to your question is that it depends. There are different ways of accomplishing vaultless tokenization. These different breeds of vaultless can be categorized into two main ideas: lookup table techniques, and format-preserving encryption (FPE).

Using FPE is obviously encryption at the very core, so calling it tokenization is a little bit of a marketing play, however, the justification of calling it a token has to do with what is produces, not how it was created. FPE, although it is encryption, can produce a ciphertext that is the same length and format as what was passed in. In this way, FPE produces "tokens" in the aesthetic sense. However, these tokens were generated using a mode of AES encryption, and so in the eyes of security, these tokens must be treated like encrypted data.

The other breed of Vaultless uses pseudo-random values and maps them to the input (i.e. lookup tables). How is this method considered vault-less? Well in order to eliminate the need for a vault (aka a big database), you must reduce the number of entries in the lookup tables, to reduce its size. If you can come up with a way to keep the size of the table down (!!! while also maintaining the reversibility of the lookup process) then you can justify calling this thing vaultless. Most lookup table varieties of vaultless use tables that are small enough to be stored in cache or something.

  • Out of curiosity, how do traditional auth tokens, for example encrypted JWTs, compare to this? – Mike Ounsworth Sep 20 '18 at 16:51
  • 1
    Auth tokens and Data Security tokens share a name but they are very different and are used for different use cases. A like to think of data security tokens as the tokens you get in an arcade. The token you get in an arcade is not really worth anything outside of the arcade. You swap the universally valuable currency for tokens, so there is no reason for someone outside of the arcade to steal the tokens. In the same way, a data security token is not valuable to a hacker, since it is essentially random data. – jburcham Sep 26 '18 at 16:04
  • Thanks, this makes a lot of sense. So in the vaultless world, even though the data is 'reversible', an attacker will have a very a hard time reversing the process, since the 'key' is quite large. Format preservation could lead to data leaks through other methods though? Like if I have a list of english names and 'Xvd3' appears the most, I can guess that Xvd3 = John. But there is no way to use that information to determine what 'Xd33v' is, because the lookup tables are not doing simple substitution..? – N West Sep 26 '18 at 18:38
  • Yes, the key would be your standard AES 128, 192, or 256 bit keys, since FPE is a mode of AES. Keep in mind, this is if you are using format preserving encryption as your vaultless solution, rather than a lookup table vaultless solution. If you are curious about the details of FPE, this is the specification: nvlpubs.nist.gov/nistpubs/specialpublications/… You can prevent the attack you are describing by using an initialization vector. This makes the ciphertext for identical inputs different each time. – jburcham Sep 27 '18 at 19:55
  • 1
    With a lookup table vaultless solution, there is no key or encryption algorithm. The highly secure secret in this case is the "mapping" or the database that can tell you which random token is assigned to which ciphertext (encrypted PAN for example). – jburcham Sep 27 '18 at 19:57

There isn't a definitional difference. The vaultless tokenization solution whose vendor fastidiously insists is "not encryption" really is encryption, definitionally speaking.

There is a substantial technical difference, however, in that they don't use a format-preserving encryption mode on top of a block cipher like some other solutions do, but rather a format-preserving primitive built on top of tables of random numbers. This conference submission abstract briefly describes it:

A Novel Approach to the Tokenization of Credit Card Numbers

Bart Preneel, COSIC, Katholieke Universiteit Leuven, Belgium
Ulf Mattsson, Protegrity, USA

Encryption techniques are used to ensure the confidentiality of sensitive data. They are typically defined as mappings on bitstrings; they can be defined as a mode of operation of a block cipher (a keyed random permutation on strings of 64 or 128 bits) or based on a stream cipher (that typically operates at the level of bits, bytes or 32-bit words). In order to satisfy strict security definitions, encryption schemes need to be randomized, which means that the ciphertext is larger than the plaintext. For some applications, such as the protection of credit card numbers in certain contexts, both constraints are undesirable: the plaintext space consists of digits rather than bits and the mapping from plaintext to ciphertext has to be a permutation, hence there is no room for randomization. The encryption operation is also called tokenization. It is definitely possible to define a secure tokenization based on a block cipher such as triple-DES or AES. In this submission, we present a completely new approach, in which a highly efficient block cipher is designed from scratch by using S-boxes defined on strings of n digits (n is typically 5 to 7), that can be interpreted as large keys. We will show that if the number of plaintexts encrypted with a single key is limited, a very high security level can be obtained using this approach. It can be proven that under realistic constraints, the security of the scheme is equal to an “ideal” tokenization scheme.

Note how they say:

  • "The encryption operation is also called tokenization." The authors are aware that tokenization is, definitionally speaking, a form of encryption.
  • They label their system a "highly efficient block cipher [...] designed from scratch." I.e., encryption.
  • Their block ciphers, instead of working on binary bitstrings like conventional ones, is "designed from scratch by using S-boxes defined on strings of n digits (n is typically 5 to 7), that can be interpreted as large keys." I.e., instead of using an FPE mode to build a format-preserving cipher from a conventional bitstring blockcipher like AES, they build a format-preserving cipher directly.
  • I guess the question is what qualifies as 'limited', and 'realistic constraints'. So yes, this is encryption, but a 'new' way of doing it using a secret set of tables of random numbers. Thanks. – N West Sep 26 '18 at 18:42
  • 1
    This is spot on. Dr Bart Preneel presents a document on slideshare entitled "Vaultless Tokenization: A Novel Approach" which states "We present a completely new approach, in which a highly efficient block cipher is designed from scratch by using S-boxes defined on strings of n digits (n is typically 5 to 7), that can be interpreted as large keys." slideshare.net/JohanDentant/qfdrpreneel – Brad Schoening Feb 8 '19 at 15:02
  • It seems to me then that a vaulted solution would be better, otherwise one might as well stick with encryption and an HSM. – Erica Kane Jul 30 '19 at 2:00

The challenge would be that in vaultless tokenization the data anonymity is lost, we can rotate/encrypt based on some lookup but for any validation of code for mock-real data it won't happen. I would always prefer vault based tokenization and encrypt data at transit & rest.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.