I have a program that can encrypt text, I tried to discover the encryption that is used in this program and could not figure it out, knowing that if the plaintext is P and the encrypted text is C, if we encrypt C, it will generate P again "double encryption generate the original message" also the encrypted text is a unicode charachters, strong text e.g. input "abcd", output "žœ›" is it possible to find the encryption technique?
As @David's answer pointed out, this is, in general very hard to do. Especially since some encryption algorithms like AES use a random initialization vector (IV) as the starting block. So correct implementations would generate a different IV each time, even if the plaintext is the same, resulting in a different message.
As an example, let's encrypt Hello World! in CBC mode with:
key: 0000 0000 0000 0000 (not secure, just for illustration)
If I use IV = 0000 0000 0000 0001, I get ciphertext (base64): KcS/oAzkKMi03Nrf65hlww==
If I use IV = 0000 0000 0000 0002, I get ciphertext (base64): 2wTIRW8sSb4XpE9Ue3xm6w==
(The IV is usually included with the encrypted message. I omitted it for ease of comparison)
What you are describing is an activity called cryptanalysis, which is the study of a cryptographic system.
Given that you've said you have a program that encrypts, I would start there. The first step is called Open Source Intelligence gathering, or OSINT (which is a fancy way of saying "google it".) Search the program's documentation for information about the encryption in use. Google the application name, looking for it to be accompanied by words like "crack" or "hack".
Next, I would analyze the program. Perhaps there are still symbols in the binary image. From a command line run a program like
strings programname to look at the various text strings in the binary image; if you see strings like
LFSR, or even
ROT13, those might be clues indicating the algorithm in use.
If all you have is the ability to encrypt and decrypt a message, you can try comparing the input and output. Encrypt a string of AAAAAAAB a couple of times, and see if the output is always the same. See if it follows a similar pattern, such as NNNNNNNO. Add another letter; take one away; see what happens to the encrypted data. If the same input message always yields the same output, it might be a simple substitution, or an ECB cipher. If the message always changes completely, it's probably using better crypto than you're going to be able to decipher. If a single letter change results in a completely different garbled message, you may be looking at a block cipher (again, difficult.) If you can ever determine that it's using a block cipher (changes always occur in multiples of 8, 16, or 32 bytes at a time) you'll probably need to disassemble and reverse engineer the actual program to recover the key.
Not in any easy, guaranteed, way, no!
If you have access to a range of encryption algorithms then you could attempt to build 'reverse lookup tables' (assuming you have the key) and then compare the result of your unknown encryption algorithm to them.
For example, given an encryption algorithm enc(text, key) create a table of: enc('a', key) enc('b', key) enc('c', key) ...
and then test your algorithm against 'enc' by comparing the outputs.
You could even try this with guessing the key if you don't know it but that's a lot of guessing and you'd have to be very lucky!
However, as @kelalaka points out, this will also only work easily with an encryption algorithm that produces the same output for the same input without some kind of 'reset'.
Other than this I can't think of a way of doing this.
Specific to this encryption, but not a general answer:
Based upon your edited additional information, it's likely a simple XOR as I suggested earlier in a comment. XOR has two interesting characteristics: It reverses itself, and XORing same values produces zero.
I suspect the reason it looks like unicode is that the KEY has a high bit set. Standard english ASCII is actually 7-bits with the 8th bit used for extended character sets. The XOR with the key produces a high bit character in Extended-ASCII.
Since you can enter your own text, enter a long string of either "space" or "@" if for whatever reason spaces don't work. If it's nothing but an XOR, it should output a repeating string where the repeat count is the key length.
The reason to use "space" or "@" is that both of these have a single bit set, hex 20 or hex 40 respectively. It makes it easier to compute the key manually if you only have to deal with a single bit change.
In your example of input "abcd", output "žœ›", Assuming Extended-ASCII hex this is
61 62 63 64 outputs 9E 9C 9B
The output you show is only for 3 characters "abc" because the "œ" is a single character in extended ascii callled "Latin small ligature". So XORing the 3 charaters of input with output to find the first 3 bytes of the key yields hex: FF FE F8
Given that key, entering a string of "@" (hex 40) should yield: "¿¾¸" in extended ascii.