See the source code for vboxmanage internalcommands. The command
passwordhash is mapped to the function
CmdGeneratePasswordHash near the bottom of the file. Above the function definition, there is an elaborate comment detailing all the algorithm's properties:
Generate a SHA-256 password hash
So it is a plain SHA-256 hash. No salt, iterations, or memory-hardness. And it's not encryption, so they are not encrypting your password: they are hashing it. But not very well.
SHA-256 by itself does not protect your password very well because it is fast to brute force. Only a very strong password is secure and will not be recovered from the configuration file, so a randomly generated password (stored in a password manager) is recommended. If you don't use a very strong password, consider the configuration to pretty much contain your password in plain text.
Testing the theory, the documentation you linked also mentions an example hash:
Googling the hash, we find not only the documentation page, but also that this maps to "secret".
While looking this up, I also found a passlib page mentioning virtualbox. I didn't know of that project, but it seems this might be useful for similar questions when you want to know what hash is used by various software.
Note that when generating your password with vboxmanage, your password will be saved in your command history. To avoid this, you could use the following:
$ read -s pwd
$ VBoxManage internalcommands passwordhash "$pwd"
read command, with the
-s flag to not show the password, you type your password. When you press enter, the prompt will return and you can enter the second command. If you would do
echo "$pwd", you will see that this variable contains your password.
Better yet, generate a password randomly instead of entering it:
pwd=$(</dev/urandom tr -dc a-zA-Z0-9 | head -c 16);
echo "Password: $pwd"
VBoxManage internalcommands passwordhash "$pwd"