would be my passphrase saved somewhere on disk, or only in memory ?
The ssh-agent only holds your unencrypted private key in memory while you’re logged in. Obviously the passphrase is used to unlock the key, but it is not stored on the disk itself.
Source: SSH Agent Explained:
[The SSH-agent] doesn't write any key material to disk.
would be someone with my login credential be able to recover the passphrase and/or the unencrypted private key ?
Because the key is only stored decrypted in short-term memory by the agent, the only is way to steal the key is while you are logged in, so logically, if someone knows your passphrase and password, they can then login and steal your ssh key.
And if while you are logged in, someone has either physical or remote access to your Ubuntu system, then can they just steal the SSH key outright.
Storing the key decrypted in memory is not quite as secure as decrypting a key only when needed (such as for temporary server authentication), but certainly better than storing a (decrypted) private key on the hard drive.