If I have a 2048-bit RSA and a 4096-bit RSA key is my signature and public key smaller when I use a 2048-bit key vs a 4096-bit one ? (In bytes)
Yes, the public key size and signature size is smaller when you use a smaller RSA modulus. The RSA public key composed primarily of the product of two very large prime numbers. This product is said to be the modulus, and this is what makes RSA keys so large. RSA 2048 has a 2048-bit modulus. The signature size also depends on the modulus size. In fact, it will always be about as large, but no larger.
An RSA signature is created by raising an integer representation of the hash of the message to the power of a secret integer kept in the private key. The result is divided by the modulus, and the remainder is your signature. Because the result is so massive, the remainder is also going to be quite large. The bigger the modulus is, the bigger the remainder is, and thus the larger the signature is.
If you need a secure signature scheme which has small public keys and signatures, try Ed25519.