First, it's not like we have much of a choice: public-key cryptosystems are standardized and widely available, change in this field is slow. We definitely need PK cryptography to secure the internet. The standardized PK protocols we have are unfortunately vulnerable to quantum computers.
Second, because the threats of quantum computers are still distant. Right now, RSA and elliptic curves are probably still secure. Recent years have shown advances in connecting more qubits, but we're still very far off from a general-purpose quantum computer that could be used to accelerate an attack on PK cryptography.
Third, because of forward secrecy, but this is a weak defense in the quantum context. In an interactive context such as TLS, RSA isn't used as an encryption algorithm but rather for authentication. The actual communication is instead encrypted using a symmetric cipher, and the ephemeral symmetric key is negotiated via a key exchange protocol. Symmetric ciphers like AES are not threatened by quantum computers. Unfortunately, common key exchange protocols like Diffie-Hellman or ECDH are a kind of PK crypto and are also vulnerable to quantum computing.
Fourth, because post-quantum cryptosystems are fairly young and have seen less analysis. For RSA and elliptic curves, we have a pretty good idea of the advantages and problems. For protecting communications right now, they are the best bet. In particular, current methods are perfectly suitable for ensuring the integrity and medium-term confidentiality of messages. But when a threat model has a horizon of decades, using one of the proposed post-quantum cryptography techniques could be more appropriate. Such long term choices are a bet about what happens first: do attackers first obtain practical quantum computers, or do they first learn about an exploitable flaw in one of the proposed post-quantum algorithms?