First of all, a 1024-bit RSA key is much too small for comfort. Current recommendations for RSA are a debate between 3072 or 4096 (better security margin) vs. 2048 (better performance). See:
I want to know the maximum size of a file that I want to encrypt with my public key? and the maximum size that I want to decrypt with my private key?
The limit here really comes down not to RSA itself, because practical public-key cryptography is almost always hybrid cryptography where RSA is used to encapsulate symmetric encryption keys (usually for some AES-based algorithm) and the actual data is encrypted with those. So what in practice limits the size of the file you can encrypt is the design of the file or message format. For example with AES-GCM you cannot encrypt individual messages bigger than 64 GiB, and you should not encrypt more than 232 messages with one key, but that really means that to encrypt a larger volume of data the software should split it into smaller chunks and encrypt each one as a separate message, rotating keys after some number of chunks is exceeded.
(Side note: even for smaller files it's best to encrypt them in modest-sized chunks so that the software can reject forgeries without having to decrypt the whole file before producing any output. For example, one of the causes of the EFail vulnerability is that GnuPG outputs the decryptions of forged files before it performs an optional anti-forgery check.)
This is just one example; the general answer is that your question cannot be answered concretely with just the information that you give, and concrete answers require looking at your specific cryptographic software to see how it's designed and implemented. For example, the documentation for GnuPG tells you for some of its symmetric algorithms you should not encrypt files bigger than 4 GiB:
Due to its 1970s-era 64-bit block size, [3DES] should not be used to encrypt more than about 4Gb of data. Beyond that, though, it is solid as a rock, and very few GnuPG users will ever notice a problem with it. Provided you’re not encrypting more than 4Gb of data you may use 3DES with confidence.
I could not quickly find what's the failure mode if you violate this rule, and I cannot warrant whether it gives you an error saying you did something bad (as it should) or just silently produces a potentially insecure output file (which would be terrible). Caveat emptor.