Suppose there is a file hosting server, and most filenames are strings with mostly CJK characters. Transmitting those characters in HTTP GET requests requires encoding them using UTF-8 (x3 overhead) then URL-encode escaping (another x3 overhead, x9 in total).
These filenames may be processed by front-end and back-end to identify and access resource metadata.
The solution I've thought of is to convert the filenames to UTF-16 encoding then base64-encode the result for use in transfer.
Currently, the filepaths are verified in our back-end for path-traversal attack (.. special file), special characters in C0 block (including tab, newline, etc.), and that's it. If verification fails, error notices are displayed.
My question being: assuming the result of decoding the "compressed" filename strings always goes through the verification in the backend, what additional verification, sanitization, or any processing should I apply in front-end and back-end?
I'm asking here because I believe the subtleties in base64 transcoding, UTF-16 Surrogates and Unicode normalization could bite me sometime.
