E.g I have Apache installed on a linux server. Web site hosted by Apache supports file upload functionality. Should I use some kind of plugin for Apache to check those files or the OS should take care of it (sandbox like cuckoo )
You could check using apache2 plugins or by programming to whitelist extensions along with some tools.
In PHP, checking filename, filesize, etc is easy. You should check for both filename and extension. For filename it mustnt be more than 64 character long and only whitelisted extensions should be allowed. Dont rely solely on extension and magic bytes, try to identify by checking magic byte along with its actual content.
If its a image file, there is a tool called
identify from imagemagick in linux that identifies whether the image is valid or not. It works by checking magic bytes and image file content. Also, you should strip all exif metadata and prevent uploading of large files.