Can it be checked if the file/directory was actually copied to the drive at a later time and not at "date modified" time that shows on the usb-drive-2?
Date stomping is the problem. It's a technique where nefarious behavior is masked by editing the dates of files. And it's incredibly simple--here's an example on Windows in PowerShell:
(Get-Item '.\somefile.lnk').LastWriteTime = [datetime]"10/26/2018"
With that being said and forensically speaking, there are places in the Operating System (on Windows) where, while not impossible, is significantly harder to alter. There is a master file table on Windows at the root of C:\$MFT which is hidden but tracks all file events. These are commonly referred to as MAC times (modified, accessed, created).
There are tools in a number of different suites to interrogate these parts of the file system and create searchable timelines for file level events--this is often used in malware hunting. SANS has a distro called SIFT workstation where forensic tools can analyze an offline disk or disk copy (via 'dd' or other disk cloning capabilities). There are also expensive forensics tools that allow you to interactively pull this information from an online machine. FTK, EnCase, and the Sleuth Kit are probably the most well known.
However, this only tracks traditional disk-level events. USB device copying events are not tracked by Windows -- you need third party software for this. With the aforementioned software though, you could get a better feel for what happened on drive-1 and then correlate those MAC times (via the $MFT) to what Explorer is saying for the USB drive to get a rough idea.