4

This TechCrunch article discusses how Dropbox uses hashing to check whether you're sharing copyrighted material "without actually looking at your stuff." Specifically, they offer the following (unsourced) "official comment" from Dropbox:

There have been some questions around how we handle copyright notices. We sometimes receive DMCA notices to remove links on copyright grounds. When we receive these, we process them according to the law and disable the identified link. We have an automated system that then prevents other users from sharing the identical material using another Dropbox link. This is done by comparing file hashes. We don’t look at the files in your private folders and are committed to keeping your stuff safe.

The hash-and-compare strategy sounds like a reasonable idea, but easily thwartable. What's to stop me from creating seed.txt, a text file with some random contents, and zipping that together with copyrightedContent.mp4 to create innocuousArchive.zip? Or, if they expand archives, adding a bunch of random (or zero) bytes to the top of the file itself?

(The purpose of this question is not necessarily to determine how Dropbox itself actually does this, but rather to explore different ways to do this in general—or whether it is feasible at all.)

  • This question isn't really answerable by us, because it's proprietary. I'd imagine they're using locality-sensitive hashing, also known as fuzzy hashing. It's the same kind of technique that Google uses to identify infringing video on YouTube. It won't help against encrypted or compressed (e.g. zipped) files, though. – Polynomial Feb 1 '15 at 23:36
  • @Polynomial The proprietary aspect is why I included the last paragraph. Fuzzy hashing is an interesting concept—thanks for the link! – wchargin Feb 2 '15 at 2:27
2

The actual processes are not publicly known, however they do maintain an official Tech Blog which gives some insight to the internals of Dropbox. For instance, their post on Streaming File Synchronization (July 2014) provides a high-level overview of the "Dropbox File System" which dictates the handling of all user data in a highly efficient and reliable manner.

File System

The post explains the upload process pretty nicely, and they confirm that files are fragmented into as few blocks of length <= 4MB as possible to be stored in that form on their block store. They are progressively hashed (SHA-256) as they are isolated, and the hashes are used to tie together blocks in Dropbox's block store (blockserver). Associations between hashes are persisted in the meta store (metaserver) and feel a bit like directory listings.

Dropbox File System

Hashes act as metadata for the file blocks (all length <= 4MB) which is used by Dropbox to determine whether or not it's necessary for the client to transmit that block as a fresh addition to their servers. Whether or not the client needs to upload any blocks to Dropbox's blockservers will be stated in a response from their metaservers when the client specifies the block makeup of a particular file.

Uploading Data

For all intents and purposes of this answer, you can just consider "Server" to be one entity acting as both a metaserver and blockserver; both are under the control of Dropbox anyway. Here's what an upload might look like between a Server and some Client:

Client: SET "/video.avi" [h1, h2, h3, h4] ---> Server # cmd: try set record
Server: GET [h1, h3, h4] --------------------> Client # err: need h1/h3/h4; h2 exists
Client: ADD [h1, h3, h4], [b1, b3, b4] ------> Server # cmd: add blocks h1/h3/h4
Server: ACK ---------------------------------> Client # ack: blocks stored
Client: SET "/video.avi" [h1, h3, h4] -------> Server # cmd: retry set record
Server: ACK ---------------------------------> Client # ack: record defined

Simplified/changed a lot, omitted details not relevant to question. Intentional. Actual proto form on blog.

Detecting Content

Data blocks and their corresponding meta/hash values are held in separate stores from one another. If hash values are known ahead of time (e.g. given a massive database of DMCA-encumbered files and their constituent blocks) and they are only checking against the metastore entries, then I suppose they could carry out automated detection without looking at your files. Would be a fairly obscene stretch of the English language if that's what was meant.

I don't think it's unlikely that a considerable portion of maximum-length blocks (in this case 4MB blocks) of original and actively copyright-protected data would persist in an altered copy of protected material, even if changes are made to some portions of the file to the extent of obscuring its visual and/or audible identity from the vantage point of a human observer.

I believe Dropbox may actually handle video differently (says it on their blog, I think), but I'm not sure if they re-encode media to a standard type when played in the browser (if so, guess it would be another opening to accidentally see things they're not supposed to look at).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.