How are hidden private URLs being "leaked" to third parties?

Question

We're hosting a web site where the server processes a stream of images, generating a unique ID (uuid) for each new picture and storing various data against the ID in a MySQL database. The site is monitored through a protected dynamic (AJAX) UI, which displays a real-time stream of image previews which include the uuid in the URL, something like:

https://<SERVER>/<root>/image/a756f99c-f0ce-469d-80b7-8c8f3c0a0581

When I trawl through the Apache access logs I see requests to some of these image URLs from a small number of third party IP addresses, fairly consistently originating from either Amazon AWS EC2 instances in the US or "Bayer Business Services GmbH" in Germany, according to iplocation.net.

The question is how these image URLs are leaking out so that third parties are able to request them?

The URLs aren't aren't listed anywhere, so wouldn't be found by any crawling/scraping process looking at the site. The fact that they include a uuid means that even if a third party had the base URL (the "https://<SERVER>/<root>/image" prefix) it would be pretty much impossible to guess a valid uuid to append.

My guess so far would be one of the following:

1. The browser I'm using to monitor the site might be leaking request URLs to external parties, maybe for supposedly innocent reasons. However, I'm seeing these "URL leaks" whether I'm using Chrome or Firefox.
2. Some other process on the monitoring PC is leaking request URLs, for example maybe Bitdefender shares a selection of request URLs with their servers for safety checking (blacklisting) purposes. However these leaked URL requests are often 1 or 2 days after the original valid request, so this would imply that Bitdefender (or other) would have to be saving URLs for days and re-testing later.
3. Something on the server is leaking the URLs, which in practice would point the finger at Apache 2.4.x. That would seem like a major security loophole, if Apache was able to send out request URLs to external services. Very unlikely I'd have thought.

In practice the server is pretty well protected so these leaks are unlikely to cause real damage. Apache is Docker containerised, running under Linux, and fairly well isolated from the database and host. All web traffic is https, and unauthenticated requests are trapped and responded to with an http 404, so the Apache logs show a 404 for these "leaked URLs".

Nevertheless this seems somewhat sinister, that either a browser, browser extension, or server component such as Apache could be pushing out request URLs to some unknown external third parties, who can then reissue those request URLs for unknown purposes.

Can anyone explain how this is happening?

Answer 1

Answering my own question in case it's useful to others. In order to distinguish between the 3 possible sources of URL 'leaks' I stopped monitoring the site from a Windows PC (Win10+Bitdefender+Chrome+various plugins) for 10 days from Oct 20-29th inclusive, and switched to monitoring only from the Linux server itself, again using Chrome.

At the end of that period the Apache logs show that no external requests were seen for the hidden & protected site URLs. This eliminates the possibility that URLs were being leaked from the server itself, and probably also rules out Chrome as the culprit (unless the source is a Windows-only plugin). To my mind this strongly points the finger at Bitdefender, which appears to be passing some user requests onto external servers, for purposes which are not clear, or documented.

In case it helps others spot the same pattern of mystery requests, the general form is either:

• An initial URL GET from 195.47.249.18 [Bayer Business Services GmbH] followed immediately by the same URL from an AWS EC2 instance in the us-west-2 region.
• Less often, a single 'region-less' EC2 request, for example from ec2-35-173-186-156.compute-1.amazonaws.com.

The complete list of external request sources for hidden URLs on Nov 1st 2020 is as follows:

195.47.249.18 - - [01/Nov/2020:01:08:10 +0000] "GET /{redacted}/02a7e770-a5e4-4879-baa5-e0d2aba771a5 HTTP/1.1" 404 - 1961 
[Bayer Business Services GmbH]

54.186.168.31 - - [01/Nov/2020:01:08:10 +0000] "GET /{redacted}/02a7e770-a5e4-4879-baa5-e0d2aba771a5 HTTP/1.1" 404 - 3630 
[ec2-54-186-168-31.us-west-2.compute.amazonaws.com]

52.38.242.51 - - [01/Nov/2020:01:25:51 +0000] "GET /{redacted}/b594cd3a-c464-4b3a-9c6e-60fb54ff07f0 HTTP/1.1" 404 - 3373
[ec2-52-38-242-51.us-west-2.compute.amazonaws.com]

34.219.247.27 - - [01/Nov/2020:01:39:48 +0000] "GET /{redacted}/60e876e4-15ef-4ad9-a92f-5701b29c3917 HTTP/1.1" 404 - 3327 
[ec2-34-219-247-27.us-west-2.compute.amazonaws.com]

34.213.140.185 - - [01/Nov/2020:01:49:58 +0000] "GET /{redacted}/b02171fa-1a12-4952-a459-7d8aa444b4d3 HTTP/1.1" 404 - 3198 
[ec2-34-213-140-185.us-west-2.compute.amazonaws.com]

52.10.55.64 - - [01/Nov/2020:01:52:15 +0000] "GET /{redacted}/b5690f82-3c51-4338-a1d8-05430ffbbff2 HTTP/1.1" 404 - 3414 
[ec2-52-10-55-64.us-west-2.compute.amazonaws.com]

54.190.25.188 - - [01/Nov/2020:02:01:31 +0000] "GET /{redacted}/5cd37165-b5a0-41c2-9939-1ef651e04cf4 HTTP/1.1" 404 - 3701 
[ec2-54-190-25-188.us-west-2.compute.amazonaws.com]

52.25.94.232 - - [01/Nov/2020:02:06:02 +0000] "GET /{redacted}/5a9d6b04-1bd3-4089-823a-d7e767f91224 HTTP/1.1" 404 - 3062 
[ec2-52-25-94-232.us-west-2.compute.amazonaws.com]

34.220.193.71 - - [01/Nov/2020:02:16:52 +0000] "GET /{redacted}/05e4d8d3-6c57-4d8d-8a50-9a4502db96e3 HTTP/1.1" 404 - 3589 
[ec2-34-220-193-71.us-west-2.compute.amazonaws.com]

3.231.31.179 - - [01/Nov/2020:02:21:45 +0000] "GET /{redacted}/8ec59182-c8fd-48fd-b826-3f01f7341c1c HTTP/1.1" 404 - 2754 
[ec2-3-231-31-179.compute-1.amazonaws.com]

35.165.202.25 - - [01/Nov/2020:02:21:45 +0000] "GET /{redacted}/8ec59182-c8fd-48fd-b826-3f01f7341c1c HTTP/1.1" 404 - 2734 
[ec2-35-165-202-25.us-west-2.compute.amazonaws.com]

35.173.186.156 - - [01/Nov/2020:02:37:32 +0000] "GET /{redacted}/02a7e770-a5e4-4879-baa5-e0d2aba771a5 HTTP/1.1" 404 - 3222 
[ec2-35-173-186-156.compute-1.amazonaws.com]

54.172.84.153 - - [01/Nov/2020:02:54:52 +0000] "GET /{redacted}/708f9b0f-252f-42da-a50f-9ade2cc96667 HTTP/1.1" 404 - 1892 
[ec2-54-172-84-153.compute-1.amazonaws.com]

18.237.236.242 - - [01/Nov/2020:02:54:52 +0000] "GET /{redacted}/708f9b0f-252f-42da-a50f-9ade2cc96667 HTTP/1.1" 404 - 1913 
[ec2-18-237-236-242.us-west-2.compute.amazonaws.com]

34.212.199.166 - - [01/Nov/2020:02:56:45 +0000] "GET /{redacted}/9b40a90e-d0d4-43df-b451-9713ea1731a6 HTTP/1.1" 404 - 3619 
[ec2-34-212-199-166.us-west-2.compute.amazonaws.com]

3.87.65.70 - - [01/Nov/2020:03:36:08 +0000] "GET /{redacted}/9b40a90e-d0d4-43df-b451-9713ea1731a6 HTTP/1.1" 404 - 1951 
[ec2-3-87-65-70.compute-1.amazonaws.com]

52.90.240.183 - - [01/Nov/2020:04:37:12 +0000] "GET /{redacted}/df96f0ce-e7c2-4de2-ba6e-8aca06e0b4d9 HTTP/1.1" 404 - 3491 
[ec2-52-90-240-183.compute-1.amazonaws.com]

Answer 2

You don't say what these images are or how they are intended to be used but the easiest way (Occam's Razor) for the links to leak that comes to mind is via a legitimate user posting a link as in:

"Hey dude, here is an awesome picture HTTPS://127.0.0.1/awesome.jpg"

The authenticated user sees the image and has no idea that an unauthenticated user will not.

I've lost count of how many times I've seen people send out presentations with links to their personal machine files. It works fine for them so they have no idea it won't work for anyone else.

If it's just a relatively small number with no massive brute forcing in the logs, Authorized user posting is my bet. Depending upon context and logs, it may even be possible to identify the authorized user(s) doing the postings.