Questions tagged [web-crawler]

For questions about how to use or defend against web-crawlers or web-spiders.

Filter by
Sorted by
Tagged with
1
vote
1answer
122 views

Why is my web site being scanned for license.txt, and should I be worried?

Lately I am seeing multiple daily 404s for variations of "license.txt", e.g., "wordpress/license.txt", "blog/license.txt", "old/license.txt", "new/license.txt". Here's a little snippet of slightly ...
0
votes
0answers
26 views

Burpsuite Crawling - Display Function Location on Result

Is there is any way for me to display Burpsuite crawling result as Function Location? What i mean by function location is like for example: URL: x.com/functions/showdata Function Location: User > ...
1
vote
1answer
85 views

Questions about SOCKS5 security

I'm planning to start a distributed crawler in order to avoid common limitations imposed by servers/CDN like rate limit, region filter, and others. My idea is to have a central server and multiple ...
1
vote
1answer
99 views

Why fingerprint a browser if a fingerprint can be replayed?

I'm facing an issue with rampant scraping and abuse on a website that costs me a good chunk of money to maintain. So, I have been looking to implement a few solutions, and apparently these solutions ...
85
votes
10answers
16k views

How and why is my site being abused?

I own a popular website that allows people to enter a phone number and get information back about that phone number, such as the name of the phone carrier. It's a free service, but it costs us money ...
1
vote
1answer
85 views

Does a searchable public database exist of (hostname; ip) mappings?

This question is not about the trivial usage of the forward/reverse DNS. Getting the IP of a hostname is trivial (DNS), and using reverse DNS, also we can get (typically) a single hostname of an IP. ...
-1
votes
1answer
210 views

Spider (web scrawling) vs Force Browsing

I'm scanning websites with OWASP ZAP and noticed that the Spider attack and Force Browse attack return somewhat different results. What are the differences between Web Spiders (Web Crawlers) and ...
0
votes
1answer
131 views

How do attackers hit a website with thousands of similar but distinct IP addresses?

I have a website that is being hit with invalid URL requests by thousands of distinct IP addresses, never the same one used twice. Most of them are in a few ranges of IP addresses and often just go ...
0
votes
1answer
103 views

Are AWS Signed URL's crawled by google?

I have used Amazon pre signed url to share content. https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-presigned-urls.html Is google able to crawl this url? I'm sharing this url with ...
1
vote
0answers
115 views

I run a web crawler on my localhost computer, can my ISP detects that? [closed]

I'm using an Internet plan of 100GB bandwidth monthly from my ISP, and I made a simple web crawler for fun and run it on my personal computer 24/7. The crawler is consuming all of the bandwidth, and I ...
3
votes
1answer
140 views

How to crawl a web site if content is only visible to registered accounts?

I am reading abpit the attack amd defense strategies of web spiders. Assume I have sensitive information on my website, which should be protected from 3rd party web spiders. Use case #1: Me: I set ...