What would be a safe number of HTTP redirects to follow when scraping a URL until giving up? [closed]

Question

I'm doing a "share box" where users can enter a link before posting and the app automatically scraps the URL to get a thumbnail and some other info. My question is: What would be a safe number of HTTP redirects to follow until giving up? The library I'm using has a default limit of 10, but I just realized some sites are doing more than 10 redirects. Take for example this curl command:

This doesn't reach the final content:

curl -L --max-redirs 10 http://www.nytimes.com/2012/03/10/world/americas/bernardo-pazs-inhotim-is-vast-garden-of-art.html?_r=0

Increasing the --max-redirs to 11 works:

curl -L --max-redirs 11 http://www.nytimes.com/2012/03/10/world/americas/bernardo-pazs-inhotim-is-vast-garden-of-art.html?_r=0

Can I just set a limit of 15 or 20 without affecting in a negative way the securty (and performance) of my app?

Answer 1

Yes, you can set a limit of 20 or 25 without causing any harm.

The only risk is that a redirect loop could cause you to keep fetching content, which might allow a malicious user to mount a Denial-of-Service attack against your server. But if you limit it to 25 redirects, then each user request can trigger at most 25 HTTP requests (not counting any embedded content), which is reasonable and shouldn't pose any significant risk.