0

I am doing some web scraping for streaming files, .m3u8 mime type. In this particular instance, I can't create a functional request that does not end up with a 403 Forbidden error.

How to reproduce:

  1. Go here in the browser: https://sbplay.org/play/0t689fl5boy8?auto=0&referer=# ( This is just an embedded stream )
  2. Open the console, and type: let f = jwplayer().getPlaylist()[0].file; f;
  3. That should return an url for a .m3u8 file:
let f = jwplayer().getPlaylist()[0].file; f;
"https://delivery166.akamai-cdn-content.com/hls2/01/00385/0t689fl5boy8_,n,h,.urlset/master.m3u8?t=g0tdXzJO_Jz132JWh7Y3FSOiwcshH60pX5dG2TtUHIg&s=1645071428&e=21600&f=1927985&srv=sto009&client=96.55.252.151"
  1. Open the dev-tools for your browser, and paste the url that you just received.
  2. Under Network, Copy as cURL. enter image description here
  3. Try it in bash. This is what I get:
 curl 'https://delivery141.akamai-cdn-content.com/hls2/01/00385/0t689fl5boy8_,n,h,.urlset/master.m3u8?t=_4z7P_f9q6JbBhRdisaalXkQKOvYQi1tnIW3xu8PnJc&s=1644996925&e=21600&f=1927985&srv=sto009&client=96.55.252.151' \
  -H 'Upgrade-Insecure-Requests: 1' \
  -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) QtWebEngine/5.15.2 Chrome/87.0.4280.144 Safari/537.36' \ 
  --compressed   
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx</center>
</body>
</html>

What is going on here? There does not seem to be a cookie involved as far as I can tell, and the user-agent seems to be on point; Why can I only initiate a download from the browser, and not from a cUrl request?

And more to the point; how do I need to modify this cUrl request so it works?

Thanks.

1 Answer 1

1

You likely face the anti-bot protection of the Akamai CDN. It tries to detect uncommon clients in order to protect the site from web scraping, scanning or other automatic access or attacks.

The exact details of how this works are not public, depend on specific setting, differ between CDN and other protection providers and can change over time. Typical approaches are the existence, content and order of HTTP headers, but also SSL fingerprints might be used or analysis of the source IP might be involved.

Helping in bypassing such security measures for your own gain are off-topic here. But one can find several approaches at stackoverflow.com which might or might not work with your specific site - search for akamai cdn bot prtection 403.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .