When testing a web app, I often start with spidering the app in order to get an idea of the structure of the app.

What bothers me with that approach is that thal all spidering tools that I tried are basically dumb - that means they blindly follow all links to the predefined depth and display a list of visited pages.

The problem is that if I for example want to test blog app which has thousands of blog posts in it, I'm not really interested in testing each page for each blog post. It would be enough to test a single blog post page because all blog posts execute the same code.

Are you aware of any spidering tool that is able to predict if a link will lead to the discovery of "more code" or just "more text/content" (for example if a link will lead to "just another" blog post or to a "more interesting" page)?

closed as off-topic by Xander, Ohnana, AviD Jun 2 '16 at 21:46

This question appears to be off-topic. The users who voted to close gave these specific reasons:

If this question can be reworded to fit the rules in the help center, please edit the question.

  • build your own with Scrapy :) – yzT Jun 2 '16 at 13:02

Browse other questions tagged or ask your own question.