Software quality engineers/testers don't rely on crawlers and spiders to test for their bugs -- I don't believe that application security engineers/testers should either...
Instead, SQEs rely heavily on dev-testing frameworks (or test harnesses) such as Selenium RC / Bromine, Watir/WatiN/Watij, Sahi, HtmlUnit, or WebDriver. Some go for the higher-end commercial QA tools such as HP QTP, IBM Rational Functional Tester, TestComplete, and VisualStudio Tester Edition 2010.
SQEs do not typically automate exercising an app for execution flow because:
- Testing requires exploratory discovery, like a tourist would do in a new city they've never been to, in order to scope conclusions and determine outcomes, especially when on a time schedule (which many testers are)
- Concepts such as equivalence classification can save testers hours of work by not having to repeat the same mistakes over and over again. Security testing is a little different in that we have to test everything, but certainly we don't do this today and could utilize this technique, especially when time-boxed
The problem with test automation is that apps change quickly and the test harness must usually be continually modified in order to keep up with these changes. In Agile methodologies such as ICONIX, robustness tests are code generated from domain models and sequence diagrams (usually in UML), but certainly there are plenty of ways to automate the rebuilding of test cases during code churn and new builds, however this more than often not probably requires metaprogramming.