Metasploit also allows us to crawl and scrape the web using auxiliaries. Scraping is useful when we want to grab something from the source code of a website via a defined pattern. It could give us information such as directories mentioned in comments, developer emails, and API calls being made in the background:
- For crawling, we can use the crawl keyword to find the module:
- We will use msfcrawler. Clicking on the module will redirect us to the options page where we define our target, port, and depth. Then, click Run Module:
- A new task will be created and we will see the list of pages found in the task window:
- Similarly, we can use the HTTP Scrape module, auxiliary/scanner/http/scraper, to scrape a web page:
The pattern field is a regex that we define to find whatever element we want. In our case, we want to grab everything inside the script tags on the https://prod.packtpub.com/ website, so our pattern is <script \ type=\"text\/javascript\" \ src=\"(.*)\"><\/script>).
Running the module will create a new task and the auxiliary will extract all of the data listed in the script tags, as shown in the following screenshot:
Next, let's scan for virtual hosts.