Scraping entire subdomain lists with Sublist3r-Scrap

As pentesters, we are usually presented with one of two scenarios whenever we are about to perform an assessment:

  • Small defined scopes.
  • Broad scopes with few limitations.

Many companies prefer the latter rather than the former, as it presents a better picture of what they are exposing to potential malicious attackers. This is also better known as red-teaming, which has a rather broad definition, but is usually understood as replicating what a real attacker would do.

So, following this logic, the first thing an attacker would do is to enumerate the subdomains of a given domain which belongs to the target. There are several tools to aid in this, however we usually use Sublist3r, as it’s easy to deploy everywhere. It’s a small but powerful tool.

Sublist3r in action

However this presents us with new challenges, one of which is to identify potentially interesting targets. This is usually done by manually inspecting each one of the subdomains acquired, which can be quite time-consuming.

So, we decided to automate the process using selenium. A framework which automates the process of visiting a website, and is able to save screenshots of them.

Enter Sublist3r-Scrap:

Sublist3r-Scrap

This tool accesses each one of the subdomains, on the ports selected (80 and 443 by default), and takes a screenshot of them, so we can better identify potential targets.

It’s easy to install and to use, the only caveat being that the geckodriver downloaded must match the Firefox version you have. However if your Firefox is updated, that shouldn’t be a problem.

Scraper options.

By default it will create a folder called “screens” with the screenshots of the web pages that were accessible.

Resulting screenshots

Hopefully this tool will save many hours to red-teamers. Of course, the code is open source and free for all, accessible in our github:

Any issues or feature requests are appreciated.

2 thoughts on “Scraping entire subdomain lists with Sublist3r-Scrap

  1. Great tool, nice work i also created a similar tool for our ChicagoDevSecops meetup about OSINT that included sublist3r, eyewitness, additionally it ran S3Enum to enumerate open S3 buckets and screenshot them in a nice presentable report. Schedule a cron job and you can see your attack surface in a nice way, all free.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s