This tool has been developed to extract endpoints marked as disallow in robots.txt file. It crawls the file directly on the web and has a wayback machine query mode (1 query for each of the previous 5 years).

Possible uses of roboXtractor:

  • Generate a customized wordlist of endpoints for later use in a fuzzing tool (-m 1).
  • Generate a list of URLs to visit (-m 0).

🛠️ Installation

If you want to make modifications locally and compile it, follow the instructions below:

> git clone
> cd roboxtractor
> go build

If you are only interested in using the program:

> go get -u

Note If you are using version 1.16 or higher and you have any errors, run the following command:

> go env -w GO111MODULE="auto"

🗒 Options

The flags that can be used to launch the tool:

Flag Type Description Example
u string URL to extract endpoints marked as disallow in robots.txt file. -u
m uint Extract URLs (0) // Extract endpoints to generate a wordlist (>1 default) -m 1
wb bool Check Wayback Machine. Check 5 years (Slow mode) -wb
v bool Verbose mode. Displays additional information at each step -v
s bool Silen mode doesn't show banner -s

You can ignore the -u flag and pass a file directly as follows:

cat urls.txt | roboxtractor -m 1 -v

Only the results are written to the standard output. The banner and information messages with the -v flag are redirected to the error output,

👾 Usage

The following are some examples of use:

roboxtractor --help
cat urls.txt | roboxtractor -m 0 -v
roboxtractor -u -m 1 -wb
cat urls.txt | roboxtractor -m 1 -s > ./customwordlist.txt
cat urls.txt | roboxtractor -s -v | uniq > ./uniquewordlist.txt
echo | roboxtractor -v
echo | roboxtractor -v -wb

🚀 Examples

Let's take a look at some examples. We have the following file:


Extracting endpoints:


Extracting URLs:


Checking Wayback Machine:


Github had many entries in the file, which were not useful, a cleaning process is done to avoid duplicates or entries with *. Check the following image:


For example:

  • /gist/*/*/* is transformed as gist.
  • /*/tarball is trasformed as tarball.
  • /, /* or similar entries are removed.

🤗 Thanks to

The idea comes from a tweet written by @remonsec that did something similar in a bash script. Check the tweet.