Use the web directory search tool

1. Introduction

Not always the links of a web page are visible to the user, finding web links is also one of the most important parts of the pentest. Searching for hidden web links adds information and value to testers. There are a variety of tools to make this easier, such as Dirbuster, Dirb, Gobuster … but each has its own limitations.

DirBuster is written in Java and has only GUI interface.

Dirb is also a popular scan tool but does not support multithreading, scans that do not support multithreading are a lot of inconveniences.

Gobuster is also a very powerful tool written in Go, but the installation is more difficult on win or ubuntu than Dirsearch .

Dirsearch is an open source tool written in Python that works in the style of brute-forcing the structure of directories and files of the web. It can run on Windows, Linux, macOS. Dirsearch uses simple but effective command lines, it supports a lot of options such as multithreading, searching by list extensions, delay between requests, cookie set, user-agent, headers, proxy … That’s why dirsearch become the common tool most hackers and pentester use.

2. Install

First we need to clone dirsearch from GitHub

3. Configuration

There are many dirsearch runs like python, bash, Symbolic Link, alias configuration, you can choose one of the following.

1. Run in Python

As I mentioned above, dirsearch is written in Python, so just run dirsearch.py with Python3.

2. Run with Bash

3. Use alias

Open .bashrc and add alias dirsearch='python3 ~/dirsearch/dirsearch.py'

Done, you just need to run the command dirsearch on the terminal when you stand in any directory.

4. Use Symbolic Link

Can create a link to the / bin directory. It is the same as above you can also run the command dirsearch on the terminal when you stand in any directory.

4. Use dirsearch

To see the details of the options and the effects of each option, just add the -h option:

We can see that the options of dirsearch are very diverse, but we should focus on important options such as -u, -e, -r, -t, -w, -x .

For example, the target is http://testphp.vulnweb.com/

In order to scan, we must pass 2 arguments à -u, -e , where -u is the url we want to scan, -e is the extension we want to search, and we can search for many extensions one by one. at, each extension is separated by a comma.

Example:

As you can see above is the parameters that the program has set to run, the bottom is the result of performing the scan, it is divided into columns clearly visible. The first column is the time, the second column is the status code of the response, the third column is the size, and the last column is the directory or file that performed the scan.

So how to pause when it is running, just press <Ctrl> + C the program will pause and there are 2 options are e and c , where e is exit to exit the program completely and c continue to continue scan on the spot you paused.

More advanced is that you can remove status code that you do not want to avoid excessive information disturbance. For example, you need to remove directories or files with unwanted status code just add the option -x <status code to remove> , also can remove multiple status code at once, each status code is separated by a comma. For example, if we need to remove directories or files with a status code 400 and 403, we need to add options -x 403,400 .

Example:

Compared to the results above, it looks a lot more compact, right? The removal of status code depends on each different case, depending on the needs of the person who needs scanning so that the removal becomes more effective.

This brute-forcing scan depends a lot on the library of words we need to try and call it wordlist, so we don’t want to use dirsearch’s default wordlist, but we use other private libraries to scan. So more effectively, dirsearch supports an option that makes this extremely useful, -w . You can refer to the wordlist of potassium. Some notes you need to make sure that the path to your wordlist is correct, so using absolute paths is a best way to avoid not finding wordlist causing errors that cannot load wordlist. For example -w /home/justx/common.txt .

Example:

Looks at the results seem very little, collected less information and you think this tool is not good, but not like I said this type of scan depends a lot on the library that we use, so choose wordlist It will also help to make the information gathering process more efficient.

For further scanning into directories dirsearch supports a -r option, for example you can scan http://testphp.vulnweb.com/admin/ but you want the program to continue scanning in that directory. then you can use this option instead of manually scanning by changing the url.

Example: