Scan subdomains are one of the testing test crawl stages. Doing manual searches of subdomains
is sometimes difficult, so in this article I will guide you to write tools to make this easier.
Installing the job requests in python:
1 2 |
pip install requests |
Regarding the implementation of this article, which is brute-forcing, we will try all subdomains
in wordlist
whenever receiving a response, it is a valid subdomain
. Sometimes the scan does not get all the subdomains
, this is completely normal because our library does not subdomains
all valid subdomains
of that target
.
Import the necessary libraries:
1 2 3 4 5 6 7 8 9 |
<span class="token keyword">import</span> requests <span class="token keyword">import</span> urllib3 <span class="token keyword">import</span> sys <span class="token keyword">import</span> threading <span class="token keyword">from</span> queue <span class="token keyword">import</span> Queue d <span class="token operator">=</span> Queue <span class="token punctuation">(</span> <span class="token punctuation">)</span> |
- The request used to send the request to the
target
and receive write the domain is valid or not. - urllib3 to format
target
. - threading makes multithreading run faster.
- The queue is more convenient in multithreading.
target
processing:
1 2 3 4 5 6 7 8 |
<span class="token keyword">def</span> <span class="token function">parse_url</span> <span class="token punctuation">(</span> url <span class="token punctuation">)</span> <span class="token punctuation">:</span> <span class="token keyword">try</span> <span class="token punctuation">:</span> host <span class="token operator">=</span> urllib3 <span class="token punctuation">.</span> util <span class="token punctuation">.</span> url <span class="token punctuation">.</span> parse_url <span class="token punctuation">(</span> url <span class="token punctuation">)</span> <span class="token punctuation">.</span> host <span class="token keyword">except</span> Exception <span class="token keyword">as</span> e <span class="token punctuation">:</span> <span class="token keyword">print</span> <span class="token punctuation">(</span> <span class="token string">"Invalid domain, try again.."</span> <span class="token punctuation">)</span> sys <span class="token punctuation">.</span> exit <span class="token punctuation">(</span> <span class="token number">1</span> <span class="token punctuation">)</span> <span class="token keyword">return</span> host |
This function tries to process the input to get a hostname
if it is invalid the program will automatically exit.
1 2 3 4 5 6 7 8 |
<span class="token keyword">def</span> <span class="token function">parse_wordlist</span> <span class="token punctuation">(</span> wordlist <span class="token punctuation">)</span> <span class="token punctuation">:</span> <span class="token keyword">try</span> <span class="token punctuation">:</span> wordlists <span class="token operator">=</span> <span class="token builtin">open</span> <span class="token punctuation">(</span> wordlist <span class="token punctuation">)</span> <span class="token punctuation">.</span> read <span class="token punctuation">(</span> <span class="token punctuation">)</span> <span class="token punctuation">.</span> splitlines <span class="token punctuation">(</span> <span class="token punctuation">)</span> <span class="token keyword">except</span> Exception <span class="token keyword">as</span> e <span class="token punctuation">:</span> <span class="token keyword">print</span> <span class="token punctuation">(</span> e <span class="token punctuation">)</span> sys <span class="token punctuation">.</span> exit <span class="token punctuation">(</span> <span class="token number">1</span> <span class="token punctuation">)</span> <span class="token keyword">return</span> wordlists |
This function helps us to read all subdomains
from wordlist
, with splitlines()
function in python will help separate all the lines in a string. This helps avoid the case that being unable to load wordlist
causes program errors, making sure that you have selected the correct path of the wordlist
file. You can refer to some wordlist about DNS on Google, Github …
The following is one of the main functions of the program:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
<span class="token keyword">def</span> <span class="token function">scan_subdomain</span> <span class="token punctuation">(</span> target <span class="token punctuation">)</span> <span class="token punctuation">:</span> <span class="token keyword">global</span> d <span class="token keyword">while</span> <span class="token boolean">True</span> <span class="token punctuation">:</span> subdomain <span class="token operator">=</span> d <span class="token punctuation">.</span> get <span class="token punctuation">(</span> <span class="token punctuation">)</span> url <span class="token operator">=</span> f <span class="token string">"http://{subdomain}.{target}"</span> <span class="token keyword">try</span> <span class="token punctuation">:</span> res <span class="token operator">=</span> requests <span class="token punctuation">.</span> get <span class="token punctuation">(</span> url <span class="token punctuation">)</span> <span class="token keyword">except</span> requests <span class="token punctuation">.</span> ConnectionError <span class="token punctuation">:</span> <span class="token keyword">pass</span> <span class="token keyword">else</span> <span class="token punctuation">:</span> <span class="token keyword">if</span> res <span class="token punctuation">.</span> status_code <span class="token operator">==</span> <span class="token number">200</span> <span class="token punctuation">:</span> <span class="token keyword">print</span> <span class="token punctuation">(</span> <span class="token string">"[+] "</span> <span class="token punctuation">,</span> url <span class="token punctuation">)</span> d <span class="token punctuation">.</span> task_done <span class="token punctuation">(</span> <span class="token punctuation">)</span> |
For subdomains
that have valid or not, we will send requests
to that domain, whenever the feedback received, it is valid. Declare a global
queue variable to get subdomains
stored in the queue, loop until all subdomains
are made to request, create a url
with the subdomain
taken from the queue and target we are targeting, and send requests to. url
and get feedback results to process. If you can not connect to that url, skip it and print nothing because the subdomains
are invalid, if you get a status code
200, you can connect to that subdomain
and you can print it completely. all valid.
With single-threaded scanning programs, it is quite slow, for large word list
, single-threaded runs can take a day, to be able to improve this you can refer to multithreading.
1 2 3 4 5 6 7 8 9 |
<span class="token keyword">def</span> <span class="token function">mutil_scan_subdomains</span> <span class="token punctuation">(</span> target <span class="token punctuation">,</span> number_threads <span class="token punctuation">,</span> subdomains <span class="token punctuation">)</span> <span class="token punctuation">:</span> <span class="token keyword">global</span> d <span class="token keyword">for</span> subdomain <span class="token keyword">in</span> subdomains <span class="token punctuation">:</span> d <span class="token punctuation">.</span> put <span class="token punctuation">(</span> subdomain <span class="token punctuation">)</span> <span class="token keyword">for</span> thread <span class="token keyword">in</span> <span class="token builtin">range</span> <span class="token punctuation">(</span> number_threads <span class="token punctuation">)</span> <span class="token punctuation">:</span> t <span class="token operator">=</span> threading <span class="token punctuation">.</span> Thread <span class="token punctuation">(</span> target <span class="token operator">=</span> scan_subdomain <span class="token punctuation">,</span> args <span class="token operator">=</span> <span class="token punctuation">(</span> target <span class="token punctuation">,</span> <span class="token punctuation">)</span> <span class="token punctuation">)</span> t <span class="token punctuation">.</span> daemon <span class="token operator">=</span> <span class="token boolean">True</span> t <span class="token punctuation">.</span> start <span class="token punctuation">(</span> <span class="token punctuation">)</span> |
First add all the subdomain
from the previously processed wordlist and put
into the queue:
1 2 3 |
for subdomain in subdomains: d.put(subdomain) |
Then initialize and run with the number of threads input:
1 2 3 4 5 |
for thread in range(number_threads): t = threading.Thread(target=scan_subdomain, args=(target,)) t.daemon = True t.start() |
main
1 2 3 4 5 6 7 |
<span class="token keyword">def</span> <span class="token function">main</span> <span class="token punctuation">(</span> <span class="token punctuation">)</span> <span class="token punctuation">:</span> target <span class="token operator">=</span> <span class="token builtin">input</span> <span class="token punctuation">(</span> <span class="token string">"Target: "</span> <span class="token punctuation">)</span> wordlist <span class="token operator">=</span> <span class="token builtin">input</span> <span class="token punctuation">(</span> <span class="token string">"Wordlist: "</span> <span class="token punctuation">)</span> number_threads <span class="token operator">=</span> <span class="token builtin">int</span> <span class="token punctuation">(</span> <span class="token builtin">input</span> <span class="token punctuation">(</span> <span class="token string">"Number threads: "</span> <span class="token punctuation">)</span> <span class="token punctuation">)</span> <span class="token keyword">print</span> <span class="token punctuation">(</span> <span class="token string">"====================="</span> <span class="token punctuation">)</span> mutil_scan_subdomains <span class="token punctuation">(</span> target <span class="token operator">=</span> target <span class="token punctuation">,</span> number_threads <span class="token operator">=</span> number_threads <span class="token punctuation">,</span> subdomains <span class="token operator">=</span> parse_wordlist <span class="token punctuation">(</span> wordlist <span class="token punctuation">)</span> <span class="token punctuation">)</span> |
With the main function, just take the input and run.
1 2 3 4 5 |
python scansubdoamin.py Target: viblo.asia Wordlist: subdomains.txt Number threads: 100 |
Here is the result when running:
Full code:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 |
<span class="token keyword">import</span> requests <span class="token keyword">import</span> urllib3 <span class="token keyword">import</span> sys <span class="token keyword">import</span> threading <span class="token keyword">from</span> queue <span class="token keyword">import</span> Queue d <span class="token operator">=</span> Queue <span class="token punctuation">(</span> <span class="token punctuation">)</span> <span class="token keyword">def</span> <span class="token function">parse_url</span> <span class="token punctuation">(</span> url <span class="token punctuation">)</span> <span class="token punctuation">:</span> <span class="token keyword">try</span> <span class="token punctuation">:</span> host <span class="token operator">=</span> urllib3 <span class="token punctuation">.</span> util <span class="token punctuation">.</span> url <span class="token punctuation">.</span> parse_url <span class="token punctuation">(</span> url <span class="token punctuation">)</span> <span class="token punctuation">.</span> host <span class="token keyword">except</span> Exception <span class="token keyword">as</span> e <span class="token punctuation">:</span> <span class="token keyword">print</span> <span class="token punctuation">(</span> <span class="token string">"Invalid domain, try again.."</span> <span class="token punctuation">)</span> sys <span class="token punctuation">.</span> exit <span class="token punctuation">(</span> <span class="token number">1</span> <span class="token punctuation">)</span> <span class="token keyword">return</span> host <span class="token keyword">def</span> <span class="token function">parse_wordlist</span> <span class="token punctuation">(</span> wordlist <span class="token punctuation">)</span> <span class="token punctuation">:</span> <span class="token keyword">try</span> <span class="token punctuation">:</span> wordlists <span class="token operator">=</span> <span class="token builtin">open</span> <span class="token punctuation">(</span> wordlist <span class="token punctuation">)</span> <span class="token punctuation">.</span> read <span class="token punctuation">(</span> <span class="token punctuation">)</span> <span class="token punctuation">.</span> splitlines <span class="token punctuation">(</span> <span class="token punctuation">)</span> <span class="token keyword">except</span> Exception <span class="token keyword">as</span> e <span class="token punctuation">:</span> <span class="token keyword">print</span> <span class="token punctuation">(</span> e <span class="token punctuation">)</span> sys <span class="token punctuation">.</span> exit <span class="token punctuation">(</span> <span class="token number">1</span> <span class="token punctuation">)</span> <span class="token keyword">return</span> wordlists <span class="token keyword">def</span> <span class="token function">banner</span> <span class="token punctuation">(</span> <span class="token punctuation">)</span> <span class="token punctuation">:</span> <span class="token keyword">print</span> <span class="token punctuation">(</span> <span class="token triple-quoted-string string">''' ____ ____ ____ ____ / ___| _ _ _ __ / ___/ ___|| _ ___ | | | | '_ | | ___ | |_) | ___) | |_| | | | | |___ ___) | _ < |____/ __,_|_| |_|____|____/|_| _ ===================================== '''</span> <span class="token punctuation">)</span> <span class="token keyword">def</span> <span class="token function">scan_subdomain</span> <span class="token punctuation">(</span> target <span class="token punctuation">)</span> <span class="token punctuation">:</span> <span class="token keyword">global</span> d <span class="token keyword">while</span> <span class="token boolean">True</span> <span class="token punctuation">:</span> subdomain <span class="token operator">=</span> d <span class="token punctuation">.</span> get <span class="token punctuation">(</span> <span class="token punctuation">)</span> url <span class="token operator">=</span> f <span class="token string">"http://{subdomain}.{target}"</span> <span class="token keyword">try</span> <span class="token punctuation">:</span> res <span class="token operator">=</span> requests <span class="token punctuation">.</span> get <span class="token punctuation">(</span> url <span class="token punctuation">)</span> <span class="token keyword">except</span> requests <span class="token punctuation">.</span> ConnectionError <span class="token punctuation">:</span> <span class="token keyword">pass</span> <span class="token keyword">else</span> <span class="token punctuation">:</span> <span class="token keyword">if</span> res <span class="token punctuation">.</span> status_code <span class="token operator">==</span> <span class="token number">200</span> <span class="token punctuation">:</span> <span class="token keyword">print</span> <span class="token punctuation">(</span> <span class="token string">"[+] "</span> <span class="token punctuation">,</span> url <span class="token punctuation">)</span> d <span class="token punctuation">.</span> task_done <span class="token punctuation">(</span> <span class="token punctuation">)</span> <span class="token keyword">def</span> <span class="token function">mutil_scan_subdomains</span> <span class="token punctuation">(</span> target <span class="token punctuation">,</span> number_threads <span class="token punctuation">,</span> subdomains <span class="token punctuation">)</span> <span class="token punctuation">:</span> <span class="token keyword">for</span> subdomain <span class="token keyword">in</span> subdomains <span class="token punctuation">:</span> d <span class="token punctuation">.</span> put <span class="token punctuation">(</span> subdomain <span class="token punctuation">)</span> <span class="token keyword">for</span> thread <span class="token keyword">in</span> <span class="token builtin">range</span> <span class="token punctuation">(</span> number_threads <span class="token punctuation">)</span> <span class="token punctuation">:</span> t <span class="token operator">=</span> threading <span class="token punctuation">.</span> Thread <span class="token punctuation">(</span> target <span class="token operator">=</span> scan_subdomain <span class="token punctuation">,</span> args <span class="token operator">=</span> <span class="token punctuation">(</span> target <span class="token punctuation">,</span> <span class="token punctuation">)</span> <span class="token punctuation">)</span> t <span class="token punctuation">.</span> daemon <span class="token operator">=</span> <span class="token boolean">True</span> t <span class="token punctuation">.</span> start <span class="token punctuation">(</span> <span class="token punctuation">)</span> <span class="token keyword">def</span> <span class="token function">main</span> <span class="token punctuation">(</span> <span class="token punctuation">)</span> <span class="token punctuation">:</span> target <span class="token operator">=</span> <span class="token builtin">input</span> <span class="token punctuation">(</span> <span class="token string">"Target: "</span> <span class="token punctuation">)</span> wordlist <span class="token operator">=</span> <span class="token builtin">input</span> <span class="token punctuation">(</span> <span class="token string">"Wordlist: "</span> <span class="token punctuation">)</span> number_threads <span class="token operator">=</span> <span class="token builtin">int</span> <span class="token punctuation">(</span> <span class="token builtin">input</span> <span class="token punctuation">(</span> <span class="token string">"Number threads: "</span> <span class="token punctuation">)</span> <span class="token punctuation">)</span> <span class="token keyword">print</span> <span class="token punctuation">(</span> <span class="token string">"====================="</span> <span class="token punctuation">)</span> mutil_scan_subdomains <span class="token punctuation">(</span> target <span class="token operator">=</span> target <span class="token punctuation">,</span> number_threads <span class="token operator">=</span> number_threads <span class="token punctuation">,</span> subdomains <span class="token operator">=</span> parse_wordlist <span class="token punctuation">(</span> wordlist <span class="token punctuation">)</span> <span class="token punctuation">)</span> <span class="token keyword">if</span> __name__ <span class="token operator">==</span> <span class="token string">"__main__"</span> <span class="token punctuation">:</span> banner <span class="token punctuation">(</span> <span class="token punctuation">)</span> <span class="token keyword">print</span> <span class="token punctuation">(</span> <span class="token string">"===================="</span> <span class="token punctuation">)</span> main <span class="token punctuation">(</span> <span class="token punctuation">)</span> d <span class="token punctuation">.</span> join <span class="token punctuation">(</span> <span class="token punctuation">)</span> |