Create a simple tool scan subdomains with Python

Tuesday, 12/05/2020

Tram Ho

Scan subdomains are one of the testing test crawl stages. Doing manual searches of subdomains is sometimes difficult, so in this article I will guide you to write tools to make this easier.

Installing the job requests in python:

pip install requests

1 2	pip install requests

Regarding the implementation of this article, which is brute-forcing, we will try all subdomains in wordlist whenever receiving a response, it is a valid subdomain . Sometimes the scan does not get all the subdomains , this is completely normal because our library does not subdomains all valid subdomains of that target .

Import the necessary libraries:

<span class="token keyword">import</span> requests
<span class="token keyword">import</span> urllib3
<span class="token keyword">import</span> sys
<span class="token keyword">import</span> threading

<span class="token keyword">from</span> queue <span class="token keyword">import</span> Queue

d <span class="token operator">=</span> Queue <span class="token punctuation">(</span> <span class="token punctuation">)</span>

import requests

import urllib3

import sys

import threading

from queue import Queue

d = Queue ( )

The request used to send the request to the target and receive write the domain is valid or not.
urllib3 to format target .
threading makes multithreading run faster.
The queue is more convenient in multithreading.

target processing:

<span class="token keyword">def</span> <span class="token function">parse_url</span> <span class="token punctuation">(</span> url <span class="token punctuation">)</span> <span class="token punctuation">:</span>
    <span class="token keyword">try</span> <span class="token punctuation">:</span>
        host <span class="token operator">=</span> urllib3 <span class="token punctuation">.</span> util <span class="token punctuation">.</span> url <span class="token punctuation">.</span> parse_url <span class="token punctuation">(</span> url <span class="token punctuation">)</span> <span class="token punctuation">.</span> host
    <span class="token keyword">except</span> Exception <span class="token keyword">as</span> e <span class="token punctuation">:</span>
        <span class="token keyword">print</span> <span class="token punctuation">(</span> <span class="token string">"Invalid domain, try again.."</span> <span class="token punctuation">)</span>
        sys <span class="token punctuation">.</span> exit <span class="token punctuation">(</span> <span class="token number">1</span> <span class="token punctuation">)</span>
    <span class="token keyword">return</span> host

def parse_url ( url ) :

try :

host = urllib3 . util . url . parse_url ( url ) . host

except Exception as e :

print ( "Invalid domain, try again.." )

sys . exit ( 1 )

return host

This function tries to process the input to get a hostname if it is invalid the program will automatically exit.

<span class="token keyword">def</span> <span class="token function">parse_wordlist</span> <span class="token punctuation">(</span> wordlist <span class="token punctuation">)</span> <span class="token punctuation">:</span>
    <span class="token keyword">try</span> <span class="token punctuation">:</span>
        wordlists <span class="token operator">=</span> <span class="token builtin">open</span> <span class="token punctuation">(</span> wordlist <span class="token punctuation">)</span> <span class="token punctuation">.</span> read <span class="token punctuation">(</span> <span class="token punctuation">)</span> <span class="token punctuation">.</span> splitlines <span class="token punctuation">(</span> <span class="token punctuation">)</span>
    <span class="token keyword">except</span> Exception <span class="token keyword">as</span> e <span class="token punctuation">:</span>
        <span class="token keyword">print</span> <span class="token punctuation">(</span> e <span class="token punctuation">)</span>
        sys <span class="token punctuation">.</span> exit <span class="token punctuation">(</span> <span class="token number">1</span> <span class="token punctuation">)</span>
    <span class="token keyword">return</span> wordlists

def parse_wordlist ( wordlist ) :

try :

wordlists = open ( wordlist ) . read ( ) . splitlines ( )

except Exception as e :

print ( e )

sys . exit ( 1 )

return wordlists

This function helps us to read all subdomains from wordlist , with splitlines() function in python will help separate all the lines in a string. This helps avoid the case that being unable to load wordlist causes program errors, making sure that you have selected the correct path of the wordlist file. You can refer to some wordlist about DNS on Google, Github …

The following is one of the main functions of the program:

<span class="token keyword">def</span> <span class="token function">scan_subdomain</span> <span class="token punctuation">(</span> target <span class="token punctuation">)</span> <span class="token punctuation">:</span>
    <span class="token keyword">global</span> d
    <span class="token keyword">while</span> <span class="token boolean">True</span> <span class="token punctuation">:</span>
        subdomain <span class="token operator">=</span> d <span class="token punctuation">.</span> get <span class="token punctuation">(</span> <span class="token punctuation">)</span>
        url <span class="token operator">=</span> f <span class="token string">"http://{subdomain}.{target}"</span>
        <span class="token keyword">try</span> <span class="token punctuation">:</span>
            res <span class="token operator">=</span> requests <span class="token punctuation">.</span> get <span class="token punctuation">(</span> url <span class="token punctuation">)</span>
        <span class="token keyword">except</span> requests <span class="token punctuation">.</span> ConnectionError <span class="token punctuation">:</span>
            <span class="token keyword">pass</span>
        <span class="token keyword">else</span> <span class="token punctuation">:</span>
            <span class="token keyword">if</span> res <span class="token punctuation">.</span> status_code <span class="token operator">==</span> <span class="token number">200</span> <span class="token punctuation">:</span>
                <span class="token keyword">print</span> <span class="token punctuation">(</span> <span class="token string">"[+] "</span> <span class="token punctuation">,</span> url <span class="token punctuation">)</span>
        d <span class="token punctuation">.</span> task_done <span class="token punctuation">(</span> <span class="token punctuation">)</span>

def scan_subdomain ( target ) :

global d

while True :

subdomain = d . get ( )

url = f "http://{subdomain}.{target}"

try :

res = requests . get ( url )

except requests . ConnectionError :

pass

else :

if res . status_code == 200 :

print ( "[+] " , url )

d . task_done ( )

For subdomains that have valid or not, we will send requests to that domain, whenever the feedback received, it is valid. Declare a global queue variable to get subdomains stored in the queue, loop until all subdomains are made to request, create a url with the subdomain taken from the queue and target we are targeting, and send requests to. url and get feedback results to process. If you can not connect to that url, skip it and print nothing because the subdomains are invalid, if you get a status code 200, you can connect to that subdomain and you can print it completely. all valid.

With single-threaded scanning programs, it is quite slow, for large word list , single-threaded runs can take a day, to be able to improve this you can refer to multithreading.

<span class="token keyword">def</span> <span class="token function">mutil_scan_subdomains</span> <span class="token punctuation">(</span> target <span class="token punctuation">,</span> number_threads <span class="token punctuation">,</span> subdomains <span class="token punctuation">)</span> <span class="token punctuation">:</span>
    <span class="token keyword">global</span> d
    <span class="token keyword">for</span> subdomain <span class="token keyword">in</span> subdomains <span class="token punctuation">:</span>
        d <span class="token punctuation">.</span> put <span class="token punctuation">(</span> subdomain <span class="token punctuation">)</span>
    <span class="token keyword">for</span> thread <span class="token keyword">in</span> <span class="token builtin">range</span> <span class="token punctuation">(</span> number_threads <span class="token punctuation">)</span> <span class="token punctuation">:</span>
        t <span class="token operator">=</span> threading <span class="token punctuation">.</span> Thread <span class="token punctuation">(</span> target <span class="token operator">=</span> scan_subdomain <span class="token punctuation">,</span> args <span class="token operator">=</span> <span class="token punctuation">(</span> target <span class="token punctuation">,</span> <span class="token punctuation">)</span> <span class="token punctuation">)</span>
        t <span class="token punctuation">.</span> daemon <span class="token operator">=</span> <span class="token boolean">True</span>
        t <span class="token punctuation">.</span> start <span class="token punctuation">(</span> <span class="token punctuation">)</span>

def mutil_scan_subdomains ( target , number_threads , subdomains ) :

global d

for subdomain in subdomains :

d . put ( subdomain )

for thread in range ( number_threads ) :

t = threading . Thread ( target = scan_subdomain , args = ( target , ) )

t . daemon = True

t . start ( )

First add all the subdomain from the previously processed wordlist and put into the queue:

for subdomain in subdomains:
        d.put(subdomain)

for subdomain in subdomains:

d.put(subdomain)

Then initialize and run with the number of threads input:

for thread in range(number_threads):
        t = threading.Thread(target=scan_subdomain, args=(target,))
        t.daemon = True
        t.start()

for thread in range(number_threads):

t = threading.Thread(target=scan_subdomain, args=(target,))

t.daemon = True

t.start()

main

<span class="token keyword">def</span> <span class="token function">main</span> <span class="token punctuation">(</span> <span class="token punctuation">)</span> <span class="token punctuation">:</span>
    target <span class="token operator">=</span> <span class="token builtin">input</span> <span class="token punctuation">(</span> <span class="token string">"Target: "</span> <span class="token punctuation">)</span>
    wordlist <span class="token operator">=</span> <span class="token builtin">input</span> <span class="token punctuation">(</span> <span class="token string">"Wordlist: "</span> <span class="token punctuation">)</span>
    number_threads <span class="token operator">=</span> <span class="token builtin">int</span> <span class="token punctuation">(</span> <span class="token builtin">input</span> <span class="token punctuation">(</span> <span class="token string">"Number threads: "</span> <span class="token punctuation">)</span> <span class="token punctuation">)</span>
    <span class="token keyword">print</span> <span class="token punctuation">(</span> <span class="token string">"====================="</span> <span class="token punctuation">)</span>
    mutil_scan_subdomains <span class="token punctuation">(</span> target <span class="token operator">=</span> target <span class="token punctuation">,</span> number_threads <span class="token operator">=</span> number_threads <span class="token punctuation">,</span> subdomains <span class="token operator">=</span> parse_wordlist <span class="token punctuation">(</span> wordlist <span class="token punctuation">)</span> <span class="token punctuation">)</span>

def main ( ) :

target = input ( "Target: " )

wordlist = input ( "Wordlist: " )

number_threads = int ( input ( "Number threads: " ) )

print ( "=====================" )

mutil_scan_subdomains ( target = target , number_threads = number_threads , subdomains = parse_wordlist ( wordlist ) )

With the main function, just take the input and run.

python scansubdoamin.py
Target: viblo.asia 
Wordlist: subdomains.txt
Number threads: 100

python scansubdoamin.py

Target: viblo.asia

Wordlist: subdomains.txt

Number threads: 100

Here is the result when running:

Full code:

<span class="token keyword">import</span> requests
<span class="token keyword">import</span> urllib3
<span class="token keyword">import</span> sys
<span class="token keyword">import</span> threading

<span class="token keyword">from</span> queue <span class="token keyword">import</span> Queue

d <span class="token operator">=</span> Queue <span class="token punctuation">(</span> <span class="token punctuation">)</span>

<span class="token keyword">def</span> <span class="token function">parse_url</span> <span class="token punctuation">(</span> url <span class="token punctuation">)</span> <span class="token punctuation">:</span>
    <span class="token keyword">try</span> <span class="token punctuation">:</span>
        host <span class="token operator">=</span> urllib3 <span class="token punctuation">.</span> util <span class="token punctuation">.</span> url <span class="token punctuation">.</span> parse_url <span class="token punctuation">(</span> url <span class="token punctuation">)</span> <span class="token punctuation">.</span> host
    <span class="token keyword">except</span> Exception <span class="token keyword">as</span> e <span class="token punctuation">:</span>
        <span class="token keyword">print</span> <span class="token punctuation">(</span> <span class="token string">"Invalid domain, try again.."</span> <span class="token punctuation">)</span>
        sys <span class="token punctuation">.</span> exit <span class="token punctuation">(</span> <span class="token number">1</span> <span class="token punctuation">)</span>
    <span class="token keyword">return</span> host

<span class="token keyword">def</span> <span class="token function">parse_wordlist</span> <span class="token punctuation">(</span> wordlist <span class="token punctuation">)</span> <span class="token punctuation">:</span>
    <span class="token keyword">try</span> <span class="token punctuation">:</span>
        wordlists <span class="token operator">=</span> <span class="token builtin">open</span> <span class="token punctuation">(</span> wordlist <span class="token punctuation">)</span> <span class="token punctuation">.</span> read <span class="token punctuation">(</span> <span class="token punctuation">)</span> <span class="token punctuation">.</span> splitlines <span class="token punctuation">(</span> <span class="token punctuation">)</span>
    <span class="token keyword">except</span> Exception <span class="token keyword">as</span> e <span class="token punctuation">:</span>
        <span class="token keyword">print</span> <span class="token punctuation">(</span> e <span class="token punctuation">)</span>
        sys <span class="token punctuation">.</span> exit <span class="token punctuation">(</span> <span class="token number">1</span> <span class="token punctuation">)</span>
    <span class="token keyword">return</span> wordlists
<span class="token keyword">def</span> <span class="token function">banner</span> <span class="token punctuation">(</span> <span class="token punctuation">)</span> <span class="token punctuation">:</span>
    <span class="token keyword">print</span> <span class="token punctuation">(</span> <span class="token triple-quoted-string string">'''
 ____               ____ ____  ____  
/ ___| _   _ _ __  / ___/ ___||  _  
___ | | | | '_ | |   ___ | |_) |
 ___) | |_| | | | | |___ ___) |  _ &lt; 
|____/ __,_|_| |_|____|____/|_| _ 
=====================================
'''</span> <span class="token punctuation">)</span>

<span class="token keyword">def</span> <span class="token function">scan_subdomain</span> <span class="token punctuation">(</span> target <span class="token punctuation">)</span> <span class="token punctuation">:</span>
    <span class="token keyword">global</span> d
    <span class="token keyword">while</span> <span class="token boolean">True</span> <span class="token punctuation">:</span>
        subdomain <span class="token operator">=</span> d <span class="token punctuation">.</span> get <span class="token punctuation">(</span> <span class="token punctuation">)</span>
        url <span class="token operator">=</span> f <span class="token string">"http://{subdomain}.{target}"</span>
        <span class="token keyword">try</span> <span class="token punctuation">:</span>
            res <span class="token operator">=</span> requests <span class="token punctuation">.</span> get <span class="token punctuation">(</span> url <span class="token punctuation">)</span>
        <span class="token keyword">except</span> requests <span class="token punctuation">.</span> ConnectionError <span class="token punctuation">:</span>
            <span class="token keyword">pass</span>
        <span class="token keyword">else</span> <span class="token punctuation">:</span>
            <span class="token keyword">if</span> res <span class="token punctuation">.</span> status_code <span class="token operator">==</span> <span class="token number">200</span> <span class="token punctuation">:</span>
                <span class="token keyword">print</span> <span class="token punctuation">(</span> <span class="token string">"[+] "</span> <span class="token punctuation">,</span> url <span class="token punctuation">)</span>
        d <span class="token punctuation">.</span> task_done <span class="token punctuation">(</span> <span class="token punctuation">)</span>

<span class="token keyword">def</span> <span class="token function">mutil_scan_subdomains</span> <span class="token punctuation">(</span> target <span class="token punctuation">,</span> number_threads <span class="token punctuation">,</span> subdomains <span class="token punctuation">)</span> <span class="token punctuation">:</span>
    <span class="token keyword">for</span> subdomain <span class="token keyword">in</span> subdomains <span class="token punctuation">:</span>
        d <span class="token punctuation">.</span> put <span class="token punctuation">(</span> subdomain <span class="token punctuation">)</span>
    <span class="token keyword">for</span> thread <span class="token keyword">in</span> <span class="token builtin">range</span> <span class="token punctuation">(</span> number_threads <span class="token punctuation">)</span> <span class="token punctuation">:</span>
        t <span class="token operator">=</span> threading <span class="token punctuation">.</span> Thread <span class="token punctuation">(</span> target <span class="token operator">=</span> scan_subdomain <span class="token punctuation">,</span> args <span class="token operator">=</span> <span class="token punctuation">(</span> target <span class="token punctuation">,</span> <span class="token punctuation">)</span> <span class="token punctuation">)</span>
        t <span class="token punctuation">.</span> daemon <span class="token operator">=</span> <span class="token boolean">True</span>
        t <span class="token punctuation">.</span> start <span class="token punctuation">(</span> <span class="token punctuation">)</span>
<span class="token keyword">def</span> <span class="token function">main</span> <span class="token punctuation">(</span> <span class="token punctuation">)</span> <span class="token punctuation">:</span>
    target <span class="token operator">=</span> <span class="token builtin">input</span> <span class="token punctuation">(</span> <span class="token string">"Target: "</span> <span class="token punctuation">)</span>
    wordlist <span class="token operator">=</span> <span class="token builtin">input</span> <span class="token punctuation">(</span> <span class="token string">"Wordlist: "</span> <span class="token punctuation">)</span>
    number_threads <span class="token operator">=</span> <span class="token builtin">int</span> <span class="token punctuation">(</span> <span class="token builtin">input</span> <span class="token punctuation">(</span> <span class="token string">"Number threads: "</span> <span class="token punctuation">)</span> <span class="token punctuation">)</span>
    <span class="token keyword">print</span> <span class="token punctuation">(</span> <span class="token string">"====================="</span> <span class="token punctuation">)</span>
    mutil_scan_subdomains <span class="token punctuation">(</span> target <span class="token operator">=</span> target <span class="token punctuation">,</span> number_threads <span class="token operator">=</span> number_threads <span class="token punctuation">,</span> subdomains <span class="token operator">=</span> parse_wordlist <span class="token punctuation">(</span> wordlist <span class="token punctuation">)</span> <span class="token punctuation">)</span>
<span class="token keyword">if</span> __name__ <span class="token operator">==</span> <span class="token string">"__main__"</span> <span class="token punctuation">:</span>
    banner <span class="token punctuation">(</span> <span class="token punctuation">)</span>
    <span class="token keyword">print</span> <span class="token punctuation">(</span> <span class="token string">"===================="</span> <span class="token punctuation">)</span>
    main <span class="token punctuation">(</span> <span class="token punctuation">)</span>
    d <span class="token punctuation">.</span> join <span class="token punctuation">(</span> <span class="token punctuation">)</span>

import requests

import urllib3

import sys

import threading

from queue import Queue

d = Queue ( )

try :

except Exception as e :

print ( "Invalid domain, try again.." )

sys . exit ( 1 )

return host

try :

except Exception as e :

print ( e )

sys . exit ( 1 )

return wordlists

def banner ( ) :

print ( '''

____ ____ ____ ____

/ ___| _ _ _ __ / ___/ ___|| _

___ | | | | '_ | | ___ | |_) |

___) | |_| | | | | |___ ___) | _ <

|____/ __,_|_| |_|____|____/|_| _

=====================================

''' )

global d

while True :

subdomain = d . get ( )

url = f "http://{subdomain}.{target}"

try :

res = requests . get ( url )

except requests . ConnectionError :

pass

else :

d . task_done ( )

for subdomain in subdomains :

d . put ( subdomain )

t . daemon = True

t . start ( )

print ( "=====================" )

if __name__ == "__main__" :

banner ( )

print ( "====================" )

main ( )

d . join ( )

Share the news now

Source : Viblo

Create a simple tool scan subdomains with Python

TikTok becomes the second largest social platform in South Africa

The fastest depreciating after 9 months of launch, iPhone 14 Pro Max continues to break the bottom in Vietnam

Beginner's guide to R: Introduction

10 essential SublimeText plugins for JavaScript developers