Given domains, try to find subdomains and related domains by querying or crawling from multiple sources. Support different query rate, timeout, concurrency and retries for each sources.

mkdir bin
go build -o bin/ ./...

Support either domains join by ',' from command line or file with domain in each line

1. given from command line

./sdfinder -d google.com,twitter.com -out out.json

2. given by file

./sdfinder -src domain.txt -out out.json

content in domain.txt

google.com
twitter.com

For sonarsearch/reverse, it returns domains with given IP. If -ip flag is given, all the input domains will be resolved to get IPs and query sonarsearch reverse API to find more related domains.

./sdfinder -d google.com,twitter.com -ip -out out.json

1. -q limits to only use some of available sources. Eg., To only query crtsh and abuseipdb

./sdfinder -d google.com -out out.json -q crtsh,abuseipdb

2. Limit from config file

./sdfinder -d google.com -out out.json -cfg config.yaml

config.yaml

enabled:
  - crtsh
  - abuseipdb

when -cfg=<config_path> is given, -q will be skipped

Define config in file for each sources, using default if not given. E.g., with below command and config, crtsh use the custom config and abuseipdb use default config.

./sdfinder -d google.com -out out.json -cfg config.yaml

config.yaml

enabled:          # mandatory
  - crtsh
  - abuseipdb
sources:          # optional
  crtsh:
    qps: 0.1      # mandatory
    timeout: 30s  # mandatory
    worker: 5     # mandatory
    retries:      # optional
      times: 1
      interval: 0.5s

If -cfg=<config_path> is not given, -worker(default: 1) controls the amount of goroutines to handle the queries for each sources. E.g, if -worker=4 -q=crtsh,abuseipdb is given, it will start 8 goroutines in total. (4 for crtsh and 4 for abuseipdb)

when -cfg=<config_path> is given, -worker will be skipped

./sdfinder -d google.com,twitter.com -out out.json -worker 4

The statistic information is print in log such as below

$ ./bin/sdfinder -d netflix.com -out out.json -q sonarsearch/subdomains,abuseipdb
INFO[0000] config file not given, using default config for sources
INFO[0000] flag                                          domains=netflix.com out=out.json queriers="sonarsearch/subdomains,abuseipdb" resolve-ip=false worker=1
INFO[0000] init queriers: [sonarsearch/subdomains abuseipdb]
INFO[0001] [unique] domain: 1, subdomain: 4798, rows: 6253
INFO[0001] abuseipdb: {"domain":1,"success":1,"found":1,"related":4797}
INFO[0001] sonarsearch/subdomains: {"domain":1,"success":1,"found":1,"related":1456}

Unique Key: root_domain + domain + method

{"root_domain":"<domain1>","domain":"<related_domain1>","method":"crawl/abuseipdb","type":"subdomain","extra_info":null}
{"root_domain":"<domain1>","domain":"<related_domain2>","method":"crawl/abuseipdb","type":"subdomain","extra_info":null}
{"root_domain":"<domain2>","domain":"<related_domain1>","method":"cert/crtsh","type":"related-domain","extra_info":null}
{"root_domain":"<domain2>","domain":"<related_domain2>","method":"api/sonarsearch/reverse","type":"related-domain","extra_info":{"ip":"111.222.111.222"}}