Webs recon

Content discovery is a significant part of web application penetration testing or bug bounty hunting. This process involves identifying and mapping out components, endpoints, directories, functionalit

Things we want to look at are:

• Subdomains

• Technology stack

• Directories and endpoints

• Parameters

• Functionality

• APIs

• JavaScript / fontend analysis

• Other open ports / services

Checklist

Web Server

• What is the server running?

  • Operating system: Linux or Windows?

  • Web server: Apache or Nginx? Etc

• Can we identify the version of the Web Server?

• Are there any subdomains?

Common files

• robots.txt

• sitemap.xml

• .htaccess

• security.txt

• manifest.json

• browserconfig.xml

• etc

Frontend checks

• Inspect the page source for frontend scripts & information

• Is there any sensitive information in the frontend?

• Are there links and other things in the frontend that aren't used?

Entry Points

• What endpoints exist

• What HTTP methods are used

• What parameters are used

• Fuzz for hidden endpoints, files, parameters, methods, etc

Map Application Architecture

• Step through the entire application

Resolution

# https://github.com/projectdiscovery/httpx
cat subdomains/subdomains.txt | httpx -follow-redirects -random-agent -status-code -silent -retries 2 -title -web-server -tech-detect -location -no-color -o websites.txt

WAF Checks

# https://github.com/EnableSecurity/wafw00f 
wafw00f -i websites.txt

# IP Wafs/CDN lists
https://github.com/MISP/misp-warninglists

CMS

# https://github.com/Tuhinshubhra/CMSeeK 
tr '\n' ',' < websites.txt > cms_test.txt 
python3 cmseek.py -l cms_test.txt --batch -r

Web screenshot

# https://github.com/sensepost/gowitness
gowitness file -f websites.txt 
gowitness report serve -D gowitness.sqlite3

Fuzzing

# https://github.com/ffuf/ffuf
ffuf -mc all -fc 404 -ac -sf -s -w wordlist.txt -u https://www.domain.com/FUZZ

URLs

URL extraction

 # https://github.com/jaeles-project/gospider
 gospider -S websites.txt --js -t 20 -d 2 --sitemap --robots -w -r > urls.txt

 # https://github.com/lc/gau
 cat websites.txt | gau --subs 
 
 # https://github.com/tomnomnom/waybackurls 
 cat websites.txt | waybackurls 
 
 # https://github.com/gwen001/github-endpoints 
 github-endpoints -q -k -d united.com -t tokens_github.txt 
 
 # https://github.com/Josue87/roboxtractor 
 cat webs.txt | roboxtractor -m 1 -wb

 # https://github.com/projectdiscovery/katana
 katana -u target.com -ps -silent -pss waybackarchive,commoncrawl,alienvault -o urls.txt ##Passive mode
 katana -u target.com -duc -silent -nc -jc -kf all -fx -xhr -ef woff,css,png,svg,jpg,woff2,jpeg,gif,svg -aff -o urls.txt ##Crawling and Spidering

 # https://github.com/xnl-h4ck3r/waymore
 waymore -i target.com -mode U -oU urls.txt

Filtering

# https://github.com/tomnomnom/qsreplace
cat urls.txt | qsreplace -a

# https://github.com/s0md3v/uro 
cat urls.txt | uro

Patterns

# https://github.com/tomnomnom/gf 
# https://github.com/1ndianl33t/Gf-Patterns 
gf sqli urls.txt

JS

# https://github.com/w9w/JSA 
cat urls.txt | python3 jsa.py 

# https://github.com/lc/subjs 
cat js.txt | subjs | httpx 

# https://github.com/GerbenJavado/LinkFinder 
python3 linkfinder.py -d -i https://domain.com/whatever.js -o cli

Wordlists generation

# https://github.com/tomnomnom/unfurl 
cat urls.txt | unfurl -u keys 
cat urls.txt | unfurl -u values