Collecting Juicy URL & Scraping URL Parameter from Passive scan. Default Source Using Web Archive, CommonCrawl, UrlScan
Regex using DFA Engine (awk,sed)
Support and Collecting URL with multi Parameter to Fuzzing-
Removing Duplicate Parameter
There will be 3 files in output:
Passive_Collecting_URLParamter_Full.txt : This File is original collecting URL Parameter without Parsing ( Original URL & Parameter Value )
Passive_Collecting_URLParamter_Uniq.txt : This File is original collecting URL Parameter with Unique URL for Fuzzing
Passive_Collecting_JuicyURL.txt: This is a huge list of urls associated with the domain. Here we can get and extract relative paths, endpoint, url and interest strings.
cat Passive_Collecting_JuicyURL.txt | grep --colour=always -e "\.js"
Like filtering the javascript for found a sensitive data from JS Files
cat Passive_Collecting_JuicyURL.txt | grep --colour=always -e "\.pdf"
NOTE : As it fetches the parameters from WebArchive, CommonCrawl, URLscan data , so chances of false positives are high.