Source code at: EndpointScanner Repository
The dev branch hosts new features that may not be the full update, main branch will only contain stable updates to install.
Note: endpoints and paths in this documentation mean the same thing.
A fast automated website reconnaissance tool for red teaming in cybersecurity that extracts paths, endpoints, files, subdomains, and even external links from modern websites. Automated IDOR and broken access control vulnerability testing through replacing variables with 1 in endpoints/paths. Has a built in rate limit tester that can test on any endpoint with 4 HTTP methods, and can bypass simple WAFs/captchas and SPAs.
Feel free to contact me at sphericalflower@gmail.com to ask for help on how to use the tool or to leave feedback.
robots.txt and sitemap.xml for more paths-so flag is passed-ed, is needed to show the extra details.)n number requests to a certain endpoint with 4 HTTP methods, custom headers, body, variable in request body.All passable flags for the tool:
| Flag | Short Form | Description |
|---|---|---|
target |
NIL |
URL |
--ratelimit |
-r |
Number of requests to send during the rate limit test. Default is 100. |
--ratelimit-type |
-rt |
HTTP Method to use for the rate limit test. Defaults to GET. |
--ratelimit-body |
-rb |
Payload data to send in request to use for POST, PATCH and PUT requests. If the custom payload contains double quotes, please use single quotes instead of double quotes to pass this flag. |
--ratelimit-var |
-rv |
Variable in payload data (e.g. {X}) to use. |
--force |
NIL |
Mandatory flag to pass if doing a rate limit test with over 2500 requests using a non-GET HTTP method. Has no short form flag. |
--testpath |
-t |
Endpoint to test for rate limiting. |
--show-404s |
-s |
Show endpoints tested that returned a 404 or an SPA shell. |
--disable-structure-files |
-ds |
Disable scanning of site structure files (robots, sitemaps, manifests, etc.) |
--show-media |
-m |
Include assets/media like images and fonts and videos in scan results |
--show-prog |
-sp |
Print endpoints to the terminal one by one in real-time as they are found. Warning: Progress will show duplicate paths if endpoints are defined multiple times in the code. Use the flag -nd to remove duplicates from progress. Results will not contain duplicates. |
--output-file |
-o |
Save formatted results directly to a local text file. |
--disable-og |
-do |
Disable code from showing the original endpoint with variables. Keeps output tidier. Will NOT remove original tag from progress if the –show-prog flag is present. |
--tidy |
-ti |
Script will not show where it got extra endpoints from, and will not show if it is a client side route and requires login, or react shell. Will also not show if an endpoint is a potential service. |
--tidy-all |
-ta |
Flags –disable-og and –tidy combined. |
--extra-details |
-ed |
Print out extra info that may not be directly related to the endpoints |
--only-original |
-oo |
Only show the original version of the flag instead of it being replaced with a 1. Will also affect show prog. |
--show-source |
-ss |
Print the source of each endpoint during progress, like printing out which file it found the endpoint from. |
--scan-timeout |
-st |
Stop scan completely after given number of minutes and print/save any results found in that time window. Will leave unsorted endpoints in a section labelled ‘UNSORTED’, and will leave out sensitive endpoints. Will NOT interrupt rate limiting test. |
--sort-output |
-so |
Sort out endpoints. |
--ratelimit-header |
-rh |
Custom headers. Must be seperated by a pipe(|), or newlines. Example use: Cookies: {ExampleCookie: example} | Accept: application/json, text/plain, */*. If the custom header contains double quotes, please use single quotes instead of double quotes to pass this flag. |
--no-duplicate-prog |
-nd |
If –show-progress is passed, duplicate endpoints in progress will not be shown. |
--local |
-l |
Necessary flag if the site being tested on is a local site like a localhost or 127.0.0.1:port. |
--no-detect-captcha |
-ndc |
Flag to disable captcha detection function, in case it returns false positives and did not actually get blocked but thinks it did. |
--external-script-loader |
-esl |
Add external domains used for loading script files into the website itself so that their code files will also be scanned for endpoints. |
--all-esl-protocol |
-aep |
Flag to automatically add https/http to every single external script loader that is not defined at the start. Does nothing if -esl is not passed. |
--extra-header |
-eH |
Add extra headers you want for the website like cookies or authorization etc. Also applies to rate limit test even if -orlt is passed. |
--no-headless |
-nh |
Playwright browser used will not be headless, serves as a debug function. |
--disable-sensitive-endpoint |
-dse |
Flag to disable testing sensitive endpoints, allowing the tool to send less requests. |
--still-show-invalid |
-ssi |
Show endpoints that were flagged as invalid. |
--only-ratelimit-test |
-orlt |
Make the scanner only test for rate limiting. |
--ratelimit-await-time |
-rat |
Adjust rate limiting test await time for small or extremely large request quantities. Default is 50. Time is in seconds. |
--pipeable |
-p |
Make the tool pipeable by automatically passing other arguments. |
--path-sub |
-ps |
Change the default path normalization replacement from 1 to a custom string. |
--only-sub |
-os |
Make the tool only output subdomains. |
--only-external |
-ox |
Make the tool only output external links. |
--only-endpoints |
-oe |
Make the tool only output endpoints. Does not include assets, services/apis. |
--depth |
-de |
How deep the recursive scanning for both js and xml files can go. Defaults to infinite. |
--parse-zip |
-pz |
Allow the tool to expand .zip and .gz files (e.g. sitemap.xml.gz) and scrape from them. |
--no-auto-input |
-nai |
Disable automatic input after a set amount of time. |
--filter-out-more |
-fm |
Filter out more unhelpful results like w3.org or schema.org |
--no-headless-browser |
-nhb |
Script will not use the headless browser at all, and use a request instead to get the html body. |
--auto-input-time |
-ait |
Customise amount of time taken for automatic input if you don’t answer in seconds. Default 90 seconds. |
You can install EndpointScanner via PyPI.
You MUST have python 3.9 or above to use this tool! To install the official endpointscanner Python package:
pip install endpointscanner
pip install endpointscanner
After that, install chromium on playwright (playwright will be installed when you install endpointscanner):
playwright install chromium
py -m playwright install chromium
Example command to run to scan a site (show inaccessible endpoints, show assets, show progress as it finds endpoints, and show files that it got endpoints from):
endpointscanner https://example.com -s -m -sp -ss
Example command to run to test a site (Assuming you are testing 5000 requests and creating 5k accounts)
endpointscanner example.com -r 5000 -t /signup -rt POST -rb '{"username":"ExampleUser{X}", "pass":"ExamplePassword"}' -rv '{X}' --force -rh 'POST /signup HTTP/2
Host: example.com
Cookie: clearedcaptcha=true
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:151.0) Gecko/20100101 Firefox/151.0
Accept: application/json, text/plain, */*
Accept-Language: en-US,en;q=0.9
Prefer: safe
Accept-Encoding: gzip, deflate, br
Content-Type: application/json
Content-Length: 60
Origin: https://example.com
Referer: https://example.com/
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
Priority: u=0
Te: trailers'
Example command to only show the original endpoint without adjusting the variable and output to a file:
endpointscanner example.com -oo -o examplescan.txt
-rat/--ratelimit-await-time flag, as th default is 50s and for a PoC with 50-100 requests 50 seconds is way too long.-p/--pipeable flag to make the output of the tool pipeable into other automated payload testing tools. (E.g. sqlmap)-ps/--path-sub flag to change the automated path normalization from replacing variables to 1 to any custom string. Defaults to 1.asset-manifest.json, web-manifest.json, manifest.json into one loop.-de/--depth flag.xml files) be able to scan .xml, .txt, .rss, .atom files.
.gz and .zip files if the flag -pz/--parse-zip is passed. (Detects based off the file extension)SENSITIVE_ENDPOINT set like actuator endpoints and swagger ui-nai/--no-auto-input flag./* paths in robots.txt with ‘1’ as * means everything, showing you the original path. This will also be affected by the -ps and -oo flag..mjs and .cjs files besides just .js-or flag to the -ed flag, now you have to pass -ed for the extra details, instead of extra details being shown by default (for example, JS Stack detected). Things like found path count will still be shown.-nhb/--no-headless-browser flag to completely disable the headless browser and use a simple GET request instead.-fm flag, --filter-more. Will filter stuff like www.w3.org if passed as results like http://www.w3.org/2000/svg are present in almost all websites.-orlt flag to only perform a rate-limiting test.href and src attributes instead of only looking for code/image files.examplegiganticcdn.com/example.js)mapfiles.pyscrapefiles.pyfiltersort.py-oe/--only-endpoints, -ox/--only-external and -os/--only-sub flag to only print endpoints, external links, and subdomains respectivelyopenid-configurationasset-manifest.jsonweb-manifest.jsonmanifest.jsonsw.jsservice-worker.js-nhb flag from showing browser UI to -nh/-no-headless to be more clear-o cli for people who don’t realise it prints to CLI by default.-d to -ds flagexample.com/path instead of just /path.? and # in websites paths.-p/--pipeable flag is not enabled-oe, -ox or -os were used the results would not be outputted into a file.Version 7.6:
openapi.json for pathsFuture plans (May be added in the future version):
none right now
ai assisted code btw
Note that this tool is strictly meant for authorised testing and security research. Running this script on websites where you are not permitted to do so can result in legal action. The author of this script assumes no responsibility for any misuse or legal consequences from running this script. Ensure you have received permission from the owner of the target website before performing tests or scans on their website.