Release Summary
Parsero
Parsero is a free script written in Golang that reads the Robots.txt file of a web server and checks the Disallow entries. These entries indicate which directories or files should not be indexed by search engines. Parsero helps to verify the accessibility of these paths by checking their HTTP status codes. Additionally, it can search Bing for indexed content that should not be publicly available according to the Robots.txt file.
Key Features
- HTTP Status Code Checking: Automatically checks the HTTP status code for each Disallow entry to determine if the directories are accessible.
- Bing Search Integration: Searches Bing to find indexed content that should not be publicly available.
- Golang Implementation: Written in Golang for optimized performance, especially with large Robots.txt files.
- Docker Support: Easily build and run Parsero as a Docker container.