Crawler

For collecting information from the web

WebShag

Webshag is a multi-threaded, multi-platform web server auditing tool coded in python. It is used for crawling a URL, port scanning, file fuzzing and audits your website.

SqId

SQID or SQL injection digger is a command line tool written in ruby by Metaeye Security Group that looks for SQL injections and common errors in web sites. It performs a Google search when finding for SQL injections and common errors in web site URLs and crawls a webpage.

Burp Spider

Burp Spider is a tool for mapping web applications. It automates the laborious task of cataloging an application's content and functionality, and lets you:

Work manually via your browser, by passively inspecting traffic passing through Burp Proxy and cataloging everything that this identifies.

Actively crawl the application, by automatically following links, submitting forms, and parsing responses for new content.

Browse a detailed site map of discovered content, in tree and table form.

Retain full control of all spidering actions, with fine-grained scope definition, automatic or user-guided submission of forms, and detailed configuration of the spidering engine.

Send interesting items to other Burp Suite tools with a single click.

Deal with complex applications, with automatic handling of login credentials and session cookies, and detection of custom "not found" responses.

Save all of your work, and resume working later.

Metagoofil

Metagoofil is an information gathering tool designed for extracting metadata of public documents (pdf,doc,xls,ppt,odp,ods) availables in the target/victim websites.

It will generate a html page with the results of the metadata extracted, plus a list of potential usernames very useful for preparing a bruteforce attack on open services like ftp, pop3,web applications, vpn, etc. Also it will extract a list of disclosed PATHs in the metadata, with this information you can guess OS, network names, Shared resources, etc.

Maltego

Maltego is an open source intelligence and forensics application. It will offer you timous mining and gathering of information as well as the representation of this information in a easy to understand format.

Skipfish

Skipfish is an active web application security reconnaissance tool. It prepares an interactive sitemap for the targeted site by carrying out a recursive crawl and dictionary-based probes. The resulting map is then annotated with the output from a number of active (but hopefully non-disruptive) security checks. The final report generated by the tool is meant to serve as a foundation for professional web application security assessments.

A fully automated, active web application security reconnaissance tool. Key features:

•High speed: pure C code, highly optimized HTTP handling, minimal CPU footprint - easily achieving 2000 requests per second with responsive targets.
•Ease of use: heuristics to support a variety of quirky web frameworks and mixed-technology sites, with automatic learning capabilities, on-the-fly wordlist creation, and form autocompletion.
•Cutting-edge security logic: high quality, low false positive, differential security checks, capable of spotting a range of subtle flaws, including blind injection vectors.

Web Data Extractor

Web data extractor is a tool that lets you :

Extract targeted company contact data (email, phone, fax) from web for responsible b2b communication. Extract url, meta tag (title, desc, keyword) for website promotion, search directory creation, web research.

Basically it is a website scraping tool

HTTrack

HTTrack is a free (GPL, libre/free software) and easy-to-use offline browser utility.

It allows you to download a World Wide Web site from the Internet to a local directory, building recursively all directories, getting HTML, images, and other files from the server to your computer. HTTrack arranges the original site's relative link-structure. Simply open a page of the "mirrored" website in your browser, and you can browse the site from link to link, as if you were viewing it online. HTTrack can also update an existing mirrored site, and resume interrupted downloads. HTTrack is fully configurable, and has an integrated help system.

WinHTTrack is the Windows 2000/XP/Vista/Seven release of HTTrack, and WebHTTrack the Linux/Unix/BSD release.

Wikto

Web Server Assessment Tool
Wikto is a tool that checks for flaws in webservers. It provides much the same functionality as Nikto but adds various interesting pieces of functionality, such as a Back-End miner and close Google integration. Wikto is written for the MS .NET environment and registration is required to download the binary and/or source code.

Mini MySqlat0r

Mini Mysqlat0r provides a graphical user interface for enumerating MySQL databases through SQL injection.

Mini Mysqlat0r is basically composed of 3 parts: Crawler, Injection Finder, Exploiter.

Syndicate content