Skip to content

Commit

Permalink
Addd quite options
Browse files Browse the repository at this point in the history
  • Loading branch information
j3ssie committed Aug 29, 2020
1 parent 8002688 commit 50658bf
Show file tree
Hide file tree
Showing 8 changed files with 882 additions and 644 deletions.
12 changes: 12 additions & 0 deletions .github/FUNDING.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# These are supported funding model platforms

github: # Replace with up to 4 GitHub Sponsors-enabled usernames e.g., [user1, user2]
patreon: j3ssie
open_collective: jaeles-project
ko_fi: # Replace with a single Ko-fi username
tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry
liberapay: # Replace with a single Liberapay username
issuehunt: # Replace with a single IssueHunt username
otechie: # Replace with a single Otechie username
custom: ['https://paypal.me/j3ssiejjj']
40 changes: 32 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,15 @@
# GoSpider

**GoSpider** - Fast web spider written in Go

## Installation

```
go get -u github.com/jaeles-project/gospider
```

## Features

* Fast web crawling
* Brute force and parse sitemap.xml
* Parse robots.txt
Expand All @@ -24,8 +27,9 @@ go get -u github.com/jaeles-project/gospider
[![asciicast](https://asciinema.org/a/301827.svg)](https://asciinema.org/a/301827)

## Usage
```
Fast web spider written in Go - v1.1.0 by @thebl4ckturtle

```shell
Fast web spider written in Go - v1.1.2 by @thebl4ckturtle & @j3ssiejjj

Usage:
gospider [flags]
Expand All @@ -36,9 +40,9 @@ Flags:
-p, --proxy string Proxy (Ex: http://127.0.0.1:8080)
-o, --output string Output folder
-u, --user-agent string User Agent to use
web: random web user-agent
mobi: random mobile user-agent
or you can set your special user-agent (default "web")
web: random web user-agent
mobi: random mobile user-agent
or you can set your special user-agent (default "web")
--cookie string Cookie to use (testA=a; testB=b)
-H, --header stringArray Header to use (Use multiple flag to set multiple header)
--burp string Load headers and cookie from burp raw http request
Expand All @@ -49,20 +53,32 @@ Flags:
-k, --delay int Delay is the duration to wait before creating a new request to the matching domains (second)
-K, --random-delay int RandomDelay is the extra randomized duration to wait added to Delay before creating a new request (second)
-m, --timeout int Request timeout (second) (default 10)
-B, --base Disable all and only use HTML content
--js Enable linkfinder in javascript file (default true)
--sitemap Try to crawl sitemap.xml
--robots Try to crawl robots.txt (default true)
-a, --other-source Find URLs from 3rd party (Archive.org, CommonCrawl.org, VirusTotal.com)
-a, --other-source Find URLs from 3rd party (Archive.org, CommonCrawl.org, VirusTotal.com, AlienVault.com)
-w, --include-subs Include subdomains crawled from 3rd party. Default is main domain
-r, --include-other-source Also include other-source's urls (still crawl and request)
--debug Turn on debug mode
-v, --verbose Turn on verbose
-q, --quite Only show URL
--no-redirect Disable redirect
--version Check version
-h, --help help for gospider
```
## Example commands
### Quite output
```
gospider -q -s "https://google.com/"
```
#### Run with single site
```
gospider -s "https://google.com/" -o output -c 10 -d 1
```
Expand Down Expand Up @@ -98,4 +114,12 @@ gospider -s "https://google.com/" -o output -c 10 -d 1 --other-source --burp bur
**P/s**: gospider blacklisted `.(jpg|jpeg|gif|css|tif|tiff|png|ttf|woff|woff2|ico)` as default
```
gospider -s "https://google.com/" -o output -c 10 -d 1 --blacklist ".(woff|pdf)"
```
```
## License
`Gospider` is made with ♥ by [@j3ssiejjj](https://twitter.com/j3ssiejjj) & [@thebl4ckturtle](https://twitter.com/thebl4ckturtle) and it is released under the MIT license.
## Donation
[![paypal](https://www.paypalobjects.com/en_US/i/btn/btn_donateCC_LG.gif)](https://paypal.me/j3ssiejjj)
Loading

0 comments on commit 50658bf

Please sign in to comment.