About

A distributed crawler

Requirements

composer require teamtnt/crawler

Each instance needs to have an identifier. This can be added in .env

NODE_NAME="Instance 1"

The domain feeder needs to start with a seed domain. After that, running

php artisan crawler

For scraping a single url

php artisan url:frontier www.example.com/something