hedii / php-crawler Goto Github PK

A php crawler that finds emails on the internets

License: MIT License

PHP 76.54% Vue 12.39% Shell 0.35% Blade 10.73%

php-crawler crawler php laravel vuejs vue webscraping webscraper webcrawler

php-crawler's Introduction

php-crawler

This repository has not been updated since a long time and I am not using it anymore. Please feel free to fork it and do whatever you want with it.

A crawler application with a php backend using Laravel, and a js frontend using vuejs, that finds email addresses on the internets.

Given an entry point url, the crawler will search for emails in all the urls available from this entry point domain name. The emails are downloadable as a text file.

Multiple users can start searching for emails without being able see the other users' searches (searches are related to a single user).

Screenshots

Server requirements

PHP >= 7.2.0
OpenSSL PHP Extension
PDO PHP Extension
Mbstring PHP Extension
Tokenizer PHP Extension
XML PHP Extension

Installation

Create a mysql database (default name: crawler)
Install the project with composer:

composer create-project hedii/php-crawler crawler
cd crawler

Open the .env file, check the database credentials, and modify it if needed:

DB_CONNECTION=mysql
DB_HOST=127.0.0.1
DB_PORT=3306
DB_DATABASE=crawler
DB_USERNAME=root
DB_PASSWORD=your_password_here

In the .env file, set the application url:

APP_URL=http://crawler.localhost

Build the crawler application

php artisan crawler:build

Point your web server's document / web root to be the public directory: /some/path/crawler/public. The index.php in this directory serves as the front controller for all HTTP requests entering your application. See Laravel documentation. I highly recommend using Laravel Valet if you are using a Mac. Otherwise, check Laravel Homestead.
Done

Usage

Navigate to your php-crawler website
Register a new account
Create a new search
Create more searches
Download the found emails

Testing

composer test

Contributing

All contributions are welcome :)

Please write some tests if you are adding or modifying features.

License

php-crawler is open-sourced software licensed under the MIT license.

php-crawler's People

Contributors

Stargazers

Watchers

php-crawler's Issues

Crawler is empty

I putted the script on the server but I got a almost white page

PHP Version => 5.6.31

Did not get any error's
(Login and register works)

Running on vestacp 0.9.8-17

Can't get it to work properly

I do everything accordingly:

BUT:
$ php artisan crawler:build

PHP Warning: require(/var/www/html/crawler/vendor/autoload.php): Failed to open stream: No such file or directory in /var/www/html/crawler/artisan on line 18
PHP Fatal error: Uncaught Error: Failed opening required '/var/www/html/crawler/vendor/autoload.php' (include_path='.:/usr/share/php') in /var/www/html/crawler/artisan:18
Stack trace:
#0 {main}
thrown in /var/www/html/crawler/artisan on line 18

Empty list of emails

Hi!
I have problem with crawling. When I put url and start crawler I always get 0 emails and 0 urls. Dont see any errors on laravel logs.

Crawler does not work

I have added several of my websites to be crawled but the crawler does not seem able to start. Iam running PHP 7.0.21.

Error during installation

I'm still receiving a nasty error on last step when I'm trying to build:

[root@unit237 php-crawler2]# php artisan crawler:build
PHP Warning:  require(/var/www/html/php-crawler2/bootstrap/../vendor/autoload.php): failed to open stream: No such file or directory in /var/www/html/php-crawler2/bootstrap/autoload.php on line 17
PHP Fatal error:  require(): Failed opening required '/var/www/html/php-crawler2/bootstrap/../vendor/autoload.php' (include_path='.:/usr/share/pear:/usr/share/php') in /var/www/html/php-crawler2/bootstrap/autoload.php on line 17

running for ever

when i strat a search it keep display "Running" and when i clic on Vie w search there is no URL there Is it normal ?

Error during installation

[root@unit237 html]# ls
php-crawler
[root@unit237 html]# composer create-project hedii/php-crawler php-crawler

[InvalidArgumentException]
Could not find package hedii/php-crawler with stability stable.

create-project [-s|--stability STABILITY] [--prefer-source] [--prefer-dist] [--repository REPOSITORY] [--repository-url REPOSITORY-URL] [--dev] [--no-dev] [--no-plugins] [--no-custom-installers] [--no-scripts] [--no-progress] [--keep-vcs] [--no-install] [--ignore-platform-reqs] [--] [] [] []

What am I doing wrong?

not found anything

anything is blank page ?

Problem crawl

Sorry but when i start any search, this not crawl anything. The search is loading ... and not look nothing.

Emails found: 0 NOT WORKING

The Urls Statistics show: NaN%
And no emails are found.

Do you know why this is? please advise thank you

CLI any way to control this with CLI?

Own logic

How can add own parse logic into crawler?
For example if I want to parse names also.

btw great tool!

'php-crawler/public/searches' not found

I installed the package and the step 'php artisan crawler:build' seems to be okay:

But I could not find anything in the browser:

My php version is 7.0.10 and the required extensions are installed: http://yz.bxaf.com:8001/phpinfo.php

no option to search or anything else

Hello,
installed everthing, registred and logged in.
But in the main screen i see only "My searches"
and nothing else, no option for new search... no buttons, just logout
what should i do?
sincerely,
Tom

search "running", but nothing happens, cant figure it out

Hello,
I installed everything properly, whwn i add a new search it says "runnig", but its is all the time 0% crawled and nothing really happens.
Went over the problems here, some people had them, tried to follow these advices, but nothing helped.
Here is my Log (ALL DB related problems has been fixed!

laravel.txt

Not working

Even after 20 days of adding starting points still it is running but it didnt find atleast a single url or mail . please help me

Update vue

Hi!
Thank you so much for sharing your project!

I would love to extend it even further (I'm not interested in crawling for emails but for other things).
Therefor I would love to use the latest version of laravel 5.5 (I already managed to update it successfully! 🎉).
To make it work with the latest version of vue a lot of things need to be changed, so I tried to get it running with the given versions you provide in your package.json, so I tried to update all npm packages but then got stuck with this error:

Vue packages version mismatch:

- [email protected]
- [email protected]

This may cause things to work incorrectly. Make sure to use the same version for both.
If you are using vue-loader@>=10.0, simply update vue-template-compiler.
If you are using vue-loader@<10.0 or vueify, re-installing vue-loader/vueify should bump vue-template-compiler to the latest.

my package.json (adjustments from the latest laravel version to use webpack instead of gulp)

{
  "private": true,
  "scripts": {
    "dev": "npm run development",
    "development": "cross-env NODE_ENV=development node_modules/webpack/bin/webpack.js --progress --hide-modules --config=node_modules/laravel-mix/setup/webpack.config.js",
    "watch": "cross-env NODE_ENV=development node_modules/webpack/bin/webpack.js --watch --progress --hide-modules --config=node_modules/laravel-mix/setup/webpack.config.js",
    "watch-poll": "npm run watch -- --watch-poll",
    "hot": "cross-env NODE_ENV=development node_modules/webpack-dev-server/bin/webpack-dev-server.js --inline --hot --config=node_modules/laravel-mix/setup/webpack.config.js",
    "prod": "npm run production",
    "production": "cross-env NODE_ENV=production node_modules/webpack/bin/webpack.js --no-progress --hide-modules --config=node_modules/laravel-mix/setup/webpack.config.js"
  },
  "devDependencies": {
    "bootstrap-sass": "^3.3.7",
    "cross-env": "^5.1",
    "laravel-mix": "^1.0",
    "moment": "^2.12.0",
    "vue": "^1.0.17",
    "vue-resource": "^0.7.0"
  }
}

Would you be able to help me?

API routes problem

Hi!
I use your crawler for quite some time already and thought I already had a working version but now coming back to the project again with the latest release (2.1.0). I have the following problem.

I can register, sign in and click on the new search button but entering a url in the form and submitting it will give me the following error:

Request URL: http://crawler.test/api/v1/users//searches
Request Method: POST
Status Code: 404 Not Found

I have no idea where the double slashes are coming from because all other routes are correct.

The other thing i noticed is that on every request I also see in the browser debug panel the following error:

Request URL: http://crawler.test/api/v1/users/me
Request Method: GET
Status Code: 401 Unauthorized

Using homestead with php 7.2.

Do you have a glue where the problem could be? Of course I followed all the installation instructions.

The start time is shown with "2 hours ago"

The time zone is UTC, but with summer time = DST, what can I do? I tried already with htaccess with php_value date.timezone 'Europe/Berlin', but this don't work.
Thanks, Eric

Nothing Works

After NPM Install and Artisan Build, the page get 404 Not Found.
You point to http://localhost/php-crawler/public/ and then he goes to
http://localhost/php-crawler/public/searches
Returns Not Found

The requested URL /php-crawler/public/searches was not found on this server.
Apache/2.4.10 (Ubuntu) Server at localhost Port 80

NPM Log:

npm WARN engine [email protected]: wanted: {"node":">=0.12"} (current: {"node":"0.10.25","npm":"1.4.21"})
npm WARN deprecated [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
npm WARN engine [email protected]: wanted: {"node":">=0.12"} (current: {"node":"0.10.25","npm":"1.4.21"})
npm WARN deprecated [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
npm WARN optional dep failed, continuing [email protected]
npm WARN engine [email protected]: wanted: {"node":">=4"} (current: {"node":"0.10.25","npm":"1.4.21"})
npm WARN deprecated [email protected]: No longer sponsored by CT - please visit http://niftylettuce.com for more info
npm WARN deprecated [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
npm WARN engine [email protected]: wanted: {"node":">=0.11.0"} (current: {"node":"0.10.25","npm":"1.4.21"})
npm WARN engine [email protected]: wanted: {"node":">=0.12"} (current: {"node":"0.10.25","npm":"1.4.21"})
npm WARN optional dep failed, continuing [email protected]
npm WARN engine [email protected]: wanted: {"node":">=4"} (current: {"node":"0.10.25","npm":"1.4.21"})
npm WARN engine [email protected]: wanted: {"node":">=0.10.40"} (current: {"node":"0.10.25","npm":"1.4.21"})
npm WARN engine [email protected]: wanted: {"node":">=0.11.0"} (current: {"node":"0.10.25","npm":"1.4.21"})
npm WARN deprecated [email protected]: graceful-fs v3.0.0 and before will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0 as soon as possible. Use 'npm ls graceful-fs' to find it in the tree.
npm WARN optional dep failed, continuing [email protected]
npm WARN deprecated [email protected]: ReDoS vulnerability parsing Set-Cookie https://nodesecurity.io/advisories/130
npm WARN deprecated [email protected]: use uuid module instead
npm WARN engine [email protected]: wanted: {"node":">=0.10.32"} (current: {"node":"0.10.25","npm":"1.4.21"})
npm WARN engine [email protected]: wanted: {"node":">=0.10.40"} (current: {"node":"0.10.25","npm":"1.4.21"})
npm WARN engine [email protected]: wanted: {"node":">=0.10.40"} (current: {"node":"0.10.25","npm":"1.4.21"})
npm WARN engine [email protected]: wanted: {"node":">=0.10.40"} (current: {"node":"0.10.25","npm":"1.4.21"})
npm WARN engine [email protected]: wanted: {"node":">=0.12"} (current: {"node":"0.10.25","npm":"1.4.21"})
npm WARN engine [email protected]: wanted: {"node":">=0.12"} (current: {"node":"0.10.25","npm":"1.4.21"})
npm WARN engine [email protected]: wanted: {"node":">= 4"} (current: {"node":"0.10.25","npm":"1.4.21"})
npm WARN engine [email protected]: wanted: {"node":">=0.10.32"} (current: {"node":"0.10.25","npm":"1.4.21"})
npm WARN engine [email protected]: wanted: {"node":">= 0.12"} (current: {"node":"0.10.25","npm":"1.4.21"})
npm WARN engine [email protected]: wanted: {"node":">=0.10.40"} (current: {"node":"0.10.25","npm":"1.4.21"})
npm WARN engine [email protected]: wanted: {"node":">=0.10.40"} (current: {"node":"0.10.25","npm":"1.4.21"})
npm WARN engine [email protected]: wanted: {"node":">=0.10.40"} (current: {"node":"0.10.25","npm":"1.4.21"})
\

[email protected] install /var/www/html/php-crawler/node_modules/laravel-elixir/node_modules/gulp-sass/node_modules/node-sass
node scripts/install.js

sh: 1: node: not found
npm WARN This failure might be due to the use of legacy binary "node"
npm WARN For further explanations, please read
/usr/share/doc/nodejs/README.Debian

npm ERR! Error: ENOENT, utime '/var/www/html/php-crawler/node_modules/laravel-elixir/node_modules/babel-preset-es2015/node_modules/babel-plugin-check-es2015-constants/node_modules/babel-runtime/node_modules/core-js/modules/es6.object.create.js'
npm ERR! If you need help, you may report this entire log,
npm ERR! including the npm and node versions, at:
npm ERR! http://github.com/npm/npm/issues

npm ERR! System Linux 3.19.0-15-generic
npm ERR! command "/usr/bin/nodejs" "/usr/bin/npm" "install"
npm ERR! cwd /var/www/html/php-crawler
npm ERR! node -v v0.10.25
npm ERR! npm -v 1.4.21
npm ERR! path /var/www/html/php-crawler/node_modules/laravel-elixir/node_modules/babel-preset-es2015/node_modules/babel-plugin-check-es2015-constants/node_modules/babel-runtime/node_modules/core-js/modules/es6.object.create.js
npm ERR! fstream_path /var/www/html/php-crawler/node_modules/laravel-elixir/node_modules/babel-preset-es2015/node_modules/babel-plugin-check-es2015-constants/node_modules/babel-runtime/node_modules/core-js/modules/es6.object.create.js
npm ERR! fstream_type File
npm ERR! fstream_class FileWriter
npm ERR! fstream_finish_call utimes
npm ERR! code ENOENT
npm ERR! errno 34
npm ERR! fstream_stack /usr/lib/nodejs/fstream/lib/writer.js:305:19
npm ERR! fstream_stack Object.oncomplete (evalmachine.:107:15)
npm ERR! error rolling back Error: ENOTEMPTY, rmdir '/var/www/html/php-crawler/node_modules/laravel-elixir/node_modules/babel-preset-es2015/node_modules/babel-plugin-transform-es2015-classes/node_modules/babel-runtime/node_modules/core-js/es6'
npm ERR! error rolling back [email protected] { [Error: ENOTEMPTY, rmdir '/var/www/html/php-crawler/node_modules/laravel-elixir/node_modules/babel-preset-es2015/node_modules/babel-plugin-transform-es2015-classes/node_modules/babel-runtime/node_modules/core-js/es6']
npm ERR! error rolling back errno: 53,
npm ERR! error rolling back code: 'ENOTEMPTY',
npm ERR! error rolling back path: '/var/www/html/php-crawler/node_modules/laravel-elixir/node_modules/babel-preset-es2015/node_modules/babel-plugin-transform-es2015-classes/node_modules/babel-runtime/node_modules/core-js/es6' }
npm ERR! [email protected] install: node scripts/install.js
npm ERR! Exit status 127
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is most likely a problem with the node-sass package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! node scripts/install.js
npm ERR! You can get their info via:
npm ERR! npm owner ls node-sass
npm ERR! There is likely additional logging output above.

npm ERR! System Linux 3.19.0-15-generic
npm ERR! command "/usr/bin/nodejs" "/usr/bin/npm" "install"
npm ERR! cwd /var/www/html/php-crawler
npm ERR! node -v v0.10.25
npm ERR! npm -v 1.4.21
npm ERR! path /var/www/html/php-crawler/node_modules/laravel-elixir/node_modules/babel-preset-es2015/node_modules/babel-plugin-transform-es2015-sticky-regex/node_modules/babel-types/node_modules/lodash/lodash.js
npm ERR! fstream_path /var/www/html/php-crawler/node_modules/laravel-elixir/node_modules/babel-preset-es2015/node_modules/babel-plugin-transform-es2015-sticky-regex/node_modules/babel-types/node_modules/lodash/lodash.js
npm ERR! fstream_type File
npm ERR! fstream_class FileWriter
npm ERR! code ENOENT
npm ERR! errno 34
npm ERR! fstream_stack /usr/lib/nodejs/fstream/lib/writer.js:284:26
npm ERR! fstream_stack Object.oncomplete (evalmachine.:107:15)
npm ERR! error rolling back Error: ENOTEMPTY, rmdir '/var/www/html/php-crawler/node_modules/laravel-elixir/node_modules/babel-preset-es2015/node_modules/babel-plugin-transform-es2015-classes/node_modules/babel-helper-define-map/node_modules/lodash'
npm ERR! error rolling back [email protected] { [Error: ENOTEMPTY, rmdir '/var/www/html/php-crawler/node_modules/laravel-elixir/node_modules/babel-preset-es2015/node_modules/babel-plugin-transform-es2015-classes/node_modules/babel-helper-define-map/node_modules/lodash']
npm ERR! error rolling back errno: 53,
npm ERR! error rolling back code: 'ENOTEMPTY',
npm ERR! error rolling back path: '/var/www/html/php-crawler/node_modules/laravel-elixir/node_modules/babel-preset-es2015/node_modules/babel-plugin-transform-es2015-classes/node_modules/babel-helper-define-map/node_modules/lodash' }
npm ERR! Error: ENOENT, lstat '/var/www/html/php-crawler/node_modules/laravel-elixir/node_modules/babel-preset-es2015/node_modules/babel-plugin-transform-es2015-classes/node_modules/babel-runtime/node_modules/core-js/modules/es6.reflect.construct.js'
npm ERR! If you need help, you may report this entire log,
npm ERR! including the npm and node versions, at:
npm ERR! http://github.com/npm/npm/issues

Assets not loading

I followed the project installation guide, but as you can see, CSS and JS are not loading:

404 not found

Hi ,

i've installed php-crawler on ubuntu. but when i try to access the webgui i get an 404 page not found. Could you please help me with this error?

images

How stay over the same 1 domain ?

Not crawling anything

This software doesnt crawling anything for me

Weird Error

[PDOException]
SQLSTATE[HY000] [1049] Unknown database 'php-crawler'

I created all the databases etc. There is the "php_crawler" database in mySQL. Error happens when I do

"php artisan crawler:build"

Thx!!

Timeout to response of URL

When entering a URL, there are times that stays constantly looking at that URL without continuing, it is possible to put a timeout period by URL?

composer test end with error

hello sir, getting error on successfully installation

ERRORS!
Tests: 54, Assertions: 0, Errors: 54.
Script vendor/bin/phpunit handling the test event returned with error code 2

block list

Is there a way I can create a block list so it does not crawl those links if fund in page like facebook, instagram etc etc

Problème avec CORS policy

Bonjour,

J'ai un soucis avec le script, il me met des erreurs :

Access to XMLHttpRequest at 'http://crawler.localhost/api/v1/users/me' from origin 'http://localhost:8000' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.

Access to XMLHttpRequest at 'http://crawler.localhost/api/v1/users//searches?page=1' from origin 'http://localhost:8000' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.

Comment puis je rectifier ça ? merci