rivermont / spidy Goto Github PK

It would be great to have spidy on the Package Index (the new one); installing would be only one command. I have tried to get it going, my efforts can be found on the pypi-dev branch.

There are some problems with imports, file saving, tests, etc.

While I would like to be the owner/maintainer of the package, some help getting it started would greatly appreciated.

Reppy doesn't work past Python 3.8 - seomoz/reppy#122, seomoz/reppy#132 - which means our robots.txt parser isn't working (#81).
Python 3.8 also reaches end-of-life next year so this needs to happen anyway.

• Branch without any robots parser
• Find replacement robots parser
• Test whole program for 3.9-3.11 compatibility

Getting the below error

  File "crawler.py", line 217
    package='reppy')
    ^
IndentationError: unexpected indent

when running python crawler.py

Incidentally, that line was changed recently in 3a62f5e

I have been asked multiple times what makes Spidy different from similar projects like Scrapy and BeautifulSoup. There is a section in the README, but it doesn't directly address other projects.

Can you please use GitHub Releases so I can Unwatch issues and only follow via Watch-> Custom->Releases.

Currently, a request is sent for a site's robots.txt every time a link is crawled. It would be much faster if results of a robots.txt query were saved in some database. Only one request should need to be sent.

Possibly something reminiscent of the Python logo, but not obvious. Other than that, anything would be great.

Crawling would go much faster connecting to multiple pages at once.

Possible Problems:

• Crawling same page twice
• Corruption of save files if reading/writing at different places at the same time.

These should be solvable using mutexes.

Arguments for:

• Overwrite existing save files
• Raise Errors (possibly only for different severity levels?)
• Save pages
• Save words
• Zip files
• Override file size check.
• Domain restriction
• Respect robots.txt
• Custom save file locations
• Autosave count
• HTTP headers
• Max errors
• Starting page

NOTES: Don't try to use sys.argv.

Checklist

• Same issue has not been opened before.

Expected Behavior

All threads to stop as crawler prints info and saves files.

Actual Behavior

Once one thread reaches SAVE_COUNT links crawled, it saves while the other threads continue. This results in [CRAWL] logs in between [INFO] logs.

It seems like this is inefficient and could result in some saving errors.

Steps to Reproduce the Problem

1. Run crawler
2. Wait for the autosave cap to be hit.

Specifications

• Crawler Version: 1.6.2
• Platform: Ubuntu (16.04 LTS)
• Python version: 3.5.2
• Dependency Versions: All latest.

After multiple tries I have yet to get PyPI to format the README correctly. Current state can be viewed here.

At the moment, I followed this SO answer and convert README.md to ReStructuredText when running setup.py. However it looks like you cannot have relative links on a PyPI page, so I changed them all to https links to the GitHub files.

It's still displaying the RST as text and not rendering it, but I don't know where to go from here.

Calling any passers-by to take a moment to submit feature requests or bugs, no matter how small!

Please see the README for a general overview of this project, docs.md for some outdated documentation, and CONTRIBUTING.md for some more words.

So far spidy has been developed on Windows for Windows, but obviously that won't work.

After doing some testing, it seems that on Linux (Ubuntu 16.04, at least) the crawler interprets folders such as config/ with the slash as part of the folder name, whereas on Windows the slash is needed to indicate being a folder.

I've been looking for a decent command-line web crawler for some time and came across this project. It seems very promising.

I have been working on getting this project working in a docker container.

Would you be interested in my contributing the dockerfile back to this project for anyone else who might be interested in the same?

hi, i tried to use spidy b.c. it looked promising.
Is it dead?

first:
sudo pip install -r requirements.txt
doest work, reppy is not installable (python 3.9)

snd:
Docker is a pita...
Please look into ConfigArgParse if you need config files BUT make sure that arguments can be used as well
with docker, there is no error log...
I ended with
docker run --rm -it -v $PWD:/data -w /data --entrypoint /src/app/spidy/crawler.py spidy
so that the error log is accessible (why is there no config option?!)

why is a suffix on the config file enforced? What is that? Windows?

thrd:
my config contained either an Ip or a hostname (resolved via /etc/hosts)
Spidy did not spider either.
For the hostname option it gave

ERROR: OSError
EXT: HTTPConnectionPool(host='example.com', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f4ae176ecc0>: Failed to establish a new connection: [Errno -2] Name or service not known',))

Seems that it doesnt respect /etc/hosts?!
But neither did the ip option work...
e.g. '192.168.1.55/wiki/'

Feature Description

Some cookie handling functionality would be pretty valuable. Setting cookies in the config file should be trivial to implement. An option to send a GET or POST every n scraped URLs / minutes, and apply the recieved cookies to subsequent request would be great too. I might do a PR if I have the time in the near future.

Checklist

• This feature does not already exist.
• This feature has not been requested before.

If you run the crawler and take a look at crawler_bad (or whatever the bad links file is for your configuration), it will have some links in byte form - another problem - but most lines will be single characters. Must fix.

CONTRIBUTING.md has some guidelines, but essentially there is simply a lot of stuff that needs filled out in the docs.

Also, if you would like to use another documentation format feel free. Listing everything is something I came up with in early development but it's probably not scalable.

I will add a .travis.yml shortly, it may or may not work.

Expected Behavior

Docker should simplify things not make them harder

Actual Behavior

Docker is a strugle, you have to build image several times before it works:
It ignores configs that in /data directory, and use only those defaults which was in repo.
It creates results as root, etc etc.

What I've tried so far:

Best workaround is to use -v $PWD:/src/app/spidy/config/ but it still ugly

Error/Bug in code:

Expected Behavior

a project management platform that integrates with GitHub
If you make changes to crawler.py...

Actual Behavior

an project management platform hat integrates with GitHub
If you make changed to crawler.py...

Steps to Reproduce the Problem

1. change "an" to "a"
2. change "hat" to "that"
3. change "changed" to "changes"

What I've tried so far:

Simple spelling (typo) and grammar correction, also testing contribute process.

Specifications

• Crawler Version: NA
• Platform: NA
• Python Version: NA
• Dependency Versions: NA

Checklist

• Same issue has not been opened before.

The initial HEAD request sent to get the document's size is not using the set HTTP headers that the GET request is. Should be a simple fix.

It would be a good idea to have templates for these. I have created TEMPLATES.md as a start but would love some more input.

I looked here to get started.

Expected Behavior

crawl like hell

Actual Behavior

dies of an unknown error

Steps to Reproduce the Problem

echo "https://www.golem.de/"> ./crawler_todo.txt
spidy

What I've tried so far:

Using spidy

Checklist

• Same issue has not been opened before.

Expected Behavior

Raise an InputError and then stop the crawler.

Actual Behavior

No output, just exits straight to console.

Steps to Reproduce the Problem

1. Run crawler
2. Choose 'No' to config file.
3. Type incorrect type of input

Specifications

• Crawler Version: 1.6.2
• Platform: Ubuntu (16.04 LTS)
• Python Version: 3.5.2
• Dependency Versions: Latest

From tests.py.

• test_make_words_given_string
  • Fails with AttributeError: 'str' object has no attribute 'text'
  • make_words needs to be passed a requests.models.Response object in order to extract to text properly, however I'm not sure there is a way to simulate that. Something like StringIO for files...?
  • I feel like getting a page and passing it would take too long to be an acceptable test, but I could be wrong. Would take ~5 seconds.
• test_mime_lookup_given_unknown_type
  • Fails with crawler.HeaderError: Unknown MIME type: this_mime_doesn't_exist
  • It should be caught with the assertRaises statement.

Feature Description

It would be great to have a check in tests.py for the multithreading and queue.

Checklist

• This feature does not already exist.
• This feature has not been requested before.

Having a clicky interface has been a goal for a long time now. There are many users who abhor the command line but are still interested in the tools that use them.

• The remnants of a TkInter interface can be found in gui.py.
• Some thoughts can be found in the docs here, as well as a wireframe sketch.

Most likely using something like py2exe.

The crawler concat the child's uri relative to the parent :
https://mysite/folder/page
=> found : /js/main.js
https://mysite/folder/page/js/main.js

Same thing when a link doesn't have protocol declared :
https://mysite/folder/page
=> found : //subdomain.mysite/images/myimage.png
https://mysite/folder/page//subdomain.mysite/images/myimage.png

Install
apt-get install python3 python3-lxml python3-requests
apt-get install python3-pip python-pip
pip3 install spidy-web-crawler

Starting spidy Web Crawler version 1.6.5

Am i the only one with this problem ?

Thx for you help

I have no experience with writing Python tests, but it seems to be necessary when a program gets big.

Since the small parts of the crawler are already broken up into separate functions, one approach might be to test that each function runs without error and returns the expected type.

We would love to confirm that Spidy will run on all systems, and fix any bugs that may be hidden!

1. Install spidy either from source, through PyPI, or a GitHub Release (instructions found in the README). Run the crawler, using any config file (preferably rivermont-infinite or heavy to test all features) or a custom configuration.
2. Report any bugs by opening a new Issue here.
3. Comment your Platform specs, Python version, spidy version, etc. No information is too much.

Checklist

• Same issue has not been opened before.

Expected Behavior

No errors.

Actual Behavior

Seemingly randomly, crawling a url will fail with a

string index out of range

error. There doesn't seem to be anything wrong with the URLs:

http://www.denverpost.com/breakingnews/ci_21119904
https://www.publicintegrity.org/2014/07/15/15037/decades-making-decline-irs-nonprofit-regulation
https://cdn.knightlab.com/libs/timeline3/latest/js/timeline-min.js
https://github.com/rivermont/spidy/
https://twitter.com/adamwhitcroft

Steps to Reproduce the Problem

1. Run the crawler.
2. Wait a few seconds.

What I've tried so far

Raising the error gave the traceback:

Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "crawler.py", line 260, in crawl_worker
    if link[0] == '/':
IndexError: string index out of range

Specifications

• Crawler Version: 1.6.0
• Platform: Linux (Ubuntu 16.04 LTS)
• Dependency Versions: All latest

There should be an option (disable-able) to ignore links that are forbidden by a site's robots.txt. Another library or huge regex might need to be used to parse out the domain the page is on.