Prevent google crawling about tor2web HOT 3 CLOSED

This has been implemented with a config flag

blockcrawl = True

from tor2web.

It seems that you are deciding for everyone.
I think that instead the website owner should have the last choice ( be indexed or not )

Idea:
Leave the blockcrawl set to True by default, but tor2web should check if there is the robot.txt on the website allow crawlers. ( and this last one should have the priority on the blockcrawl setting )
Example:
User-agent: tor2web
Allow: /
( this will open to all crawlers )

User-agent: tor2web
Allow: /
User-agent: Googlebot
Disallow: /
( this will open to all clearnet crawlers but not the google one ... )

from tor2web.

Yes, you are right.
In future, when we'll have something more stable and diffused the idea is to provide a general framework to let the Tor HS owner to configure different behavior of Tor2web (for example also if using https or whenever to configure a easier nickname for his TorHS).

Consider that Tor2web will evolve also including a standard first page disclaimer that require manual approval by end-user and this would break any kind of crawler.

To enable crawler we would need explicitly to enable crawler as a structured feature.

from tor2web.