Unicode handling of --include and --exclude about suckit HOT 8 OPEN

Maybe we can add an option to output URL filtering information to stdout or a file, e.g, if the include or exclude regex matches?
I think this would lead to more transparency what suckit is doing.
I also plan to implement functionality to rewrite the local URLs that could profit from this debug feature.

from suckit.

This looks correct. The best way to know is by testing it, and I would love to see the result of such a test. If you can build this directory tree, just serve it using a webserver and try to run suckit on localhost

from suckit.

@mr-bo-jangles Did it worked ?

from suckit.

Maybe we can add an option to output URL filtering information to stdout or a file

Good idea

I also plan to implement functionality to rewrite the local URLs that could profit from this debug feature.

What do you mean?

from suckit.

What do you mean?

To download a phpBB forum, I added a hack to rewrite some URLs, namely remove a ?sid=<hash> parameter. Otherwise the same pages get downloaded over and over again with different sid hashes.
If you want to take a look:
https://github.com/raphCode/suckit/blob/fusornet_hack/src/scraper.rs#L191

I originally planned to flesh this out into a dedicated feature / command line option, but eventually didn't. I already achieved my goal and I could not figure out a way to do it properly.

from suckit.

The problem with removing parameters such as ?sid is that they might have changed the content of the requested page. If you remove them, 2 links identical except the parameters will have a common page downloaded by suckit while they should have 2 different pages

from suckit.

In general you are correct, but in the specific case of phpBB the content is always the same, no matter the ?sid parameter value.
One solution would be to just ignore all links with this parameter, like suggested here, but this may create a swath of broken links. I just removed the parameter from the URL and collapsed all links into their "canonical" form without the session id parameter.

I actually just found a different solution, namely to send session cookies, which avoids ?sid parameters getting appended to links in the first place.

from suckit.

We could imagine a solution where you whould have a list of tuple with a regex and list of arguments to remove

Vec<(regex, Vec<parameter>)>

But it might be really costly

from suckit.