More options for URL filtering/cleaning about requestcontrol HOT 28 CLOSED

Ok, it's here now:
https://github.com/tumpio/requestcontrol/wiki/Testing-page-for-redirection-filtering

The wiki is public so you are able to edit it directly.

from requestcontrol.

Hi there. These are good suggestions. The first one sounds like a bug. Support for adding multiple match patterns (hosts, paths) for rules can be added. Most of it will be UI related work. A rule with multiple match patterns would just add multiple webRequest API request listeners. I'll work on these when I have some free time. Any help is appreciated.

Also a question. Which regex flavor is used in this extension? Javascript? I'm trying to create a redirection rule to get direct links to Dropbox, but I can't get it to work. Here is the rule: {href/.*www.dropbox.com(/s/.+/.+)?dl=\d/https://dl.dropboxusercontent.com$1}

Oh, I have forgotten to release the fixed version.. I'll do it shortly but you can also try the beta version that includes a fix for this: (and yes it's JavaScript flavor)
https://addons.mozilla.org/addon/requestcontrol/versions/1.4.2beta1

from requestcontrol.

I fixed the first issue with global url param filtering. Can you confirm that it's fixed:
https://addons.mozilla.org/fi/firefox/addon/requestcontrol/versions/1.4.3beta1

from requestcontrol.

@tumpio
Webpage with a few links for testing: http://pipes.droppages.com/links

Regex behavior is strange. With Any URL filter rule enabled I can't get this regex pattern to work: {href/.*site\.com\/away\.php\?to=(.*?)&cc_key=.*/$1}
It should clean up this type of URL: https://site.com/away.php?to=https://github.com&cc_key=.
At the same time regex for Dropbox links works.

Also with Any URL filter rule disabled, all the links with redirection, but without parameters get skipped by addon. For example:
Skipped: https://steamcommunity.com/linkfilter/?url=https://www.linuxmint.com/
Filtered: https://out.reddit.com/t3_5pq7qd?url=https%3A%2F%2Finternethealthreport.org%2Fv01%2F&token=AQAAZV6JWHBBnIcVjV1wvxVg5gKyCQQSdUhGIvuEUmdPZhxhm8kH&app_name=reddit.com

from requestcontrol.

For me it redirects to https://github.com&cc_key=. Is this what happens also for you?

Also with Any URL filter rule disabled, all the links with redirection, but without parameters get skipped by addon. For example:

Can you tell me step by step how to reproduce this? Which rules do you have enabled and which disabled?

from requestcontrol.

I investigate the site.com redirection issue more. I think the problem is with how the promise is being resolved.

from requestcontrol.

For me it redirects to https://github.com&cc_key=. Is this what happens also for you?

Yes. &cc_key= remains and therefore website can't be loaded. With Any URL filter rule disabled or removed, this regex works as expected. Steps to reproduce:

1. Create a Filter rule with Pattern: Any URL; Types: Document and enable it.
2. Create a Redirect rule with regex: {href/.*site\.com\/away\.php\?to=(.*?)&cc_key=.*/$1} and enable it.
3. Click on https://site.com/away.php?to=https://github.com&cc_key=
   Result will be https://github.com&cc_key=
4. Disable Filter rule with Pattern: Any URL.
5. Click on https://site.com/away.php?to=https://github.com&cc_key=
   Result will be https://github.com

Can you tell me step by step how to reproduce this?

My bad. It turns out I've accidentally removed a rule for steamcommunity.com. But I've just found another bug: with Any URL rule enabled other rules keep working even if you disable them. Steps to reproduce:

1. Create a Filter rule with Pattern: Any URL; Types: Document and enable it.
2. Disable any other rule (for example, filter rule for steamcommunity.com).
3. Click on https://steamcommunity.com/linkfilter/?url=https://addons.mozilla.org/
   Result will be https://addons.mozilla.org/ - rule is disabled, but redirection works.

from requestcontrol.

I get the issue now. It's actually problem of two things. First, if multiple rules match a single request, only the action of the first rule gets applied. This is how the webRequest API resolves the request listeners. Second, the default "Filter" action does not strip trailing search parameters from the filtered url if is not encoded. If I remember right some redirection pattern followed this practice.

I'm not certain how these should be fixed. I can change it so that global rules (Any URL) will be set after other rules. This way any site specific rule will get a preference to the global rules.

Do you have other ideas?

from requestcontrol.

First, if multiple rules match a single request, only the action of the first rule gets applied.

How about removing that global URL parameters list from the bottom and just adding an additional field to the Filter rules? Also you can add a button to trim all parameters by default - it might be useful for site specific rules. So, trimming should be applied only to Filters, because regular expressions allow you to do anything you want anyway.
Something like this: https://dl.dropboxusercontent.com/s/i4qcdyabt4yezns/requestcontrol4.png
Just my thoughts, it might be too difficult to implement all this stuff.

Second, the default "Filter" action does not strip trailing search parameters from the filtered url if is not encoded.

But URL can be encoded/decoded before processing, right? encodeURI/decodeURI/unescape

I can change it so that global rules (Any URL) will be set after other rules. This way any site specific rule will get a preference to the global rules.

Yes, I think site specific rules should have a priority.

P. S. Here is an old bookmarklet: https://pastebin.com/wLxJqGf1. It does a pretty good job at removing redirections with or without parameters, either encoded or decoded. Maybe you would find it useful.

from requestcontrol.

Another beta version for you to test. I fixed the default "Filter" action redirection url parsing from a query string that contains additional search parameters separated by ("&" or ";"). Encoded url parameters are kept unlike in the bookmarklet you shared.

https://addons.mozilla.org/firefox/addon/requestcontrol/versions/1.4.3beta2

from requestcontrol.

"Curiosier and curiosier!"
I've tested different scenarios on a test page. + is a good thing, - is a bad thing.

1. Any URL rule doesn't exist:
    - Pre-defined rule for Google (№6 on test page) doesn't work;
    + All other rules work as expected;
    + Links 11, 12, 13 don't get filtered (this is totally fine, because there are no specific rules for them).
2. Any URL rule exists and is disabled:
    - Pre-defined rule for Google (№6 on test page) doesn't work;
    + All other rules work as expected;
    + Links 11, 12, 13 don't get filtered (this is totally fine, because there are no specific rules for them).
3. Any URL rule exists and is enabled:
    + All rules work as expected, all links get filtered.
4. Any URL rule exists and is enabled, but all other rules are disabled:
    + Links 11, 12, 13 get filtered;
    + Rules №10 (Amazon) and №14 (Dropbox) don't work;
    - All other rules work. And this is definitely not as expected. I mean the only purpose of Any URL filter rule is trimming URL parameters, not cleaning redirections.

Testing was done on Firefox 54.0b3 with a new profile.
Tools: HttpFox, Fiddler

from requestcontrol.

• Pre-defined rule for Google (№6 on test page) doesn't work;

Fixed this one. The pre-defined rule was missing "main_frame" in types.

• All other rules work. And this is definitely not as expected. I mean the only purpose of Any URL filter rule is trimming URL parameters, not cleaning redirections.

Hmm, I see. Would a toggle button to turn the redirection cleaning on/off for the "Filter" action suffice? Currently the Filter action always does both the redirection cleaning and global url parameters filtering. I'll add the list of trimmed url parameters also to the Filter rule and remove the global list as you suggested.

from requestcontrol.

Would a toggle button to turn the redirection cleaning on/off for the "Filter" action suffice?

Sounds good to me. With a global list it could be a bit confusing, but it will definitely work for per-site lists.

from requestcontrol.

@Faberman Can you test new beta version?
https://addons.mozilla.org/en-US/firefox/addon/requestcontrol/versions/1.5.0beta1

Changes in this version:
URL parameter filtering is now Filter rule specific.
Redirection cleaning can now be turned off in Filter rule.
Added wildcard "*" support for url parameter trimming.

from requestcontrol.

Everything seems to work fine, but URL parameters trimming is a bit confusing.
For example, this url: http://site3.com/?r=https%3A%2F%2Fwww.yr.no%2Fplace%2FNorway%2FNordland%2FBr%C3%B8nn%C3%B8y%2FBr%C3%B8nn%C3%B8ysund%2Fhour_by_hour.html?key=ms&ww=51802

With URL parameters trimming disabled I get [...].html?key=ms, but it should be [...].html?key=ms&ww=51802, if i'm not mistaken. With 'Trim all' enabled I get [...].html as expected.

That's fine for me, because in most cases I want URL parameters to be trimmed. So, just an observation. :)

Thank you for improving this great extension!

from requestcontrol.

Thanks for testing. That problem is with the url redirection cleaning and in the redirection address extraction. I'm expecting to remove the query parameters of the original url and not to include them in the final redirection address. That url is exceptional because it is not encoding the query string of the redirection address and as such the "ww" parameter is actually interpreted to be a url parameter of site3.com and not yr.no.

This is actually how the Firefox URL interface also reads that address. It reads it as a url of site3.com with a query string that contains two parameters "r" and "ww".

So for this special case I would write a custom redirect rule with regular expression to keep all the paramters :)

from requestcontrol.

Got it. Anyway, some of the redirection patterns on my test page are intended for testing purposes only and most users probably will not encounter them.

I guess, now the extension can be considered "full-featured". As for the multiple match patterns, I think most users will use this feature for whitelists, so maybe it will be easier to implement two global whitelists for hosts and paths. The less code the better.

from requestcontrol.

Yes thanks for creating the test page. It helps to debug the add-on in future as well when I do changes and if things go wrong. Can I mention the test page in add-on description? And if hosting is problem we could place it to as a wiki page in here. https://github.com/tumpio/requestcontrol/wiki

I think I'll still go with the multi match pattern way so that it's possible to set up own global whitelist rule with multiple hosts and paths.

from requestcontrol.

Sure. Wiki is better, because you can edit it if necessary.
Page source (I think Github Wikis support basic html): https://dl.dropboxusercontent.com/s/3idit4f65oc4clc/links.txt

from requestcontrol.

Here's new beta version that adds support for adding multiple hosts and paths for rules:
https://addons.mozilla.org/addon/requestcontrol/versions/1.6.0beta3

from requestcontrol.

After a quick test everything seems to work fine, except one thing: it's impossible to save * (any domain) in hosts. And if you create a whitelist with an empty host field, all filter rules will stop working.

from requestcontrol.

Fixed them in
https://addons.mozilla.org/addon/requestcontrol/versions/1.6.0beta4

Thanks for testing! :)

from requestcontrol.

Awesome. Yes, now everything is ok.

And the last suggestion. How about combining different rule types into sections to visually separate them? Just like you did it for Whitelist rules: https://dl.dropboxusercontent.com/s/d1uj359zoa9pgsd/requestcontrol5.png

from requestcontrol.

Looks great! I've also thought about adding separate tabs for each rule type.

from requestcontrol.

@Faberman a new beta! :P

I added support for multiple rule matching for a single request. This allows to create e.g. a global rule to trim url parameters while having other site specific rules. If more than one redirect or filter rule applies to a single request they will be applied one by one.

Also the rules follow this priority model:

1. whitelist rule
2. block rule
3. redirect rule
4. filter rule

Whitelist rules have the top priority and revokes all other rules. Next are the block rules and they revoke redirect and filter rules. Finally redirect rules will be applied before filter rules but all of them will still be applied.

https://addons.mozilla.org/addon/requestcontrol/versions/1.6.0beta7

This version needs restoring default rules if you have created any whitelist rules.

Edit: 1.6.0beta7 fixes multiple filtering rules handling and showing the right icons.

from requestcontrol.

Nice! I didn't even know that addon couldn't process multiple rules.

As for the 1.6.0beta7, Redirect rules with regex pattern (Dropbox links, for example) don't work.

from requestcontrol.

Fixed the redirection rules and added sections for each rule types in options.

I released 1.6.0 now. Will close this issue now. For new suggestions it's better to open new issue page :)

Thanks for the help!

from requestcontrol.

Nice! I didn't even know that addon couldn't process multiple rules.

Yeah, I finally figured out how to work around the webRequest API limitation. It's a big change in background but should not affect anything else.

from requestcontrol.