Add Web Annoyances Ultralist about filtersregistry HOT 39 CLOSED

@ameshkov Thanks for the testing and taking a look!

I understand your thoughts around splitting rules and originally the list was setup this way but I was encouraged by others to optimize & reduce the filterlist file size and filter counts being displayed.

I honestly see Pros & Cons to both approaches of list maintenance.

That being said, I'll be the first to admit I am not great with coding (strange, I know 😆) and don't really know of an easy way to automate splitting out the list on my end. Any suggestions on that front you would recommend?

One other thought I had - (please forgive my ignorance as I have no idea of the technical feasibility of this) but wondering if the list could be parsed into new lines when there are multiple hiding rules at the time the list gets loaded into AdGuard? This would preserve the list file size reductions/optimizations with the direction I've moved. I recognize this could be a technical challenge to implement something like that on your side however so would be happy to hear any suggestions or thoughts from you.

from filtersregistry.

It's mainly for floating headers and footers making static. I assume that probably in that cases :style() must be used, sometimes are duplicated static and floating headers, but more often are only floating, so that can be solved only by using :style() or sometimes by script:inject() (but that's not supported by AdGuard).

from filtersregistry.

Thanks for the forum link. Looking forward to seeing your ideas for how to make the changes in VS Code.

@yourduskquibbles uh you have a LOT of combined rules:)

My way will make it easier to split them, but it still will take some time.

Here's what you need to do:

1. Open your repo in VS code
2. Edit -> Replace in Files
3. Enable regular expressions (.* icon in the search string)
4. Use this regular expression: ^(.*?##)(.*?),\s*(.*)
5. Replacement string: $1$2\n$1$3
6. Click ... and put this mask to files to include: *.txt
7. Press "Enter": https://uploads.adguard.com/up04_ocdv7_webannoyances.png
8. Now you can preview every replace and apply them one by one: https://uploads.adguard.com/up04_kq35e_ultralist.txt__ultralist.txt_Replace_Preview__webannoyances.png
9. Alternatively, you can apply all replacements immediately and then review git diff.

Please note, that you'll need to apply this regex/replace several times because there are a lot of rules where you have more than 2 selectors combined, and it just splits one in two.

from filtersregistry.

the difference is obvious to me

You forgot to check "Disable cache" there

from filtersregistry.

1. The plain file size is 135KB

2. The compressed file size is 54KB (that's what ad blockers download)

3. My point was that if the rules are unrolled, the plain size will be about 200KB, but the compressed size will still be under 60KB so the bandwidth savings are minuscule.

4. And 1MB is the size of the complete file history, and this is not what we're downloading.

from filtersregistry.

Re-assigned to myself -- I'll do it on Monday (we are a little bit too busy this week).

from filtersregistry.

not the most optimised list I've seen.

from filtersregistry.

Had a bit more of a dive into this subscription;

Some constructive criticism;

1. The filter sub seems to cover ads, paywalls, social and others? not sure if the author know what they're wanting? (basically lots of overlap on existing subs).
2. How does it differ from other subscriptions?
3. Its needs to be more modular/split.
4. Very unoptimized, some filters are longer than they should. The use of :style in nearly every filter will cause slow downs.

from filtersregistry.

@ryanbr when I originally published Web Annoyances Ultralist I was using it as a learning tool to help me learn CSS and ABP filter syntax and to more easily sync my personal filterlist between devices. Now that the project has a number eyes on it I realize I never really prioritized setting it up from the beginning in the most efficient manner or tried to find tools to help maintain it.

I have been building my own tools to help with the maintenance recently but am wondering if you (or other list maintainers) have any tools that you would be willing to share (either privately or publicly) that you use to help optimize and maintain your lists. It would be great to know what others are doing so I can continue cleaning up the list in the most efficient way possible.

from filtersregistry.

At AdGuard, we're using this VS Code plugin for syntax highlighting:
https://marketplace.visualstudio.com/items?itemName=adguard.adblock

from filtersregistry.

@ameshkov & @Alex-302

I've had some time to optimize the list quite a bit and am wondering if you would like to revisit for inclusion into adguard filter lists?

from filtersregistry.

@yourduskquibbles thank you for reminding us!

Looks good to me, but I'll need a day or two to test it.

There's just one issue I noticed -- you often combine different element hiding rules in one:
verywell.com,verywellfamily.com,verywellfit.com,verywellhealth.com,verywellmind.com##.active.tmp-placeholder, .header-title

The problem with this approach is that it would be unclear which one of the selectors did hide the element. For instance, when you analyze rules in the filtering log or if you're trying to find redundant rules automatically.

I'd suggest splitting it into multiple rules:

verywell.com,verywellfamily.com,verywellfit.com,verywellhealth.com,verywellmind.com##.active.tmp-placeholder
verywell.com,verywellfamily.com,verywellfit.com,verywellhealth.com,verywellmind.com##.header-title

from filtersregistry.

but I was encouraged by others to optimize & reduce the filterlist file size and filter counts being displayed

I don't think that this is what Ryan meant by "filters are longer than they should". I guess what he meant was that some CSS selectors were suboptimal.

Btw, let me please advertise a nice new forum which purpose is to discuss such things: https://hub.filterlists.com/

That being said, I'll be the first to admit I am not great with coding (strange, I know 😆) and don't really know of an easy way to automate splitting out the list on my end. Any suggestions on that front you would recommend?

You are using VS Code, right? Tomorrow I'll show how to do it in VS Code rather easily, you don't even need to code anything.

One other thought I had - (please forgive my ignorance as I have no idea of the technical feasibility of this) but wondering if the list could be parsed into new lines when there are multiple hiding rules at the time the list gets loaded into AdGuard? This would preserve the list file size reductions/optimizations with the direction I've moved. I recognize this could be a technical challenge to implement something like that on your side however so would be happy to hear any suggestions or thoughts from you.

Interesting thing is that on iOS we have to do exactly the opposite -- combine rules in order to fit into the Safari content blocker max rules limit.

However, splitting them into multiple rules is more complicated because you cannot simply split them by comma -- in some CSS selectors comma might be a part of the selector. For example, div[attr=",,,,"]

from filtersregistry.

Thanks for the forum link. Looking forward to seeing your ideas for how to make the changes in VS Code.

from filtersregistry.

@yourduskquibbles one more thing I forgot to mention. Some of AdGuard software is paid, but any filter maintainer can get a license, just let us know if you need any: [email protected]

from filtersregistry.

Thanks for the forum link. Looking forward to seeing your ideas for how to make the changes in VS Code.

@yourduskquibbles uh you have a LOT of combined rules:)

Correct @ameshkov I honestly wouldn't be surprised if the filter count "unrolled" for the main list and sublists was something like 80K rules. Most likely I will just unroll the filters in the main ultralist.txt and leave the subfilter lists rolled together. Do you see an issue with that approach? The sublists are much less likely to be blocking something that is hard to figure out what is being blocked.

Here's what you need to do:

Open your repo in VS code
Edit -> Replace in Files
Enable regular expressions (.* icon in the search string)
Use this regular expression: ^(.?##)(.?),\s*(.*)
Replacement string: $1$2\n$1$3
Click ... and put this mask to files to include: *.txt
Press "Enter": https://uploads.adguard.com/up04_ocdv7_webannoyances.png
Now you can preview every replace and apply them one by one: https://uploads.adguard.com/up04_kq35e_ultralist.txt__ultralist.txt_Replace_Preview__webannoyances.png
Alternatively, you can apply all replacements immediately and then review git diff.

Please note, that you'll need to apply this regex/replace several times because there are a lot of rules where you have more than 2 selectors combined, and it just splits one in two.

Thank you! I will test this out in the next day or two.

from filtersregistry.

I honestly wouldn't be surprised if the filter count "unrolled" for the main list and sublists was something like 80K rules.

It's over 18K actually. Nevertheless, this is a substantial number.

Most likely I will just unroll the filters in the main ultralist.txt and leave the subfilter lists rolled together.

I just realized that AdGuard will not properly apply rules like this:
10best.com###nav-branding, #secondary:style(position: relative !important;)

AG internally will try to convert the whole rule to #$# syntax:

10best.com#$##nav-branding, #secondary { position: relative !important; }

Instead of:

10best.com###nav-branding
10best.com#$##secondary { position: relative !important; }

from filtersregistry.

Something that can be changed in how AdGuard applies filter rules to make it interpret & apply filter rules as uBO does or is it something that can not / not worth changing in how the AdGuard filtering engine works?

Just as an FYI, some sites combine DOM elements in their CSS like I have even if you ignore the "extra" grouping/optimization I have done. I will try to find an example link to demonstrate what I mean in pictures in case that wasn't worded clearly.

Edit: Here is an example of what I am talking about for anyone else stumbling upon this.

Example domain from what I am talking about: https://www.aboutamazon.com/ check how they apply one set of CSS properties to multiple div elements because their homepage uses different element names than an article.

Test Link 1: https://www.aboutamazon.com/

Test Link 2: https://www.aboutamazon.com/working-at-amazon

As an example, (not the greatest one) but if you wanted to adjust the display property from flex to block for all of those .XXXXX-hat-contentWrapper elements it's much more efficient to do so on one line vs 20 individual filter lines just as aboutamazon.com does in their site CSS.

from filtersregistry.

Something that can be changed in how AdGuard applies filter rules to make it interpret & apply filter rules as uBO does or is it something that can not / not worth changing in how the AdGuard filtering engine works?

Yeah, it's just your filter list is the first one where I encountered this.

Just as an FYI, some sites combine DOM elements in their CSS like I have even if you ignore the "extra" grouping/optimization I have done. I will try to find an example link to demonstrate what I mean in pictures in case that wasn't worded clearly.

Sure, this is normal for developers to group CSS styles for similar elements.
But in the case of an ad blocker, it is more convenient to deal with individual rules.

Take your nytimes rule for example, how would you understand which of these selectors are no more needed?
nytimes.com###HPBottom2, #HPHalf, #Top, #TopAd, #acm-wrapper, #bottom-wrapper, #critical-alerts, #digital_hoverCraft, #dual_hoverCraft, #duallink, #gatewayUnit, #masthead-placeholder, #navigation-edge, #nytmm_fixie, #recipeDetailsPromoSlot, #saudi-series-box, #upNext, .Dock-dock--1EAX7, .InlineMessage-inline--2ysnq, .Ribbon-dockHidden--3r3jo, .app-links, .growl-v2, .growl-v2-gradient, .lazyloaded.embeddedContent, .max300.min300.iframe-holder, .miniNavContainer, .nav-wrapper, .not-signedin, .nytc---sitesearch---siteSearch, .opinion-c-col-bottom-region, .ribbon-page-navigation, .series-nav-button, .show.lie-continue, .shown, .subColumn-2.noBackground.mobilePromo.nocontent, .no-fill.rad-top-nav, .meter_default, .css-13ta9od, .css-15u353c, .css-1plijz3, .css-1ragzwk, .css-1y7ysfx, .css-eqw76k, .css-ho7eae, .css-ilfk4y > .css-y8aj3r, .css-kp204g, .e12j3pa50.css-1iv69x4, .e1p1csu90, .efqptxt0, .e1am8aqz0.css-1597c7c, .edz5sly0.css-190ncxp, .css-1xz93ux, .masthead.in-content .kicker, .masthead.in-content .kicker-label, .masthead.masthead-theme-standard .comments-button, .masthead.masthead-theme-standard.in-content .story-meta, BlockAdvert-ad--2QvJV, .BlockAdvert-adContainer--1aZlM, .ResponsiveAd-topAd--3WpE4, .ad, .e12j3pa51, .g-ad-wrapper, .ribbon-ad-container, .text-ad, [id^="dfp-ad-"], [id^="story-ad-"], div[data-testid^="inline-message"], #masthead .story-meta, .masthead-theme-transparent.masthead.in-content .sections-button, .masthead.in-content.masthead-theme-transparent-ffffff .sections-button, .viewport-small-20 .masthead-theme-transparent.masthead.in-content .search-button, .viewport-small-20 .masthead.in-content.masthead-theme-transparent-ffffff .search-button, .css-1ahhg7f, .css-4g4cvq, .has-comments .masthead-theme-transparent.masthead.in-content .comments-button .icon, .has-comments .masthead.in-content.masthead-theme-transparent-ffffff .comments-button .icon, .theme-speech-bubble-small.comments-button.button, .masthead .collection-meta .kicker .kicker-label, .masthead.in-content .story-meta, .masthead.in-content .collection-meta

from filtersregistry.

Something that can be changed in how AdGuard applies filter rules to make it interpret & apply filter rules as uBO does or is it something that can not / not worth changing in how the AdGuard filtering engine works?

Yeah, it's just your filter list is the first one where I encountered this.

Just as an FYI, some sites combine DOM elements in their CSS like I have even if you ignore the "extra" grouping/optimization I have done. I will try to find an example link to demonstrate what I mean in pictures in case that wasn't worded clearly.

Sure, this is normal for developers to group CSS styles for similar elements.
But in the case of an ad blocker, it is more convenient to deal with individual rules.

Take your nytimes rule for example, how would you understand which of these selectors are no more needed?
nytimes.com###HPBottom2, #HPHalf, #Top, #TopAd, #acm-wrapper, #bottom-wrapper, #critical-alerts, #digital_hoverCraft, #dual_hoverCraft, #duallink, #gatewayUnit, #masthead-placeholder, #navigation-edge, #nytmm_fixie, #recipeDetailsPromoSlot, #saudi-series-box, #upNext, .Dock-dock--1EAX7, .InlineMessage-inline--2ysnq, .Ribbon-dockHidden--3r3jo, .app-links, .growl-v2, .growl-v2-gradient, .lazyloaded.embeddedContent, .max300.min300.iframe-holder, .miniNavContainer, .nav-wrapper, .not-signedin, .nytc---sitesearch---siteSearch, .opinion-c-col-bottom-region, .ribbon-page-navigation, .series-nav-button, .show.lie-continue, .shown, .subColumn-2.noBackground.mobilePromo.nocontent, .no-fill.rad-top-nav, .meter_default, .css-13ta9od, .css-15u353c, .css-1plijz3, .css-1ragzwk, .css-1y7ysfx, .css-eqw76k, .css-ho7eae, .css-ilfk4y > .css-y8aj3r, .css-kp204g, .e12j3pa50.css-1iv69x4, .e1p1csu90, .efqptxt0, .e1am8aqz0.css-1597c7c, .edz5sly0.css-190ncxp, .css-1xz93ux, .masthead.in-content .kicker, .masthead.in-content .kicker-label, .masthead.masthead-theme-standard .comments-button, .masthead.masthead-theme-standard.in-content .story-meta, BlockAdvert-ad--2QvJV, .BlockAdvert-adContainer--1aZlM, .ResponsiveAd-topAd--3WpE4, .ad, .e12j3pa51, .g-ad-wrapper, .ribbon-ad-container, .text-ad, [id^="dfp-ad-"], [id^="story-ad-"], div[data-testid^="inline-message"], #masthead .story-meta, .masthead-theme-transparent.masthead.in-content .sections-button, .masthead.in-content.masthead-theme-transparent-ffffff .sections-button, .viewport-small-20 .masthead-theme-transparent.masthead.in-content .search-button, .viewport-small-20 .masthead.in-content.masthead-theme-transparent-ffffff .search-button, .css-1ahhg7f, .css-4g4cvq, .has-comments .masthead-theme-transparent.masthead.in-content .comments-button .icon, .has-comments .masthead.in-content.masthead-theme-transparent-ffffff .comments-button .icon, .theme-speech-bubble-small.comments-button.button, .masthead .collection-meta .kicker .kicker-label, .masthead.in-content .story-meta, .masthead.in-content .collection-meta

Well nytimes.com is interesting because they use many different page templates so not all filters would ever apply to any one page on their domain.

They stylize some of their "featured" articles (e.g. some articles will have a light colored background header image and black NYT logo, while others will have a dark background header pic and white logo and change page templates on different subsections of their site.

I guess you are asking, if nytimes.com decided to change their site template and the filter never applied to the site on any page how would I know to remove the old filter that is no longer necessary? I don't have a great answer on that but I do revisit sites frequently and look for filters that no longer apply.

I imagine if you grep my repository, nytimes.com will be one of my most updated list of domains because of constantly changing and adding new page templates/site design elements and revisiting my filters to improve and iterate on the site design/cluttering filters without breaking the page.

from filtersregistry.

I guess you are asking, if nytimes.com decided to change their site template and the filter never applied to the site on any page how would I know to remove the old filter that is no longer necessary?

Yep, you are right, but I am not talking about the filter, but about some individual selectors from it.

In theory, ad blockers could help filters maintainers and count the number of hits for each filter. Or you could monitor the filtering log and see which rules are being triggered and remove those which are never used. Currently, you simply cannot see it. Check out the example, how would you tell that .redundant-class is no more needed? https://uploads.adguard.com/up04_qrvoi_uBlock__Logger.png

So the only way to keep the rule up-to-date would be to inspect the DOM tree manually, which is tedious.

Now imagine that we're trying to build a tool that will help automate searching for obsolete rules. We will face the same issue: it is harder to find an obsolete rule. If one of 79 selectors matches something, the rule cannot be considered obsolete.

from filtersregistry.

We will face the same issue: it is harder to find an obsolete rule. If one of 79 selectors matches something, the rule cannot be considered obsolete.

True, but my thought on this is that 1 obsolete selector contained within 1 line of 79 selectors is a whole lot more optimized from a size perspective than 78 lines of 78 non-obsolete selectors after removing the 1 obsolete selector but maybe that is only looking at it from a file size/bandwidth perspective and not from any other point-of-view. 😉

Repeating nytimes.com## 78 times vs 1 has some bandwidth cost especially when the list is primarily domain targeted rather than generic in nature.

Edit: Just FYI I am still thinking of unrolling ultralist.txt but leaving the sublists rolled together but wanted to put my rationale out for why I originally started rolling the filters together as a counterpoint.

from filtersregistry.

From the ad blocker's perspective, it does not matter how many lines it takes.
It is compiled in an internal indexed structure that looks different.

What for the bandwidth argument, gzip is used automatically anyway.

For instance, your nytimes rule is about 2KB plain.
gzipped size is about 880 bytes.
Unrolled rule gzipped size is about 900 bytes.

from filtersregistry.

Right but when updating the filterlist and fetching the data its still making a request for the full txt file everytime the filters get updated?

The original reason why I started consolidating the list is due to that thought as well as github not showing the full contents of the file in the preview pane on the overview blob when the filesize is over 1MB.

from filtersregistry.

Right but when updating the filterlist and fetching the data its still making a request for the full txt file everytime the filters get updated?

Github (and our filters server for what it's worth) serves static files gzipped.

Check the "network" dev tools panel:

• https://uploads.adguard.com/up04_p5e1q_httpsraw.githubusercontent.comyourduskquibbleswebannoyancesmasterultralist.txt.png
• https://uploads.adguard.com/up04_ud85z_httpsraw.githubusercontent.comyourduskquibbleswebannoyancesmasterultralist.txt.png

Edit: Just FYI I am still thinking of unrolling ultralist.txt but leaving the sublists rolled together but wanted to put my rationale out for why I originally started rolling the filters together as a counterpoint.

Well, we'll anyways take some time to fix the:style conversion issue at least on the filters server (later we'll fix it in the products as well).

from filtersregistry.

Github (and our filters server for what it's worth) serves static files gzipped.

Check the "network" dev tools panel:

https://uploads.adguard.com/up04_p5e1q_httpsraw.githubusercontent.comyourduskquibbleswebannoyancesmasterultralist.txt.png
https://uploads.adguard.com/up04_ud85z_httpsraw.githubusercontent.comyourduskquibbleswebannoyancesmasterultralist.txt.png

OK this is interesting! I never knew this...

After hearing that github is gzipping the txt files when fetched, I have no problem unrolling ultralist.txt and will have that done in the next day or two.

By the way, I found an example commit of the overview blob being useless from before I started optimizing the list size since that was referenced earlier.

That github site feature being useless is not much of a huge concern for me now since I use external tools rather than commit directly through github.com site though and I imagine even after I unroll ultralist.txt because there are sublists now it will help reduce the chances of going over the 1MB threshold.

Well, we'll anyways take some time to fix the:style conversion issue at least on the filters server (later we'll fix it in the products as well).

Awesome! I'll also work on some improvements on my end to address your concerns about list maintenance with combined filters here. Thanks for the quality discussion!

from filtersregistry.

from filtersregistry.

but is still 135.49 KB vs 1 MB in github repo

from filtersregistry.

Thanks for the regex suggestion @ameshkov, that was surprisingly easy.

I really need to learn regex, been wasting too much time doing things without it!

from filtersregistry.

I just realized that AdGuard will not properly apply rules like this:
10best.com###nav-branding, #secondary:style(position: relative !important;)

I don't think this is supported in uBO.

Or this was only for procedural selectors? Hmmm...

style() is applied to whole list of selectors.

from filtersregistry.

style() is applied to whole list of selectors.

Well, this is good news then as it works the same way in AdGuard.

I am just not sure this was the @yourduskquibbles's intention.

@yourduskquibbles could you please check it again?

Let's take the rule 10best.com###nav-branding, #secondary:style(position: relative !important;)

Did you want to hide #nav-branding and change #secondary position, or did you want to change the position of both elements?

from filtersregistry.

style() is applied to whole list of selectors.

Well, this is good news then as it works the same way in AdGuard.

I am just not sure this was the @yourduskquibbles's intention.

@yourduskquibbles could you please check it again?

Let's take the rule 10best.com###nav-branding, #secondary:style(position: relative !important;)

Did you want to hide #nav-branding and change #secondary position, or did you want to change the position of both elements?

Whether intentional behavior or not, uBO applies the style filter to all elements in the line (and has for many versions). There is no hiding of any elements when combining elements with the same style properties into one line as I do, all elements in that line get the style filter applied and nothing is cosmetically hidden.

from filtersregistry.

@yourduskquibbles

There is no hiding of any elements when combining elements with the same style properties into one line as I do, all elements in that line get the style filter applied and nothing is cosmetically hidden.

Right, the same in AG. For some reason, I was thinking that you wanted to hide those without :style pseudo-class. If I was wrong and it works properly, then I guess there are no problems and we can finally add the Ultralist.

Meanwhile, I've been using it for a few days already, and haven't experienced any issues. Everything works great!

from filtersregistry.

@yourduskquibbles

There is no hiding of any elements when combining elements with the same style properties into one line as I do, all elements in that line get the style filter applied and nothing is cosmetically hidden.

Right, the same in AG. For some reason, I was thinking that you wanted to hide those without :style pseudo-class. If I was wrong and it works properly, then I guess there are no problems and we can finally add the Ultralist.

Nope, that is working "as-intended"! If an element should be removed entirely, it doesn't go in the CSS Style Modifying Sublist.

Meanwhile, I've been using it for a few days already, and haven't experienced any issues. Everything works great!

Awesome 😁

from filtersregistry.

No rush on when you can add the list, I've been maintaining the list for a few years with it being fairly hard to discover for most users, so don't think an extra week hurts anyone. 😎

from filtersregistry.

@yourduskquibbles sorry for the huge delay, our 10th anniversary and all the new releases took all my time, but I'll get back to this soon.

from filtersregistry.

No problem at all, I'm in no rush!

from filtersregistry.

Done: #203

Available in AG for Mac, Android, Win, iOS
It will be added to the browser extensions pre-defined list in the next update.

from filtersregistry.

Awesome @ameshkov, thank you sir!

from filtersregistry.