yubiuser / pihole_adlist_tool Goto Github PK

A tool to analyse how your pihole adlists cover you browsing behavior

License: MIT License

Shell 100.00%

pihole_adlist_tool's Introduction

Maintenance mode

This repo is in maintenance mode: It's unlikely that I will add new features myself but are willing to accept external pull requests. There are several reasons for this decisions:

When I released the first version 4 years ago, Pi-hole' blocking capabilities were more limited than today. It was way more simple to deduct from the gravity database which queries would have been blocked than it is today. Pi-hole's regex filtering could only be taken into account by parsing the output of the pihole-FTL regex engine - which is already kind of a hack. Adding all the code to apply the ABP-style filtering to the queries is out-of-scope (maybe even impossible in a shell script). So the gap between the filtering by Pi-hole and what's reported by the tool gets wider.
The tool is quite slow and I'm not sure if this could be improved within the limits of a shell script.
There is a lack of motivation and time to address the above issues.

I'll added a MIT licence so anyone who would like to continue to improve or use the tool should be able to do so. If I would start fresh, I'll probably would use another language (maybe C, maybe GO) which would allow to implement/reuse FTL's regex engine allowing to parse the regex domains and the ABP-style entries.

Pihole Adlist Tool

This script tries to provide you with a bunch of information that enables you to decide which adlists you need based on your browsing behavior. It does that by matching your browsing history (FTL's querylog) with your current adlist configuration (gravity database) generating a list of domains that you have visited in the past and which would have been blocked if your current adlist configuration would have been in place back then. In a second step the scripts takes this list and attributes each domain to the adlists it is on (similar to what pihole -q does). The final output is a table of all your adlists with the corresponding number of covered domains (domains that you have visited and that would have been blocked if only this particular adlist would have been used).

The script outputs

the number of adlists (and how many are enabled)
the number of unique domains in your gravity.db
the number of blocked domains as reported by Pi-hole ('blocking status == blocked by gravity' or blocking status == blocked by gravity+blocked during CNAME inspection) and how often those domains have been blocked ('hits')
the number of covered domains and how often those would have been blocked ('hits')
special case: domains on your (personal) blacklist which are also on an adlist and have been visited in the past, including hits (run 'pihole -q' to see on which adlist those domains appear)
optional: top blocked domains and number of hits if your current adlist configuration would have been used
adlist table id, status, total domains on adlist, covered domains, hits, unique covered domains, address
the sum of unique covered domains
optional: list of unique covered domains with adlist_id, address
optional: analyse regex blacklist (will be disabled when running Pi-hole in Docker Container!)

As domains usually appear on more then one adlist I introduce the concept of unique covered domains. Those are domains that have been visited, would have been blocked and appear on just one adlist. This might help you to value your adlists not just by how many domains are covered but also what would happen if you disable this adlist.

Limits

~~Disabled blocklist won't be analyzed as gravity is not including domains from deactivated adlists. You can enable all adlists from within the script.~~ The script will warn you, if there is a mismatch between the enabled adlists and data found in the gravity database. Users have the choice to run gravity to clear the mismatch or proceed anyway. In this case the tool will analyze all available data, but results must be interpreted with caution. (see 8dab71)
Black/Whitelisted domains (~~including regex~~ see PR #19 are not considered when calculating the number of covered domains (and hits)
- Whitelisted domains reduce the number of blocked domains as reported by Pi-hole compared to the calculated numbers
- Blacklisted domains increase the number of blocked domains as reported by Pi-hole compared to the calculated numbers
This tool can not deal with domains that have been blocked due to CNAME inspection because Pi-hole doesn't store the actual blocked domain but the CNAME and a corresponding status ("Blocked during deep CNAME inspection"). This CNAME domain will not match a domain from an adlist - if it would it would have been blocked directly. (see PR #3)
Other differences between the number of domains/hits as reported by Pi-hole and calculated numbers are due to change in adlist configuration over time
For the limits of the regex analysis see the notes of PR #19

Caveat

Depending on the number of enabled adlists and the number of visited domains in the selected time period the calculation might take some time - please be patient. On my NanoPi NeoPlus2 (ARM, Quad-core Cortex A53) it takes ~17-18sec to analyse 2.3 million queries from pihole-ftl.db and 347603 domains in gravity.db
Analysis of regex blacklist can take minutes easily!
While lists that have attracted no or only very few hits in the analysis are prime candidates for removal, you should also consider the type of blocklist before you ultimately decide do remove a list, e.g. you may want to keep malware or telemetry focused blocklists nonetheless.

Requirements

Pi-hole FTL v5.5 (see PR #13)
Notes on Docker: I don't run Pi-hole on docker myself and have only limited ability to test the script. Expect things to break anytime.

Installation

Download the tool, either via git clone or link. Make the script executable with chmod +x ./pihole_adlist_tool and run it with ./pihole_adlist_tool

Options

pihole_adlist_tool [options]

Options:
  -d [Num]                        Consider the last [Num] days (Default: 30). Enter 0 for all-time analysis.

  -t [Num]                        Show top blocked domains. [Num] defines the number to show.

  -s [total/covered/hits/unique]  Set sorting order to total (total domains) covered (domains covered), hits (hits covered) or unique (covered unique domains) DESC. (Default sorting: id ASC).

  -u                              Show covered unique domains.

  -a                              Run in 'automatic mode'. No user input is required at all, assuming default choice would be to leave everything untouched.

  -r                              Analyse RegEx as well. Depending on the amount of domains and RegEx this might take a while. Please note: Can only be used, if Pi-hole is NOT running in a Docker Container!

  -v                              Display pihole_adlist_tool's version.

  -h                              Show this help dialog.

Background

As adlist configuration might have changed over time (add/removed adlists, enabled/disabled adlists) this script doesn't rely on Pi-holes blocking status for the analysis but rather determine if queries from the long-term database had been blocked with the current adlist configuration. Relying on the blocking status could lead to wrong assumptions about the coverage of adlist with your current adlist configuration: some domains might have been blocked in the past but wouldn't be blocked now (removed adlist) and some might be blocked now but haven't in the past (added adlist). If the adlist configuration hasn't changed over time, there should be no huge difference between this approach and using Pi-hole's blocking status.

The deeper reason for re-analyzing the queries is that this tool should help you to make predictions for the future: assuming your online behavior is rather stable over time and you analyse a long enough dataset from the past, this tool will tell you which adlist might be worth keeping (because it contains a lot of covered domains) and which you could safely remove (no covered domains and/or covered domains but no unique covered domains).

Support, Contribute & Todo

I'm not a developer. This script is mostly done by copy-pasting snippets I found online. I know there is no proper error and exception handling. If you are willing to improve the script feel free to submit pull requests. Things on my todo list:

Further improve speed of the database handling. The slow steps are
- Select all domains from pihole-ftl.db that are also found in gravity.db
- Get the total number of blocked domains from pihole-ftl.db
- Get the total number of hits from pihole-ftl.db
- ~~Update adlist with the total number of domains from gravity.db for each adlist~~ (see e0af664)
Format SQL output with awk

pihole_adlist_tool's People

Contributors

Stargazers

Watchers

pihole_adlist_tool's Issues

Database locked error

So I ran the script using bash, and it said Error: database is locked. I think I'm running it wrong.

List the domains which generated a hit

Hey,
Thanks so much for your tool, it allowed me to really streamline my lists.

I was wondering if you might be able to help with another issue. I am trying to figure out a way to see which domains have triggered in each list.
I have a malware list loaded and it is really difficult to figure out which domains are being triggered on my network. Using the pihole webgui there is no way to search for blocked domains by list.

I thought with some tweaking your tool might be able to do that?

This is my output when I run the tool with -a

`[i] Adlist coverage

id enabled total_domains domains_covered hits_covered unique_domains_covered address

1 1 209627 593 60232 589 https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
5 1 167912 1 2 https://v.firebog.net/hosts/RPiList-Malware.txt `

I want to know what the domain is that is listed against the malware list so I can search for it in the webgui to find out which client triggered the hit

Many thanks

Instructions?

Could you please provide a step-by-step guide on how to install/use this tool?

Rgds
newbie

Errors encountered

Thanks for the nice tool! It runs for me, but appears to encounter a few issues. Here's a portion of the output --

  [i]  Calculating.....
  [i]  This might take some time - please be patient.
Error: near line 10: no such column: COUNT
Error: near line 11: no such column: COUNT
Error: near line 12: no such column: COUNT

  [i]  You have 9 adlists configured (5 enabled)
  [i]  Your gravity.db contains 157966 unique domains
  [i]  Since 22.12.2020 10:25:22 1108 different domains from your adlists have been blocked 336922 times in total
       (blocked directly by gravity or during deep CNAME inspection)
  [i]  Using you current adlist configuration 1316 domains would have been blocked 440581 times


  Press enter to continue...
./pihole_adlist_tool: line 557: [: : integer expression expected
./pihole_adlist_tool: line 577: [: : integer expression expected

Pi-hole is running on a Ubuntu 20.04 VM and was just updated today to the latest version ( Pi-hole v5.2.4 Web Interface v5.3.1 FTL v5.5.1)

Any idea on why these errors are occurring?

Starting with version 2.4.0, you get multiple 'environment: line 5: docker: command not found' messages

Am still checking what's happening here, but it seems that somewhere in the script the docker command gets called when it shouldn't (and as end result no changes get applied).

Other info:

The script detects/sets PIHOLE_DOCKER to 0 (when it should).
To reproduce the bug run the script with 0 params and pick option 1(Current adlist configuration) followed by 3 (Enable the minimal number of adlists that cover all domains that would have been blocked).
I think this might be happening due to SUDO_SQLITE. Am assuming the value of PIHOLE_DOCKER doesnt stay/is visible when we call SUDO_SQLITE, but am really not sure.

Line 7 problem

Hello, After downloading the file, here is what I have:

Any Idea?
Thanks

[Recommendation] - Make Current Adlist Default Option

Thank you for making this wonderful tool, I noticed that after you run the tool, an option relating to Analysis of Either All or Current Adlist comes up. This could either be (A) Defaults to 5 seconds to automatically select current or (B) be available only if Interactive mode has been used to run the script.

"SUDO_SQLITE: command not found"

Keep all adlists enabled

Enable only adlists with covered unique domains

Enable the minimal number of adlists that cover all domains that would have been blocked

Restore previous adlist configuration

Please select: 2

[i] Enabling adlists with covered unique domains....
bash: 0=0: command not found
bash: SUDO_SQLITE: command not found
bash: sqlite: command not found
bash: 0=0: command not found
bash: SUDO_SQLITE: command not found
bash: sqlite: command not found
bash: 0=0: command not found
bash: SUDO_SQLITE: command not found
bash: sqlite: command not found
[✓] Reloading DNS lists

[✓] Adlists with covered unique domains enabled

[✓] Of the 833 domains that would have been blocked by the adlist configuration you chose to analyze,
0 domains are covered by your current adlist configuration.

[i] Removing temporary database...
[✓] Temporary database removed

Note that:
$ sqlite

-bash: sqlite: command not found

$ sqlite3

SQLite version 3.27.2 2019-02-25 16:06:06
Enter ".help" for usage hints.

So I created an alias "sqlite='sqlite3'" but that did not solve the error message.
How would I be able to fix this?

I'm also a bit worries about the message:

[✓] Of the 833 domains that would have been blocked by the adlist configuration you chose to analyze,
0 domains are covered by your current adlist configuration.

Does this mean that there's domains not covered (833) that should be covered?

Wrong detection of PIHOLE_DOCKER?!

I executed docker exec -it pihole bash and then:

root@pihole-merz-nimbus:/etc/pihole# ./pihole_adlist_tool
  *** Pihole Adlist Tool 2.6 ***
   ++++++++ Info ++++++++
  [i]  PIHOLE_DOCKER: No
  [i]  PIHOLE_DNSMASQ_VERSION: 2.87test4-18
  [i]  SQLITE_VERSION: 3.37.1
  [i]  DAYS_REQUESTED: 30
  [i]  TOP: Not shown
  [i]  SORT_ORDER: id ASC
  [i]  UNIQUE: Not shown
  [i]  REGEX_MODE: Disabled
  [i]  There is an update available: 2.6.0
…

So I can say for sure that this Pi-Hole is running within Docker 😆 But your really nice script is detecting this wrong/not.

Also the update-detection is not correct - or version of 2.6 is set wrong and should be 2.6.0!?

Please add a licence

There doesn't seem to be a licence attached to this repo. In this state no one would be allowed to publish a derivative work, or even copy the code for a personal project. This might be an intentional, but usually it's an oversight. Quick read up on licenses here.

pihole_adlist_tool .. start probleme ^^

Bin also wegen der Installation Routine hier ..

Und frage mich was soll passieren nachdem ich starten will..

pi@ras:~ $ chmod +x ./pihole_adlist_tool
pi@ras:~ $
pi@ras:~ $ ./pihole_adlist_tool
-bash: ./pihole_adlist_tool: Is a directory
pi@ras:~ $ /pihole_adlist_tool
-bash: /pihole_adlist_tool: No such file or directory

https://ibb.co/wRzCwzz

Dnsmasq 2.84

@yubiuser: Can you update to 2.84?

http://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2021q1/014640.html

There was a bug in Dnsmasq 2.83.

Changelog:

http://www.thekelleys.org.uk/dnsmasq/CHANGELOG

Installation

How do you install/run this?

würde gerne das Script nutzen brauchen aber hilfe.

Hi, ich würde gerne das pihole adlist tool nutzen /ausprobieren. habe allergings keinen plan wo ich da anfangen soll/wie das geht.

Does regex checking account for Filter Lists?

When using the regex checking feature, does the list of blocked domains include the fact that they might also be blocked by a filter list, or is the regex checker independent of the filter list checker?

I am trying to determine if I need the regex configurations that I have applied.

Can't read /etc/pihole/pihole-FTL.db

When pihole_adlist_tool I get the following output:

~/pihole_adlist_tool (master) --ϐ ./pihole_adlist_tool                                                                                   


  *** Pihole Adlist Tool 2.6.3 ***


   ++++++++ Info ++++++++

  [i]  PIHOLE_DOCKER: No
  [i]  PIHOLE_DNSMASQ_VERSION: v2.87
  [i]  SQLITE_VERSION: 3.39.2
  [i]  DAYS_REQUESTED: 30
  [i]  TOP: Not shown
  [i]  SORT_ORDER: id ASC
  [i]  UNIQUE: Not shown
  [i]  REGEX_MODE: Disabled

   ++++++++++++++++++++++



  [i]  Warning: You requested to analyze the last 30 day(s) (starting from 14.09.2022	19:56:48),
       but last query is from 31.12.1969	17:00:00
  [i]  Nothing to do here. Exiting

Running

pihole-FTL sqlite3 "file:/etc/pihole/pihole-FTL.db" "SELECT MIN(timestamp) FROM queries;"

pihole-FTL sqlite3 "file:/etc/pihole/pihole-FTL.db" "SELECT MAX(timestamp) FROM queries;"

by itself returns nothing, so I believe that there's an issue with the command getting data from the db.

Output of pihole_adlist_tool -v

~/pihole_adlist_tool (master) --ϐ ./pihole_adlist_tool -v                                                                                
Pihole Adlist Tool v2.6.3

Docker compability

The newest version doesn't work with docker anymore...

Can you add docker support again? Thank you!

Regex Mode does not work in docker

When running the command
./pihole_adlist_tool -d 0 -r
It gets into an infinite loop (even starting to heat up the raspberry pi 4) saying the same error over and over again:
./pihole_adlist_tool: line 987: pihole-FTL: command not found

Full logs are attached:
pihole_adlist_tool_docker_error.txt

DB Access denied

~/pihole_adlist_tool$ sudo ./pihole_adlist_tool

*** Pihole Adlist Tool ***

++++++++ Info ++++++++

[i] DAYS_REQUESTED: 30
[i] TOP: Not shown
[i] SORT_ORDER: id ASC
[i] UNIQUE: Not shown
[i] NEW_ADLIST_FILENAME_SCHEMA: no
Error: unable to open database "file:/etc/pihole/pihole-FTL.db?mode=ro": unable to open database file
[i] CNAME_AVAILABLE: deep CNAME info not available
[i] SQLITE_VERSION: 3.31.1

++++++++++++++++++++++

Error: unable to open database "file:/etc/pihole/pihole-FTL.db?mode=ro": unable to open database file
Error: unable to open database "file:/etc/pihole/pihole-FTL.db?mode=ro": unable to open database file
Error: unable to open database "file:/etc/pihole/pihole-FTL.db?mode=ro": unable to open database file
Error: unable to open database "file:/etc/pihole/pihole-FTL.db?mode=ro": unable to open database file

[i] Warning: You requested to analyze the last 30 day(s) (starting from 28.11.2020 3:11:16),
but last query is from 01.01.1970 0:00:00
[i] Nothing to do here. Exiting

Strange analysis result

I am using the tool to optimize my adlists. Currently I am adding lists from https://github.com/blocklistproject, but the analysis results are not plausible and not stable.

initial run

$ pihole -g && ./pihole_adlist_tool 
(...)
id          enabled     total_domains  domains_covered  hits_covered  unique_domains_covered  address                                                         
----------  ----------  -------------  ---------------  ------------  ----------------------  ----------------------------------------------------------------
1           0                                                                                 https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
2           0                                                                                 https://mirror1.malwaredomains.com/files/justdomains            
3           1           20567          108              27820                                 http://sysctl.org/cameleon/hosts                                
5           0                                                                                 https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
6           0                                                                                 https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt      
8           1           23660          16               32            16                      https://raw.githubusercontent.com/kboghdady/youTube_ads_4_pi-hol
9           1           156039         468              66207         309                     https://blocklistproject.github.io/Lists/ads.txt                
10          1           1220                                                                  https://blocklistproject.github.io/Lists/abuse.txt

disable http://sysctl.org/cameleon/hosts

$ pihole -g && ./pihole_adlist_tool 
(...)
id          enabled     total_domains  domains_covered  hits_covered  unique_domains_covered  address                                                         
----------  ----------  -------------  ---------------  ------------  ----------------------  ----------------------------------------------------------------
1           0                                                                                 https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
2           0                                                                                 https://mirror1.malwaredomains.com/files/justdomains            
3           0                                                                                 http://sysctl.org/cameleon/hosts                                
5           0                                                                                 https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
6           0                                                                                 https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt      
8           1           23660          16               32            16                      https://raw.githubusercontent.com/kboghdady/youTube_ads_4_pi-hol
9           1           156039         468              66212                                 https://blocklistproject.github.io/Lists/ads.txt                
10          1           156039         468              66212                                 https://blocklistproject.github.io/Lists/abuse.txt

-> Unexpectedly, the number of total domains for https://blocklistproject.github.io/Lists/abuse.txt has increased from 1220 to 156039! And why do ads.txt and abuse.txt show identical numbers?

Add assigned group(s) for adlist in result

It would really be nice to have the assigned group for each adlist also in the resulting table in addition to "id, enabled, total_domains, domains_covered, hits_covered, unique_domains_covered, address" which should also be sortable:

# sqlite3 ./gravity.db "select * from 'group'"
0|1|Default|1604414479|1614120328|The default group
22|1|adult-content|1614119689|1614120454|
23|1|test-group|1637245963|1637245963|for testing before going in production

# sqlite3 ./gravity.db "select * from adlist_by_group order by 1"
1|0
5|0
13|0
19|0
20|0
21|0
23|0
27|0
28|0
30|0
32|0
33|0
34|0
35|0
37|0
42|22
43|0
59|0
60|0
62|0
63|0
64|0
67|0
68|0
69|0
71|22
72|22
73|22
74|0
77|22
80|22
81|22
82|22
82|23
83|0
84|0
85|0
87|0
88|22
89|0
90|0

If I find some time, I will make a PR and tell you when I'm starting before you're also trying to add this enhancement.

Possibility to Deactivate redundant Blocklists

Hello,

is there a possibility to show and/or deaktivate blocklists that are redundant, so that the unique domains on the first page does not change and only the unnecessary blocklists with their domains are deactivated. Thanks in advance.

Receiving an error "database disk image is malformed"

I recently installed this tool to check the efficiency of the adlists I have on my PiHole. When I ran the tool for the first time, this is the output:

*** Pihole Adlist Tool ***

++++++++ Info ++++++++

[i] PIHOLE_DNSMASQ_VERSION: 2.84
[i] SQLITE_VERSION: 3.34.0
[i] DAYS_REQUESTED: 30
[i] TOP: Not shown
[i] SORT_ORDER: id ASC
[i] UNIQUE: Not shown
[i] REGEX_MODE: Disabled

++++++++++++++++++++++

Error: database disk image is malformed
Would you like to analyze your current adlist configuration or first enable all adlists (current can be restored later)?

Current adlist configuration
Enable all adlists (runs pihole -g)

Please select: 1

[i] Keeping current adlist configuration

[i] There is a mismatch between your enabled adlists and the data found in the gravity database.
You have 47 adlists enabled, but data from 46 (patially different) adlists in your gravity database.
You're likely disabled/enabled adlist without running gravity afterwards.
It's highly recommended to run gravity now to solve the differences, otherwise this tool will analyze the available data.

Would you like to run gravity now?

Please select: 2

[i] Not running gravity, keeping mismatch between enabled adlists and the data found in the gravity database.

[i] Calculating.....
[i] This might take some time - please be patient.
Error: near line 4: database disk image is malformed
Error: near line 14: database disk image is malformed

Use extended adlist info from gravity database

Note to myself:
Table adlist is updated with total number of domains for each adlist_id .
Since commit 73963fecda6dc65b10d1dd3e43a5931dc531304a to pihole's core, locally saved adlist copies contain the adlist_id in the filename. We use that to count the lines in each file and use the adlist_id to attribute it to the corresponding adlist in TEMP_DB This is faster than to count the domains for each adlist from gravity_db.

But: in a next Pi-hole version, Pi-hole hopefully will store the number of domains for each adlist as additional info. This can then be used. This saves time (no calculation has to be done again) and makes the script Independent of the locally saved adlist copies.

pi-hole/pi-hole#3951

Question

HI
Please can you tell me how i install and run this?

Parse error near line 14: no such column: additional_info

I downloaded 2.6.4, but it contains 2.6.3

And it error-ed out as below:

$./pihole_adlist_tool -d 0 -s total

*** Pihole Adlist Tool 2.6.3 ***

++++++++ Info ++++++++
[i] PIHOLE_DOCKER: No
[i] PIHOLE_DNSMASQ_VERSION: v2.89-9461807
[i] SQLITE_VERSION: 3.42.0
[i] DAYS_REQUESTED: all time
[i] TOP: Not shown
[i] SORT_ORDER: total_domains DESC
[i] UNIQUE: Not shown
[i] REGEX_MODE: Disabled
[i] There is an update available: 2.6.4
++++++++++++++++++++++

Would you like to analyze your current adlist configuration or first enable all adlists (current can be restored later)?

Current adlist configuration
Enable all adlists (runs pihole -g)

Please select: 1

[i] Keeping current adlist configuration

[i] Calculating.....
[i] This might take some time - please be patient.
Parse error near line 14: no such column: additional_info
INSERT INTO cname(additional_info, hits) SELECT additional_info, COUNT(domain)
error here ---^

Shellcheck is complaining a lot about pihole_adlist_tool

Missing shellcheck compatibility is no fun because there are some severe and critical errors - and this breaks my pipeline because I copied your script in my bin directory which is shellcheck'ed on every commit:

("passed" now with my changes from PR #37 )

First round:

shellcheck pihole_adlist_tool
pihole_adlist_tool: <stdout>: commitBuffer: invalid argument (invalid character)

In pihole_adlist_tool line 87:
	local remote_version="$(
              ^------------^ SC2155: Declare and assign separately to avoid masking return values.


In pihole_adlist_tool line 125:
    local DATE=$(date -d @$1 +%d.%m.%Y%t%k:%M:%S)
          ^--^ SC2155: Declare and assign separately to avoid masking return values.
                          ^-- SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    local DATE=$(date -d @"$1" +%d.%m.%Y%t%k:%M:%S)


In pihole_adlist_tool line 134:
    TIMESTAMP_FIRST_QUERY=$(sqlite $PIHOLE_FTL "SELECT MIN(timestamp) FROM queries;")
                                   ^---------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    TIMESTAMP_FIRST_QUERY=$(sqlite "$PIHOLE_FTL" "SELECT MIN(timestamp) FROM queries;")


In pihole_adlist_tool line 135:
    TIMESTAMP_LAST_QUERY=$(sqlite $PIHOLE_FTL "SELECT MAX(timestamp) FROM queries;")
                                  ^---------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    TIMESTAMP_LAST_QUERY=$(sqlite "$PIHOLE_FTL" "SELECT MAX(timestamp) FROM queries;")


In pihole_adlist_tool line 141:
            TIMESTAMP_REQUESTED=$TIMESTAMP_REQUESTED-86400*${DAYS_REQUESTED}
                                ^-- SC2125: Brace expansions and globs are literal in assignments. Quote it or use an array.


In pihole_adlist_tool line 143:
    TIMESTAMP_FIRST_ANALYZED=$(sqlite $PIHOLE_FTL "SELECT min(timestamp) FROM queries WHERE timestamp>=$TIMESTAMP_REQUESTED;")
                                      ^---------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    TIMESTAMP_FIRST_ANALYZED=$(sqlite "$PIHOLE_FTL" "SELECT min(timestamp) FROM queries WHERE timestamp>=$TIMESTAMP_REQUESTED;")


In pihole_adlist_tool line 149:
    DATE_FIRST_QUERY="$(timestamp2date $TIMESTAMP_FIRST_QUERY)"
                                       ^--------------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    DATE_FIRST_QUERY="$(timestamp2date "$TIMESTAMP_FIRST_QUERY")"


In pihole_adlist_tool line 150:
    DATE_LAST_QUERY="$(timestamp2date $TIMESTAMP_LAST_QUERY)"
                                      ^-------------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    DATE_LAST_QUERY="$(timestamp2date "$TIMESTAMP_LAST_QUERY")"


In pihole_adlist_tool line 151:
    DATE_REQUESTED="$(timestamp2date $TIMESTAMP_REQUESTED)"
                                     ^------------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    DATE_REQUESTED="$(timestamp2date "$TIMESTAMP_REQUESTED")"


In pihole_adlist_tool line 152:
    DATE_FIRST_ANALYZED="$(timestamp2date $TIMESTAMP_FIRST_ANALYZED)"
                                          ^-----------------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    DATE_FIRST_ANALYZED="$(timestamp2date "$TIMESTAMP_FIRST_ANALYZED")"


In pihole_adlist_tool line 171:
echo "  [For more information:
  https://www.shellcheck.net/wiki/SC2068 -- Double quote array expansions to ...
  https://www.shellcheck.net/wiki/SC2145 -- Argument mixes string and array. ...
  https://www.shellcheck.net/wiki/SC2027 -- The surrounding quotes actually u...

After removing "invalid character"(s):

In pihole_adlist_tool line 87:
	local remote_version="$(
              ^------------^ SC2155: Declare and assign separately to avoid masking return values.


In pihole_adlist_tool line 125:
    local DATE=$(date -d @$1 +%d.%m.%Y%t%k:%M:%S)
          ^--^ SC2155: Declare and assign separately to avoid masking return values.
                          ^-- SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    local DATE=$(date -d @"$1" +%d.%m.%Y%t%k:%M:%S)


In pihole_adlist_tool line 134:
    TIMESTAMP_FIRST_QUERY=$(sqlite $PIHOLE_FTL "SELECT MIN(timestamp) FROM queries;")
                                   ^---------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    TIMESTAMP_FIRST_QUERY=$(sqlite "$PIHOLE_FTL" "SELECT MIN(timestamp) FROM queries;")


In pihole_adlist_tool line 135:
    TIMESTAMP_LAST_QUERY=$(sqlite $PIHOLE_FTL "SELECT MAX(timestamp) FROM queries;")
                                  ^---------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    TIMESTAMP_LAST_QUERY=$(sqlite "$PIHOLE_FTL" "SELECT MAX(timestamp) FROM queries;")


In pihole_adlist_tool line 141:
            TIMESTAMP_REQUESTED=$TIMESTAMP_REQUESTED-86400*${DAYS_REQUESTED}
                                ^-- SC2125: Brace expansions and globs are literal in assignments. Quote it or use an array.


In pihole_adlist_tool line 143:
    TIMESTAMP_FIRST_ANALYZED=$(sqlite $PIHOLE_FTL "SELECT min(timestamp) FROM queries WHERE timestamp>=$TIMESTAMP_REQUESTED;")
                                      ^---------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    TIMESTAMP_FIRST_ANALYZED=$(sqlite "$PIHOLE_FTL" "SELECT min(timestamp) FROM queries WHERE timestamp>=$TIMESTAMP_REQUESTED;")


In pihole_adlist_tool line 149:
    DATE_FIRST_QUERY="$(timestamp2date $TIMESTAMP_FIRST_QUERY)"
                                       ^--------------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    DATE_FIRST_QUERY="$(timestamp2date "$TIMESTAMP_FIRST_QUERY")"


In pihole_adlist_tool line 150:
    DATE_LAST_QUERY="$(timestamp2date $TIMESTAMP_LAST_QUERY)"
                                      ^-------------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    DATE_LAST_QUERY="$(timestamp2date "$TIMESTAMP_LAST_QUERY")"


In pihole_adlist_tool line 151:
    DATE_REQUESTED="$(timestamp2date $TIMESTAMP_REQUESTED)"
                                     ^------------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    DATE_REQUESTED="$(timestamp2date "$TIMESTAMP_REQUESTED")"


In pihole_adlist_tool line 152:
    DATE_FIRST_ANALYZED="$(timestamp2date $TIMESTAMP_FIRST_ANALYZED)"
                                          ^-----------------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    DATE_FIRST_ANALYZED="$(timestamp2date "$TIMESTAMP_FIRST_ANALYZED")"


In pihole_adlist_tool line 171:
echo "  [i]  Of the ${bold}"$NUM_DOMAINS_BLOCKED_CURRENT"${normal} domains that would have been blocked by the adlist configuration you chose to analyze,"
                            ^--------------------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                            ^--------------------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
echo "  [i]  Of the ${bold}""$NUM_DOMAINS_BLOCKED_CURRENT""${normal} domains that would have been blocked by the adlist configuration you chose to analyze,"


In pihole_adlist_tool line 172:
echo "       ${bold}"$NUM_DOMAINS_BLOCKED_FUTURE"${normal} domains are covered by your current adlist configuration."
                     ^-------------------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                     ^-------------------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
echo "       ${bold}""$NUM_DOMAINS_BLOCKED_FUTURE""${normal} domains are covered by your current adlist configuration."


In pihole_adlist_tool line 184:
        docker exec $CONTAINER_ID rm -f $TEMP_DB
                    ^-----------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
        docker exec "$CONTAINER_ID" rm -f $TEMP_DB


In pihole_adlist_tool line 205:
        docker exec -i $CONTAINER_ID pihole-FTL sqlite3 "$@"
                       ^-----------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
        docker exec -i "$CONTAINER_ID" pihole-FTL sqlite3 "$@"


In pihole_adlist_tool line 260:
        PIHOLE_DNSMASQ_VERSION=$(docker exec $CONTAINER_ID pihole-FTL -vv |awk '/dnsmasq/{getline; print substr($2,9)}')
                                             ^-----------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
        PIHOLE_DNSMASQ_VERSION=$(docker exec "$CONTAINER_ID" pihole-FTL -vv |awk '/dnsmasq/{getline; print substr($2,9)}')


In pihole_adlist_tool line 305:
        SQLITE_VERSION=$(docker exec $CONTAINER_ID pihole-FTL sqlite3 --version|awk '{print $1}')
                                     ^-----------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
        SQLITE_VERSION=$(docker exec "$CONTAINER_ID" pihole-FTL sqlite3 --version|awk '{print $1}')


In pihole_adlist_tool line 360:
if [ -n "$remote_version" -a "$remote_version" != "$PIHOLE_ADLIST_TOOL_VERSION" ]; then
                          ^-- SC2166: Prefer [ p ] && [ q ] as [ p -a q ] is not well defined.


In pihole_adlist_tool line 386:
FTL_ID=$(sqlite $PIHOLE_FTL "SELECT MIN(id) FROM queries WHERE timestamp>=$TIMESTAMP_REQUESTED;")
                ^---------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
FTL_ID=$(sqlite "$PIHOLE_FTL" "SELECT MIN(id) FROM queries WHERE timestamp>=$TIMESTAMP_REQUESTED;")


In pihole_adlist_tool line 394:
        echo "  [i]  ${bold}Warning:${normal} You requested to analyze the last "${DAYS_REQUESTED}" day(s) (starting from "$DATE_REQUESTED"),"
                                                                                 ^---------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                                                 ^---------------^ SC2086: Double quote to prevent globbing and word splitting.
                                                                                                                           ^-------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                                                                                           ^-------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
        echo "  [i]  ${bold}Warning:${normal} You requested to analyze the last ""${DAYS_REQUESTED}"" day(s) (starting from ""$DATE_REQUESTED""),"


In pihole_adlist_tool line 395:
        echo "       but last query is from "$DATE_LAST_QUERY""
                                             ^--------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                             ^--------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
        echo "       but last query is from ""$DATE_LAST_QUERY"""


In pihole_adlist_tool line 404:
        echo -e "  [i]  ${bold}Warning:${normal} You requested to analyze the last "${DAYS_REQUESTED}" days (starting from "$DATE_REQUESTED"),"
                                                                                    ^---------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                                                    ^---------------^ SC2086: Double quote to prevent globbing and word splitting.
                                                                                                                            ^-------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                                                                                            ^-------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
        echo -e "  [i]  ${bold}Warning:${normal} You requested to analyze the last ""${DAYS_REQUESTED}"" days (starting from ""$DATE_REQUESTED""),"


In pihole_adlist_tool line 405:
        echo -e "       but oldest query is from "$DATE_FIRST_QUERY". Using this instead\n"
                                                  ^---------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                  ^---------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
        echo -e "       but oldest query is from ""$DATE_FIRST_QUERY"". Using this instead\n"


In pihole_adlist_tool line 411:
adlist_conf_old_enabled=(`sqlite $GRAVITY "select id from adlist where enabled=1;"`)
                         ^-- SC2207: Prefer mapfile or read -a to split command output (or quote to avoid splitting).
                         ^-- SC2006: Use $(...) notation instead of legacy backticked `...`.

Did you mean:
adlist_conf_old_enabled=($(sqlite $GRAVITY "select id from adlist where enabled=1;"))


In pihole_adlist_tool line 426:
  read -p "  Please select: " ENABLE_ALL_ADLISTS_FOR_ANALYSIS
  ^--^ SC2162: read without -r will mangle backslashes.


In pihole_adlist_tool line 441:
    adlist_enabled_in_gravity=(`sqlite $GRAVITY "select distinct adlist_id from gravity;"`)
                               ^-- SC2207: Prefer mapfile or read -a to split command output (or quote to avoid splitting).
                               ^-- SC2006: Use $(...) notation instead of legacy backticked `...`.

Did you mean:
    adlist_enabled_in_gravity=($(sqlite $GRAVITY "select distinct adlist_id from gravity;"))


In pihole_adlist_tool line 442:
    if [ -n "$(echo ${adlist_conf_old_enabled[@]} ${adlist_enabled_in_gravity[@]} | tr ' ' '\n' |sort |uniq -u)" ]; then
                    ^---------------------------^ SC2068: Double quote array expansions to avoid re-splitting elements.
                                                  ^-----------------------------^ SC2068: Double quote array expansions to avoid re-splitting elements.


In pihole_adlist_tool line 446:
        echo "       You have ${bold}"${#adlist_conf_old_enabled[@]}"${normal} adlists enabled, but data from ${bold}"${#adlist_enabled_in_gravity[@]}"${normal} (patially different) adlists in your gravity database."
                                      ^----------------------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                                                                                      ^-- SC2027: The surrounding quotes actually unquote this. Remove or escape them.


In pihole_adlist_tool line 463:
            read -p "  Please select: " RUN_GRAVITY_NOW
            ^--^ SC2162: read without -r will mangle backslashes.


In pihole_adlist_tool line 474:
                        docker exec $CONTAINER_ID pihole -g
                                    ^-----------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
                        docker exec "$CONTAINER_ID" pihole -g


In pihole_adlist_tool line 513:
           docker exec $CONTAINER_ID pihole -g
                       ^-----------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
           docker exec "$CONTAINER_ID" pihole -g


In pihole_adlist_tool line 637:
read NUM_DOMAINS_BLOCKED HITS_TOTAL <<<$(sqlite -separator " " $PIHOLE_FTL "SELECT COUNT(DISTINCT domain),count(domain) FROM queries WHERE id>=${FTL_ID} AND status in (1,9);")
^--^ SC2162: read without -r will mangle backslashes.
                                       ^-- SC2046: Quote this to prevent word splitting.
                                                               ^---------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
read NUM_DOMAINS_BLOCKED HITS_TOTAL <<<$(sqlite -separator " " "$PIHOLE_FTL" "SELECT COUNT(DISTINCT domain),count(domain) FROM queries WHERE id>=${FTL_ID} AND status in (1,9);")


In pihole_adlist_tool line 650:
echo "  [i]  You have ${bold}"$NUM_ADLISTS" adlists${normal} configured ("$NUM_ADLISTS_ENABLED" enabled)"
                              ^----------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                              ^----------^ SC2086: Double quote to prevent globbing and word splitting.
                                                                          ^------------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                                          ^------------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
echo "  [i]  You have ${bold}""$NUM_ADLISTS"" adlists${normal} configured (""$NUM_ADLISTS_ENABLED"" enabled)"


In pihole_adlist_tool line 651:
echo "  [i]  Your gravity.db contains ${bold}"$NUM_GRAVITY_UNIQUE_DOMAINS" unique domains${normal}"
                                              ^-------------------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                              ^-------------------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
echo "  [i]  Your gravity.db contains ${bold}""$NUM_GRAVITY_UNIQUE_DOMAINS"" unique domains${normal}"


In pihole_adlist_tool line 652:
echo "  [i]  Since "$DATE_FIRST_ANALYZED" ${bold}"$NUM_DOMAINS_BLOCKED" different domains${normal} from your adlists have been blocked ${bold}"$HITS_TOTAL" times${normal} in total"
                    ^------------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                    ^------------------^ SC2086: Double quote to prevent globbing and word splitting.
                                                  ^------------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                  ^------------------^ SC2086: Double quote to prevent globbing and word splitting.
                                                                                                                                               ^---------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                                                                                                               ^---------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
echo "  [i]  Since ""$DATE_FIRST_ANALYZED"" ${bold}""$NUM_DOMAINS_BLOCKED"" different domains${normal} from your adlists have been blocked ${bold}""$HITS_TOTAL"" times${normal} in total"


In pihole_adlist_tool line 654:
echo "  [i]  Using you current adlist configuration ${bold}"$NUM_DOMAINS_BLOCKED_CURRENT" domains${normal} would have been blocked ${bold}"$HITS_TOTAL_CURRENT" times${normal}"
                                                            ^--------------------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                            ^--------------------------^ SC2086: Double quote to prevent globbing and word splitting.
                                                                                                                                           ^-----------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                                                                                                           ^-----------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
echo "  [i]  Using you current adlist configuration ${bold}""$NUM_DOMAINS_BLOCKED_CURRENT"" domains${normal} would have been blocked ${bold}""$HITS_TOTAL_CURRENT"" times${normal}"


In pihole_adlist_tool line 706:
        echo "       Those would have been the ${bold}"$TOP" top blocked adlist domains${normal} since "$DATE_FIRST_ANALYZED""
                                                       ^--^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                       ^--^ SC2086: Double quote to prevent globbing and word splitting.
                                                                                                        ^------------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                                                                        ^------------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
        echo "       Those would have been the ${bold}""$TOP"" top blocked adlist domains${normal} since ""$DATE_FIRST_ANALYZED"""


In pihole_adlist_tool line 709:
        sqlite -column -header $TEMP_DB "SELECT domain, hits FROM blocked_domains LIMIT "${TOP}";"
                                                                                         ^----^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                                                         ^----^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
        sqlite -column -header $TEMP_DB "SELECT domain, hits FROM blocked_domains LIMIT ""${TOP}"";"


In pihole_adlist_tool line 735:
echo "  [i]  In total your adlists contain ${bold}"$NUM_TOTAL_UNIQUE_DOMAINS" visited (covered) unique domains${normal} - meaning those domains are contained only in a single adlist. "
                                                   ^-----------------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                   ^-----------------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
echo "  [i]  In total your adlists contain ${bold}""$NUM_TOTAL_UNIQUE_DOMAINS"" visited (covered) unique domains${normal} - meaning those domains are contained only in a single adlist. "


In pihole_adlist_tool line 756:
          read -p "  Please select: " FURTHER_ACTION
          ^--^ SC2162: read without -r will mangle backslashes.


In pihole_adlist_tool line 773:
          read -p "  Please select: " FURTHER_ACTION
          ^--^ SC2162: read without -r will mangle backslashes.


In pihole_adlist_tool line 792:
    adlist_conf_unique_enabled=(`sqlite $TEMP_DB "select id from adlist where unique_domains_covered IS NOT NULL;"`)
                                ^-- SC2207: Prefer mapfile or read -a to split command output (or quote to avoid splitting).
                                ^-- SC2006: Use $(...) notation instead of legacy backticked `...`.

Did you mean:
    adlist_conf_unique_enabled=($(sqlite $TEMP_DB "select id from adlist where unique_domains_covered IS NOT NULL;"))


In pihole_adlist_tool line 807:
            docker exec $CONTAINER_ID pihole restartdns reload-lists
                        ^-----------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
            docker exec "$CONTAINER_ID" pihole restartdns reload-lists


In pihole_adlist_tool line 836:
    adlist_conf_minimal_enabled=(`sqlite $TEMP_DB "select id from adlist where unique_domains_covered IS NOT NULL;"`)
                                 ^-- SC2207: Prefer mapfile or read -a to split command output (or quote to avoid splitting).
                                 ^-- SC2006: Use $(...) notation instead of legacy backticked `...`.

Did you mean:
    adlist_conf_minimal_enabled=($(sqlite $TEMP_DB "select id from adlist where unique_domains_covered IS NOT NULL;"))


In pihole_adlist_tool line 844:
    LEFT_DOMAINS=(`sqlite $TEMP_DB "SELECT COUNT (domain) from gravity_dup;"`)
                  ^-- SC2207: Prefer mapfile or read -a to split command output (or quote to avoid splitting).
                  ^-- SC2006: Use $(...) notation instead of legacy backticked `...`.

Did you mean:
    LEFT_DOMAINS=($(sqlite $TEMP_DB "SELECT COUNT (domain) from gravity_dup;"))


In pihole_adlist_tool line 846:
    while [[ $LEFT_DOMAINS != [0] ]]; do
             ^-----------^ SC2128: Expanding an array without an index only gives the first element.


In pihole_adlist_tool line 847:
        current_id=(`sqlite $TEMP_DB "Select adlist_id from gravity_dup group by adlist_id order by count (domain) desc, adlist_id asc limit 1;"`);
                    ^-- SC2207: Prefer mapfile or read -a to split command output (or quote to avoid splitting).
                    ^-- SC2006: Use $(...) notation instead of legacy backticked `...`.

Did you mean:
        current_id=($(sqlite $TEMP_DB "Select adlist_id from gravity_dup group by adlist_id order by count (domain) desc, adlist_id asc limit 1;"));


In pihole_adlist_tool line 849:
        adlist_conf_minimal_enabled[${#adlist_conf_minimal_enabled[@]}]="$current_id"
                                                                         ^---------^ SC2128: Expanding an array without an index only gives the first element.


In pihole_adlist_tool line 850:
        sqlite $TEMP_DB "DELETE FROM gravity_dup WHERE domain IN (SELECT domain from gravity_dup where adlist_id=$current_id);"
                                                                                                                 ^---------^ SC2128: Expanding an array without an index only gives the first element.


In pihole_adlist_tool line 851:
        LEFT_DOMAINS=(`sqlite $TEMP_DB "SELECT COUNT (domain) from gravity_dup;"`)
                      ^-- SC2207: Prefer mapfile or read -a to split command output (or quote to avoid splitting).
                      ^-- SC2006: Use $(...) notation instead of legacy backticked `...`.

Did you mean:
        LEFT_DOMAINS=($(sqlite $TEMP_DB "SELECT COUNT (domain) from gravity_dup;"))


In pihole_adlist_tool line 855:
    echo "  [i]  Enabling adlists with id ${adlist_conf_minimal_enabled[@]}"
                                          ^-- SC2145: Argument mixes string and array. Use * or separate argument.


In pihole_adlist_tool line 871:
            docker exec $CONTAINER_ID pihole restartdns reload-lists
                        ^-----------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
            docker exec "$CONTAINER_ID" pihole restartdns reload-lists


In pihole_adlist_tool line 908:
            docker exec $CONTAINER_ID pihole restartdns reload-lists
                        ^-----------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
            docker exec "$CONTAINER_ID" pihole restartdns reload-lists


In pihole_adlist_tool line 984:
        all_domains=(`sqlite $TEMP_DB "SELECT domain FROM all_domains"`)
                     ^-- SC2207: Prefer mapfile or read -a to split command output (or quote to avoid splitting).
                     ^-- SC2006: Use $(...) notation instead of legacy backticked `...`.

Did you mean:
        all_domains=($(sqlite $TEMP_DB "SELECT domain FROM all_domains"))


In pihole_adlist_tool line 987:
                pihole-FTL regex-test $CURRENT_DOMAIN |grep -E -o "DB ID [0-9]*"|awk '{print $3}' | while read REGEX_ID; do
                                      ^-------------^ SC2086: Double quote to prevent globbing and word splitting.
                                                                                                          ^--^ SC2162: read without -r will mangle backslashes.

Did you mean:
                pihole-FTL regex-test "$CURRENT_DOMAIN" |grep -E -o "DB ID [0-9]*"|awk '{print $3}' | while read REGEX_ID; do


In pihole_adlist_tool line 1003:
        all_exact_domains=(`sqlite $GRAVITY "SELECT domain FROM domainlist WHERE type in (0,1);"`)
                           ^-- SC2207: Prefer mapfile or read -a to split command output (or quote to avoid splitting).
                           ^-- SC2006: Use $(...) notation instead of legacy backticked `...`.

Did you mean:
        all_exact_domains=($(sqlite $GRAVITY "SELECT domain FROM domainlist WHERE type in (0,1);"))


In pihole_adlist_tool line 1006:
          pihole-FTL regex-test $CURRENT_DOMAIN |grep -E -o "DB ID [0-9]*"|awk '{print $3}' | while read REGEX_ID; do
                                ^-------------^ SC2086: Double quote to prevent globbing and word splitting.
                                                                                                    ^--^ SC2162: read without -r will mangle backslashes.

Did you mean:
          pihole-FTL regex-test "$CURRENT_DOMAIN" |grep -E -o "DB ID [0-9]*"|awk '{print $3}' | while read REGEX_ID; do


In pihole_adlist_tool line 1036:
        echo "  [i]  Since "$DATE_FIRST_ANALYZED" you have been visiting ${bold}"$NUM_ALL_DOMAINS" different domains${normal}."
                            ^------------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                            ^------------------^ SC2086: Double quote to prevent globbing and word splitting.
                                                                                 ^--------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                                                 ^--------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
        echo "  [i]  Since ""$DATE_FIRST_ANALYZED"" you have been visiting ${bold}""$NUM_ALL_DOMAINS"" different domains${normal}."


In pihole_adlist_tool line 1037:
        echo "       You have ${bold}"$NUM_REGEX" blacklist RegEx${normal} configured ("$NUM_ENABLED_REGEX" enabled)"
                                      ^--------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                      ^--------^ SC2086: Double quote to prevent globbing and word splitting.
                                                                                        ^----------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                                                        ^----------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
        echo "       You have ${bold}""$NUM_REGEX"" blacklist RegEx${normal} configured (""$NUM_ENABLED_REGEX"" enabled)"


In pihole_adlist_tool line 1038:
        echo "       With your enabled blacklist RegEx you would have covered ${bold}"$NUM_DOMAINS_BLOCKED_BY_REGEX" different domains${normal}."
                                                                                      ^---------------------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                                                      ^---------------------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
        echo "       With your enabled blacklist RegEx you would have covered ${bold}""$NUM_DOMAINS_BLOCKED_BY_REGEX"" different domains${normal}."


In pihole_adlist_tool line 1055:
            echo "       ${bold}"$NUM_DOMAINLIST_REGEX" distinct domain(s)${normal} from your domainlist are covered by your RegEx."
                                 ^-------------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                 ^-------------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
            echo "       ${bold}""$NUM_DOMAINLIST_REGEX"" distinct domain(s)${normal} from your domainlist are covered by your RegEx."

For more information:
  https://www.shellcheck.net/wiki/SC2068 -- Double quote array expansions to ...
  https://www.shellcheck.net/wiki/SC2145 -- Argument mixes string and array. ...
  https://www.shellcheck.net/wiki/SC2027 -- The surrounding quotes actually u...

If I find the time, I will shellcheck this and open a PR… But if another person has the time and fun to shellcheck this - let's go, and leave an update here! 👍🏻

Regex analysis not supported on docker

ABP style adlists

Report currently does not show the number for ABP style adlists.
Please consider include calculation for ABP style adlists.

Thank you

  [i]  Adlist coverage

id   enabled  total_domains  domains_covered  hits_covered  unique_domains_covered  address
---  -------  -------------  ---------------  ------------  ----------------------  ---------------------------------------------------------------------------------
413  1        0                                                                     https://big.oisd.nl/
469  1        0                                                                     https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/multi.txt
470  1        0                                                                     https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/tif.txt

Error: stepping, database is locked

Any ideas?


  [i]  Domains from disabled adlists are not stored in gravity's database.
       If you want to include them in the analysis, run this script again and select 'Enable all adlists'
       As the same domains usually appears on more than one adlist the sum of covered domains from this table is greater 
       than the number of calculated blocked domains shown above


  [i]  In total your adlists contain 190 visited (covered) unique domains - meaning those domains are contained only in a single adlist. 


  Would you like to ... 

  1)  Keep your current adlist configuration
  2)  Enable only adlists with covered unique domains
  3)  Enable the minimal number of adlists that cover all domains that would have been blocked

  Please select: 3

  [i]  Enabling minimum number of adlists that cover all domains that would have been blocked....
Error: stepping, database is locked (5)

  [i]  Enabling adlists with id 56 58 60 66 67 74 81 82 83 85 90 102 104 108
Error: stepping, database is locked (5)
Error: stepping, database is locked (5)
Error: stepping, database is locked (5)
Error: stepping, database is locked (5)
Error: stepping, database is locked (5)
Error: stepping, database is locked (5)
Error: stepping, database is locked (5)
Error: stepping, database is locked (5)
Error: stepping, database is locked (5)
Error: stepping, database is locked (5)
Error: stepping, database is locked (5)
Error: stepping, database is locked (5)
Error: stepping, database is locked (5)
Error: stepping, database is locked (5)
  [✓] Reloading DNS lists

  [✓]  Minimal number of adlists that cover all domains (that would have been blocked) enabled


  [i]  Of the 704 domains that would have been blocked by the adlist configuration you chose to analyze,
       704 domains are covered by your current adlist configuration.

  [i]  Removing temporary database...
  [✓]  Temporary database removed

[Question] If a domain exists in >=2 adlists, each of which have no unique domains, will the domain no longer be blocked (if told to remove adlists without unique ones)?

The question in the title may be phrased a little oddly, but I'll try to explain it better here. Take the following example scenario:

I have 5 adlists. For simplicity's sake, I'll just reference them by their id 1-5.

Let's use the following data for these 5 adlists (the specific numbers don't really matter, just simply whether a domain/unique domain is present or not):

id	domains_covered	unique_domains_covered
1	24	3
2	43	6
3	1
4	1
5

Now, let's say I choose to "Enable only adlists with covered unique domains" on the final question. From my understanding, only adlists 1 and 2 will remain enabled, while the rest will be disabled.

Notice how lists 3 and 4 have one covered domain. Let's say this domain is example.com. example.com exists in both list 3 and 4, but not any of the other lists. I assume that this would mean that neither list 3 nor list 4 are considered to have a unique covered domain.

Would it be correct to say that, if you were to let the script enable only adlists containing unique domains, that protection against the example.com domain would be lost? Or is there something in the script that accounts for this that I'm missing? I feel that for people who use a large amount of adlists, this could be a fairly common "special case."

More importantly, if example.com does end up being lost, is there an easy way to determine a list of domains like this one that would be lost by skimming the adlists, in a real scenario where you don't know the all the specific domains in each adlist?

Running tool outside the Docker-container throws some errors

I was running it from within the docker-container and from outside the container. From outside it throws some errors:

  Would you like to ...

  1)  Keep your current adlist configuration
  2)  Enable only adlists with covered unique domains
  3)  Enable the minimal number of adlists that cover all domains that would have been blocked

  Please select: 2

  [i]  Enabling adlists with covered unique domains....
Error: in prepare, unrecognized token: "'"
  '
  ^--- error here
Error: in prepare, unrecognized token: "'"
  '
  ^--- error here
Error: in prepare, unrecognized token: "'"
  '
  ^--- error here
Error: in prepare, unrecognized token: "'"
  '
  ^--- error here
Error: in prepare, unrecognized token: "'"
  '
  ^--- error here
Error: in prepare, unrecognized token: "'"
  '
  ^--- error here
Error: in prepare, unrecognized token: "'"
  '
  ^--- error here
Error: in prepare, unrecognized token: "'"
  '
  ^--- error here
Error: in prepare, unrecognized token: "'"
  '
  ^--- error here
Error: in prepare, unrecognized token: "'"
  '
  ^--- error here
Error: in prepare, unrecognized token: "'"
  '
  ^--- error here
Error: in prepare, unrecognized token: "'"
  '
  ^--- error here

  [✓]  Adlists with covered unique domains enabled


  [i]  Of the 713 domains that would have been blocked by the adlist configuration you chose to analyze,
       711 domains are covered by your current adlist configuration.

  [i]  Removing temporary database...
  [✓]  Temporary database removed

I like the tool. I have about 40 adlists and after the check only 12 need to be enabled.

Instructions

Hello,

Would you kindly provide instructions for installation and execution of script in the README? Thanks!

yubiuser / pihole_adlist_tool Goto Github PK

pihole_adlist_tool's Introduction

Maintenance mode

Pihole Adlist Tool

The script outputs

Limits

Caveat

Requirements

Installation

Options

Background

Support, Contribute & Todo

pihole_adlist_tool's People

Contributors

Stargazers

Watchers

Forkers

pihole_adlist_tool's Issues

Recommend Projects

Recommend Topics

Recommend Org