Comments (4)
yubiuser
commented on July 22, 2024
Thanks for the report. I could not reproduce the issue.
3 1 20567 242 10385 http://sysctl.org/cameleon/hosts
52 1 156039 723 101128 https://blocklistproject.github.io/Lists/ads.txt
53 1 1220 https://blocklistproject.github.io/Lists/abuse.txt
disabled http://sysctl.org/cameleon/hosts, pihole -g
1 59130 423 68944 1 https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
2 0 https://mirror1.malwaredomains.com/files/justdomains
3 0 http://sysctl.org/cameleon/hosts
50 0 https://raw.githubusercontent.com/blocklistproject/Lists/master/
52 1 156039 723 101119 https://blocklistproject.github.io/Lists/ads.txt
53 1 1220 https://blocklistproject.github.io/Lists/abuse.txt
Can you reproduce the issue?
What's the output of pihole -g for https://blocklistproject.github.io/Lists/abuse.txt
from pihole_adlist_tool.
abaumfalk
commented on July 22, 2024
Thank you for the quick response.
Currently, I am trying to reproduce the issue. When I just logged into the pihole system it was again in the described state:
$ pihole_adlist_tool
*** Pihole Adlist Tool ***
++++++++ Info ++++++++
[i] DAYS_REQUESTED: 30
[i] TOP: Not shown
[i] SORT_ORDER: id ASC
[i] UNIQUE: Not shown
[i] NEW_ADLIST_FILENAME_SCHEMA: no
++++++++++++++++++++++
Would you like to analyze your current adlist configuration or first enable all adlists (current can be restored later)?
1) Current adlist configuration
2) Enable all adlists (runs pihole -g)
Please select: 1
[i] Keeping current adlist configuration
[i] Calculating.....
[i] This might take some time - please be patient.
[i] You have 9 adlists configured (4 enabled)
[i] Your gravity.db contains 175613 unique domains
[i] Since 22.07.2020 8:41:33 302 different domains from your adlists have been blocked 72629 times in total
[i] Using you current adlist configuration 448 domains would have been blocked 79324 times
Press enter to continue...
[i] Adlist coverage
id enabled total_domains domains_covered hits_covered unique_domains_covered address
---------- ---------- ------------- --------------- ------------ ---------------------- ----------------------------------------------------------------
1 0 https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
2 0 https://mirror1.malwaredomains.com/files/justdomains
3 0 http://sysctl.org/cameleon/hosts
5 0 https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
6 0 https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
8 1 23660 24 57 24 https://raw.githubusercontent.com/kboghdady/youTube_ads_4_pi-hol
9 1 156039 456 82037 https://blocklistproject.github.io/Lists/ads.txt
11 1 156039 456 82037 https://blocklistproject.github.io/Lists/ransomware.txt
12 1 1903 https://blocklistproject.github.io/Lists/tracking.txt
[i] Domains from disabled adlists are not stored in gravity's database.
If you want to include them in the analysis, run this script again and select 'Enable all adlists'
As the same domains usually appears on more than one adlist the sum of covered domains from this table is greater
than the number of calculated blocked domains shown above
[i] In total your adlists contain 24 visited (covered) unique domains - meaning those domains are contained only in a single adlist.
Would you like to keept your adlist configurartion, or keep only adlists enabled with at least one unique covered domain?
1) Keep adlist configuration
2) Enable only adlists with covered unique domains
Please select: 1
[i] Keeping current adlist configuration
[i] Removing temporary database...
[✓] Temporary database removed
Then, I removed all disabled lists and started pihole -g && pihole_adlist_tool - twice because of a 'database locked' error message - but now it seems that I cannot reproduce the error anymore.
The output was:
$ pihole -g && pihole_adlist_tool
[i] Neutrino emissions detected...
[✓] Pulling blocklist source list into range
[✓] Preparing new gravity database
[i] Target: https://raw.githubusercontent.com/kboghdady/youTube_ads_4_pi-hole/master/youtubelist.txt
[✓] Status: Retrieval successful
[i] Received 3560 domains
[i] Target: https://blocklistproject.github.io/Lists/ads.txt
[✓] Status: No changes detected
[i] Received 154697 domains
[i] Target: https://blocklistproject.github.io/Lists/ransomware.txt
[✓] Status: No changes detected
[i] Received 1903 domains
[i] Target: https://blocklistproject.github.io/Lists/tracking.txt
[✓] Status: No changes detected
[i] Received 15052 domains
[✓] Storing downloaded domains in new gravity database
[✓] Building tree
[✓] Swapping databases
[i] Number of gravity domains: 175212 (173344 unique domains)
[i] Number of exact blacklisted domains: 0
[i] Number of regex blacklist filters: 0
[i] Number of exact whitelisted domains: 4
[i] Number of regex whitelist filters: 0
[✓] Flushing DNS cache
[✓] Cleaning up stray matter
[✓] DNS service is running
[✓] Pi-hole blocking is Enabled
*** Pihole Adlist Tool ***
++++++++ Info ++++++++
[i] DAYS_REQUESTED: 30
[i] TOP: Not shown
[i] SORT_ORDER: id ASC
[i] UNIQUE: Not shown
[i] NEW_ADLIST_FILENAME_SCHEMA: no
++++++++++++++++++++++
Error: database is locked
Error: database is locked
Error: database is locked
Error: database is locked
[i] Warning: You requested to analyze the last 30 day(s) (starting from 22.07.2020 8:45:17),
but last query is from 01.01.1970 0:00:00
[i] Nothing to do here. Exiting
pi@raspberrypi:~ $ pihole -g && pihole_adlist_tool
[i] Neutrino emissions detected...
[✓] Pulling blocklist source list into range
[✓] Preparing new gravity database
[i] Target: https://raw.githubusercontent.com/kboghdady/youTube_ads_4_pi-hole/master/youtubelist.txt
[✓] Status: Retrieval successful
[i] Received 3560 domains
[i] Target: https://blocklistproject.github.io/Lists/ads.txt
[✓] Status: No changes detected
[i] Received 154697 domains
[i] Target: https://blocklistproject.github.io/Lists/ransomware.txt
[✓] Status: No changes detected
[i] Received 1903 domains
[i] Target: https://blocklistproject.github.io/Lists/tracking.txt
[✓] Status: No changes detected
[i] Received 15052 domains
[✓] Storing downloaded domains in new gravity database
[✓] Building tree
[✓] Swapping databases
[i] Number of gravity domains: 175212 (173344 unique domains)
[i] Number of exact blacklisted domains: 0
[i] Number of regex blacklist filters: 0
[i] Number of exact whitelisted domains: 4
[i] Number of regex whitelist filters: 0
[✓] Flushing DNS cache
[✓] Cleaning up stray matter
[✓] DNS service is running
[✓] Pi-hole blocking is Enabled
*** Pihole Adlist Tool ***
++++++++ Info ++++++++
[i] DAYS_REQUESTED: 30
[i] TOP: Not shown
[i] SORT_ORDER: id ASC
[i] UNIQUE: Not shown
[i] NEW_ADLIST_FILENAME_SCHEMA: no
++++++++++++++++++++++
Would you like to analyze your current adlist configuration or first enable all adlists (current can be restored later)?
1) Current adlist configuration
2) Enable all adlists (runs pihole -g)
Please select: 1
[i] Keeping current adlist configuration
[i] Calculating.....
[i] This might take some time - please be patient.
[i] You have 4 adlists configured (4 enabled)
[i] Your gravity.db contains 173344 unique domains
[i] Since 22.07.2020 8:46:24 302 different domains from your adlists have been blocked 72635 times in total
[i] Using you current adlist configuration 444 domains would have been blocked 79879 times
Press enter to continue...
[i] Adlist coverage
id enabled total_domains domains_covered hits_covered unique_domains_covered address
---------- ---------- ------------- --------------- ------------ ---------------------- ----------------------------------------------------------------------------------------
8 1 3560 6 11 6 https://raw.githubusercontent.com/kboghdady/youTube_ads_4_pi-hole/master/youtubelist.txt
9 1 154697 430 79804 410 https://blocklistproject.github.io/Lists/ads.txt
11 1 1903 https://blocklistproject.github.io/Lists/ransomware.txt
12 1 15052 28 10593 8 https://blocklistproject.github.io/Lists/tracking.txt
[i] Domains from disabled adlists are not stored in gravity's database.
If you want to include them in the analysis, run this script again and select 'Enable all adlists'
As the same domains usually appears on more than one adlist the sum of covered domains from this table is greater
than the number of calculated blocked domains shown above
[i] In total your adlists contain 424 visited (covered) unique domains - meaning those domains are contained only in a single adlist.
Would you like to keept your adlist configurartion, or keep only adlists enabled with at least one unique covered domain?
1) Keep adlist configuration
2) Enable only adlists with covered unique domains
Please select: 1
[i] Keeping current adlist configuration
[i] Removing temporary database...
[✓] Temporary database removed
Still trying to reproduce the problem, but no success so far...
from pihole_adlist_tool.
abaumfalk
commented on July 22, 2024
After many iterations of adding/removing lists I still cannot reproduce the problem. I'm giving up :-|
from pihole_adlist_tool.
yubiuser
commented on July 22, 2024
Thanks for your effort!
Being not able to reproduce the issue strengthens my assumption that this is an issue outside the counting algorithm of the tool itself. (Maybe due to database locked error? Gravity hiccup?)
I therefore would close the issue, feel free to re-open if the issue returns.
from pihole_adlist_tool.
Related Issues (20)
- Regex analysis not supported on docker HOT 2
- Receiving an error "database disk image is malformed" HOT 1
- Starting with version 2.4.0, you get multiple 'environment: line 5: docker: command not found' messages HOT 3
- "SUDO_SQLITE: command not found" HOT 3
- Does regex checking account for Filter Lists? HOT 2
- Regex Mode does not work in docker HOT 2
- Wrong detection of PIHOLE_DOCKER?! HOT 8
- Shellcheck is complaining a lot about pihole_adlist_tool HOT 1
- Installation HOT 3
- Question HOT 3
- Can't read /etc/pihole/pihole-FTL.db HOT 5
- Running tool outside the Docker-container throws some errors HOT 7
- Error: stepping, database is locked HOT 1
- Possibility to Deactivate redundant Blocklists HOT 7
- Add assigned group(s) for adlist in result HOT 3
- Instructions HOT 2
- Parse error near line 14: no such column: additional_info HOT 12
- ABP style adlists HOT 4
- Please add a licence HOT 1
- List the domains which generated a hit HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pihole_adlist_tool.