Giter VIP home page Giter VIP logo

scripttiger / unified-hosts-autoupdate Goto Github PK

View Code? Open in Web Editor NEW
293.0 14.0 26.0 3.32 MB

Quickly and easily install, uninstall, and set up automatic updates for any of Steven Black's unified hosts files.

Home Page: https://scripttiger.github.io/more/

License: MIT License

Batchfile 100.00%
unified-hosts malware ad-blocker porn-filter social-media-filter hosts gambling-filter updater updater-script autoupdate

unified-hosts-autoupdate's Introduction

Say Thanks!

Unified Hosts AutoUpdate

Quickly and easily install, uninstall, and set up automatic updates for any of Steven Black's unified hosts files.

This AutoUpdate project is maintained by ScriptTiger: https://github.com/ScriptTiger/Unified-Hosts-AutoUpdate

The Unified Hosts project is maintained by Steven Black: https://github.com/StevenBlack/hosts

Further project contributors are noted with their contributions in the Unified Hosts data, both available online from Steven Black's project as well as in the data injected into the local hosts file by this script, as it is downloaded directly from Steven Black's most recently pre-generated Unified Hosts files.

You can download this repo from the below link to get started:
https://github.com/ScriptTiger/Unified-Hosts-AutoUpdate/archive/master.zip
If your script worked in the past but has stopped working since, please update your repo files and try again before posting a new issue.

Extract the contents to the same directory and run the Hosts_Update.cmd script. There is also an option to remove the Unified Hosts from your local hosts file. In the event critical changes are made to either the AutoUpdate repo or to the Unified Hosts repo, this script is also capable of automatically updating itself.

If you experience problems with performance or seemingly unrelated networking issues after running this script, please try adjusting your compression level to one that best suits your needs.

This script requires either BITS or PowerShell to function. While both are native to modern Microsoft Windows installations, there are also compatible versions which can be manually installed for older systems.

If you want to ignore or whitelist certain entries from the Unified Hosts and prevent them from appearing in your local hosts, just add them to the ignore.txt. These entries are made with literal expressions and can match all or only part of an entry. So if you want to only ignore one specific URL, it's better to put the whole line just in case. For example, if you want to ignore the www.google.com domain specifically, put this:
0.0.0.0 www.google.com
If you want to ignore all .de websites, you can simply put:
.de
If you want to ignore all subdomains of google.com, you can put:
.google.com
If you want to ignore all google subdomains in any top level domain:
.google.

If you would also like to manage custom hosts file entries with this script, you can do so using the custom.txt. As this script does not alter custom entries in the hosts file itself, this is simply an option to make things easier if you would prefer to manage custom entries this way rather than manually managing them in the hosts file.

Also, if you send your preferred URL to the script as a parameter, it will bypass all the prompts and automatically install/update the Unified Hosts in the local hosts file. This is useful for things like scheduling a task to update your Unified Hosts daily or weekly, etc. You can even update your hosts randomly throughout the day at certain times to switch between Unified Hosts to, for example, only allow social in the evenings or on the weekends. And if you would like to remove the Unified Hosts from your hosts file, you can just enter "remove" as the URL. If you do decide to make a scheduled task, also remember the account issuing the task must still have administrative privileges to be able to write to the local hosts file. You can optionally add your preferred compression level as a second parameter, as well, but this must always accompany a URL as the first parameter.

Because no backup of your local hosts file is needed, entries in the Unified Hosts relating to the localhost and other loopback addresses have been removed to prevent possible conflict with preexisting entries. No backup is needed because this script implements the Unified Hosts within opening and closing tags to clearly segment it from the user's preexisting entries and allow the script to know what area of the file to overwrite during an update or remove during removal.

If you are deploying the update script across an organization via a shared network location and group policies, edit the "VERSION" file by adding an X to the end of the version number like this:
1.10X
This will disable the script from checking for script updates and attempting to update itself. This does not affect updates to the hosts file or whatever scheduled tasks you may have in place, this strictly disables the Hosts_Update.cmd from updating itself within the shared network location which remote system accounts running the scheduled tasks may not have write access to.

By default, the script forces the command processor instance to close upon completion just to ensure you don't have an unattended command prompt with administrative permissions lingering where it's not needded. However, for debugging or other purposes you can send /DFC as an initial argument to prevent the command prompt from closing after script completion. For further debugging, please reference the following error code table.

By default, log entries are only kept from the most recent scheduled task in the log.txt within the script home directory. If script updates are disabled, there will be no automatic logging since the assumption is the script directory is read-only. However, to force persistent logging to the script directory, you can also send /LOG as an initial argument to force writing a persistent log that always logs everything and never clears the log. You also have the option of alternatively using /LOG:<file> to write a persistent log to a directory and file of your choosing, which is recommended to be set to a local machine directory for deployments across networks so that logs are not written to the shared script directory which should be read-only. If you choose to configure persistent logging, please remember that managing that logging and associated file sizes then becomes your own responsibility.

Please note, all initial arguments (/DFC,/LOG:<file>, etc.) must be placed before the URL and compression parameters (i.e. Hosts_update.cmd /DFC <URL>).

Decimal Error Code Hexadecimal Error Code Explanation
0 0x0 The operation completed successfully. (No errors)
1 0x1 Must be run with administrative permissions
2 0x2 "#### END UNIFIED HOSTS ####" not properly marked in hosts file
3 0x3 Hosts file is not properly marked
4 0x4 Currently disabled due to maintenance, please try again later
5 0x5 Your system cannot connect to GitHub
6 0x6 Neither BITS nor PowerShell installed
7 0x7 Download mechanism cannot connect to GitHub
8 0x8 Download failed
255 0xFF The script terminated unexpectedly

This script is in active development, so please share your feedback on what you like and don't like so we know what direction to take and don't inadvertently make things worse

For more ScriptTiger scripts and goodies, check out ScriptTiger's GitHub Pages website:
https://scripttiger.github.io/

Donate

unified-hosts-autoupdate's People

Contributors

agyild avatar ohadschn avatar scripttiger avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

unified-hosts-autoupdate's Issues

choice.com for 64bits Windows 10

Hi
When I run the .cmd on my Windows 10 64bits, I get a lot of errors with system32/choice.com not being suitable on 64b systems. It should work with /system32/choice.exe. How to solve this?
Thanks
LT

.cmd emptied

for the second time now, I have gone to script to run it manually (because of a new addition to custom.txt) and I've found out that the file is now empty. Like literally 0kb file names Hosts_Update.cmd, I'm guessing something went terribly wrong during auto-update or something when the task was run automatically.

User Interface

I would like to get everyone's opinion of the current interface. Currently the process is a series of yes or no questions that dynamically change depending on your current setup. I was thinking if maybe a menu might be easier? I like the yes/no setup for the first time you run it just so it can take you through everything step by step, but maybe subsequent times the yes/no might be annoying and maybe a menu to jump to specific things might be better? But then again if everyone is using it via a scheduled task, then they won't have time to find the interactive mode annoying. If the only time people are using it interactively is to do the install and uninstall, then maybe guiding users through with a yes/no series is better. Anything is possible, so just drop your comments so I can get some direction on this.

Social block

The social blocking filter is really cumbersome to work with, like many reddit domains are also under the tumblr list and so on. Is there a way for the script to just ask me which social media sites to whitelist? I mean when people want to block social, they usually want all of them blocked, except the one or two that they still use, so it would be great to have the option of telling the script which ones to ignore.
Having to hunt down the entire list of convoluted domains and subdomains of the social media sites you use to add to the ignore list (and hope it actually gets ignored) is really a pain.

script can't connect

`Checking for script updates...

This script cannot connect to the Internet!
You are either not connected or BITS does not have permission.
If BITS does not have permission, daily automatic updates will still work.
BITS permissions only affect updating interactively on-demand with this script.
The Unified Hosts has not yet been marked in your local hosts file`

I don't have any virus scanner or firewall, and am running Windows 10 ENT B1803. The script can for some reason not connect to the internet while everything else on my pc connects just fine. I have also restarted the BITS service. No idea what to do, I also disabled and enabled my network adapter, but same issue still.

Slow to connect on computer restart

I've found it takes a couple minutes for the network to connect on my computer after running this. After it is connected everything works fine but when the computer is restarted I have to wait for awhile before it will connect to the network. Any ideas on what is causing this?

Script closes cmd.exe

Currently the user's cmd.exe is closed when the script ends. This is intrusive and can be annoying if for example you had other stuff you wanted to do in that terminal (you set up environment variables, you wanted to look at command output, you needed the command history, etc). This also interferes with debugging, I can't look at your script's logs or copy them because the windows immediately closes.

I suggest you always use exit /b so that the user's cmd.exe remains open.

how to uninstall ?

hello
i tryed this script on windows 10
after a restart it blocked completely my internet.

how to uninstall it please ?

How to schedule AutoUpdate host file in Windows

First of all, very big thanks for your hard work in developing this scripts and i'm sure it will benefit others.

Just 1 request, can provide step by step guide on how to schedule a task to update the Unified Hosts daily in Windows OS without user intervention.

Thanks.

Update.cmd

Every time I run the script and it says there's a new version, I get '"C:\Unified Hosts\Update.cmd"' is not recognized as an internal or external command, operable program or batch file.
and then it continues to: You currently have a scheduled task already in place Would you like to run the current task now? [Y,N]?Y

Paths with spaces need to be encapsulated with apostrophes for PowerShell (BITS)

Hi and first thanks for the great script :)

I had some trouble with this and long story short, it was because I had the script in a path with a space. Turns out that downloads would fail and it would run endless retries.

This was caused by the destination paths for the BITS transfers, being encapsulated in quotes (") but powershell wants apostrophes (').

Naturally, I could just run it from a path without spaces, but still, the script can be made resilient to this issue fairly easily.

I feel like a pull request is kinda overkill to change 8 characters - plus, maybe you'd rather fix it a different way.... so here are the offending lines - I hope this is OK:

Direct BITS calls:

Line 113:
%BITS_FROM% %GH%/master/VERSION %BITS_TO% "%CTEMP%" > nul || call :Connectivity

%BITS_FROM% %GH%/master/VERSION %BITS_TO% '%CTEMP%' > nul || call :Connectivity

Line 284:
%BITS_FROM% %URL% %BITS_TO% "%CTEMP%" > nul || call :Connectivity

%BITS_FROM% %URL% %BITS_TO% '%CTEMP%' > nul || call :Connectivity

Calls to :Download

Line 54:
call :Download %GH%/%COMMIT%/README.md "%README%" readme

call :Download %GH%/%COMMIT%/README.md '%README%' readme

Line 137:
call :Download %GH%/%COMMIT%/%CMD% "%UPDATE%" update

call :Download %GH%/%COMMIT%/%CMD% '%UPDATE%' update

Line 404:
if not !REMOVE!==1 call :Download %URL% "%CTEMP%" hosts

if not !REMOVE!==1 call :Download %URL% '%CTEMP%' hosts

Obviously, there may be other ways to use apostrophe rather than quote, but I'll leave that in your very capable hands.

I hope this helps, and thanks again :)

custom.txt and ignore,txt MD5 checksum does not work

I installed the script along with an ignore.txt file. When I looked at the hosts file, the lines for the MD5 checksum for ignore list and custom list is blank. After looking at the code it looks like the first "for" needs a "delims=" since for uses space a the standard delimiter (otherwise you only get the first 2 characters of the checksum.) Also certutil is case sensitive when selecting hash type ("MD5" needed) so the lines should be:
for /f "delims=" %%0 in ('certutil -hashfile "%IGNORE%" MD5 ^| findstr /v :') do set NEWIGNORE=%%0

and:
for /f "delims=" %%0 in ('certutil -hashfile "%CUSTOM%" MD5 ^| findstr /v :') do set NEWCUSTOM=%%0

Generated update scheduled task succeeds but returns result 0x1

Please see:
https://stackoverflow.com/questions/18370547/windows-scheduled-task-succeeds-but-returns-result-0x1

Fix is very simple, first bullet in the highest voted answer:
https://stackoverflow.com/a/34362341/67824

  • Right click on your task > Properties > Actions > Edit
  • Put ONLY the file name under 'Program/script' (no quotes)
  • Put ONLY the directory under 'Start in' as described (again, no quotes)

So basically I suggest you create the update task like that (using schtasks) to begin with.

Otherwise script works great, thanks a bunch for it!

ignorelist ignored

on the left is my ignore.txt on the right is my hosts file.
image
I literally copied the list of hosts I didn't want blocked, from the hosts file to the ignore.txt and yet the script (after updating) still puts these hosts in the hosts file.

flushdns

I would like to suggest that ipconfig /flushdns be added to the script just before exit.

The Official Chocolatey Package!

I just found out this project has an official Chocolatey package! It sounds pretty cool, but I don't use Chocolatey personally and don't have any experience with it. I would really like to promote this in the future though if it's legit. Can someone verify this package for me and let me know their findings?

P.S.: For those that don't know, Chocolatey is a package manager for Windows, similar to the Linux package managers. choco install unified-hosts-autoupdate will install my script on Windows similar to how you do apt-get install on Linux. So if you don't have Chocolatey now but you're now thinking about getting it because it sounds cool, try installing the unified-hosts-autoupdate first and let me know how it goes.

Literals vs. Regular Expressions

When I first put this script together I wasn't sure what kind of user base I would be dealing with and I decided to go with literal expressions in the ignore file because you can just put the line exactly as it is that you want to ignore and you don't need any special characters and it makes it a bit easier for the less technically inclined. However, from what I have seen I think the users here are more than able to navigate the much more powerful regular expressions, but I'll be sure to include plenty of examples as needed to get those less technically inclined where they need to be to get the outcome that they want. This move will, however, break whatever ignore lists users currently have in place, so I just wanted to post this as an issue before implementing anything to get some feedback, give some warning, etc. The change itself would just be changing a single command, but the implications for the users are much greater. So let me know what you think and hopefully make me feel less guilty about going through with it.

A similar update script which already uses regular expressions is my DualServer updater:
https://github.com/ScriptTiger/DualServer
I went with regular expressions with DualServer because literals aren't as helpful in the DualServer format and being a DNS server it should be as configurable as possible.

You can google more info on regular expressions (RegEx), but here's a brief cheat sheet:

. Wildcard: any character
* Repeat: zero or more occurrences of previous character or class
^ Line position: beginning of line
$ Line position: end of line
[class] Character class: any one character in set
[^class] Inverse class: any one character not in set
[x-y] Range: any characters within the specified range
\x Escape: literal use of metacharacter x
<xyz Word position: beginning of word
xyz> Word position: end of word

invoke-webrequest : Invalid URI: The hostname could not be parsed.

Hi,

It seems like one of the recent updates broke the script for me.

Here's what my task is running:
Hosts_Update.cmd /log:%TEMP%\UAHU-%DATE:~-4%.%DATE:~4,2%.%DATE:~7,2%-%random%.log https://raw.githubusercontent.com/StevenBlack/hosts/master/alternates/fakenews-gambling/hosts 1

The msg error is:

ERROR: PowerShell failed downloading /log:C:\WINDOWS\TEMP\UAHU-2020.03.20-20351.log! 
Refer to your log file for more information: 
"G:\Util\Unified-Hosts-AutoUpdate\log.txt" 

Note that it didn't catch the custom log file I specified. I think it mistakes the log argument for the hosts URL:
Fri 03/20/2020 @ 3:00:29.03: Executing: powershell invoke-webrequest /log:C:\WINDOWS\TEMP\UAHU-2020.03.20-31543.log -outfile 'C:\WINDOWS\TEMP\Unified-Hosts-AutoUpdate\ctemp'

I thought the custom log location might be the culprit so I tried changing my execution to: Hosts_Update.cmd /log https://raw.githubusercontent.com/StevenBlack/hosts/master/alternates/fakenews-gambling/hosts 1 but I'm getting the same (Fri 03/20/2020 @ 5:11:35.22: Executing: powershell invoke-webrequest /log -outfile 'C:\WINDOWS\TEMP\Unified-Hosts-AutoUpdate\ctemp').

Here's the full log:

Fri 03/20/2020 @  3:00:02.61: Initializing...
Fri 03/20/2020 @  3:00:02.65: Executing: bitsadmin /list

BITSADMIN version 3.0
BITS administration utility.
(C) Copyright Microsoft Corp.

{645970C1-2174-4898-A454-E9FF1B001815} 'PreSignInSettingsConfigJSON' SUSPENDED 0 / 1 0 / UNKNOWN
Listed 1 job(s).
Fri 03/20/2020 @  3:00:07.02: Executing: powershell $host.version

Major  Minor  Build  Revision
-----  -----  -----  --------
5      1      18362  628     


Fri 03/20/2020 @  3:00:09.22: Checking for script updates...
Fri 03/20/2020 @  3:00:09.27: Executing: powershell invoke-webrequest https://raw.githubusercontent.com/ScriptTiger/Unified-Hosts-AutoUpdate/master/VERSION  -outfile 'C:\WINDOWS\TEMP\Unified-Hosts-AutoUpdate\ctemp'
Fri 03/20/2020 @  3:00:12.86: Downloaded version successfully
Fri 03/20/2020 @  3:00:12.90: A new script update is available!
Fri 03/20/2020 @  3:00:12.93: Updating script...
Fri 03/20/2020 @  3:00:15.19: Executing: powershell invoke-webrequest https://raw.githubusercontent.com/ScriptTiger/Unified-Hosts-AutoUpdate/20faf8f92ffe7bbae603544b9ba3f4e33b0a9757/Hosts_Update.cmd  -outfile 'G:\Util\Unified-Hosts-AutoUpdate\Update.cmd'
Fri 03/20/2020 @  3:00:18.00: Downloaded update successfully
Fri 03/20/2020 @  3:00:21.24: Initializing...
Fri 03/20/2020 @  3:00:21.30: The updated script has been loaded
Fri 03/20/2020 @  3:00:21.33: Executing: powershell invoke-webrequest https://raw.githubusercontent.com/ScriptTiger/Unified-Hosts-AutoUpdate/20faf8f92ffe7bbae603544b9ba3f4e33b0a9757/README.md  -outfile 'G:\Util\Unified-Hosts-AutoUpdate\README.md'
Fri 03/20/2020 @  3:00:24.00: Downloaded readme successfully
Fri 03/20/2020 @  3:00:24.04: Executing: bitsadmin /list

BITSADMIN version 3.0
BITS administration utility.
(C) Copyright Microsoft Corp.

{645970C1-2174-4898-A454-E9FF1B001815} 'PreSignInSettingsConfigJSON' SUSPENDED 0 / 1 0 / UNKNOWN
Listed 1 job(s).
Fri 03/20/2020 @  3:00:24.11: Executing: powershell $host.version

Major  Minor  Build  Revision
-----  -----  -----  --------
5      1      18362  628     


Fri 03/20/2020 @  3:00:25.80: Checking for script updates...
Fri 03/20/2020 @  3:00:25.86: Executing: powershell invoke-webrequest https://raw.githubusercontent.com/ScriptTiger/Unified-Hosts-AutoUpdate/master/VERSION  -outfile 'C:\WINDOWS\TEMP\Unified-Hosts-AutoUpdate\ctemp'
Fri 03/20/2020 @  3:00:28.55: Downloaded version successfully
Fri 03/20/2020 @  3:00:28.59: Your script is up to date
Fri 03/20/2020 @  3:00:28.93: The Unified Hosts is already installed in your local hosts file
Fri 03/20/2020 @  3:00:28.95: Checking for Unified Hosts updates...
Fri 03/20/2020 @  3:00:29.01: Updating the hosts file...
Fri 03/20/2020 @  3:00:29.03: Executing: powershell invoke-webrequest /log:C:\WINDOWS\TEMP\UAHU-2020.03.20-31543.log  -outfile 'C:\WINDOWS\TEMP\Unified-Hosts-AutoUpdate\ctemp'
invoke-webrequest : Invalid URI: The hostname could not be parsed.
At line:1 char:1
+ invoke-webrequest /log:C:\WINDOWS\TEMP\UAHU-2020.03.20-31543.log -out ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Invoke-WebRequest], UriFormatException
    + FullyQualifiedErrorId : System.UriFormatException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
 
Fri 03/20/2020 @  3:00:31.05: Waiting 10 seconds before retry...
Fri 03/20/2020 @  3:00:51.18: Download Retry 1: hosts...
Fri 03/20/2020 @  3:00:51.21: Executing: powershell invoke-webrequest /log:C:\WINDOWS\TEMP\UAHU-2020.03.20-31543.log  -outfile 'C:\WINDOWS\TEMP\Unified-Hosts-AutoUpdate\ctemp'
invoke-webrequest : Invalid URI: The hostname could not be parsed.
At line:1 char:1
+ invoke-webrequest /log:C:\WINDOWS\TEMP\UAHU-2020.03.20-31543.log -out ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Invoke-WebRequest], UriFormatException
    + FullyQualifiedErrorId : System.UriFormatException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
 
Fri 03/20/2020 @  3:00:53.08: Waiting 20 seconds before retry...
Fri 03/20/2020 @  3:01:33.11: Download Retry 2: hosts...
Fri 03/20/2020 @  3:01:33.14: Executing: powershell invoke-webrequest /log:C:\WINDOWS\TEMP\UAHU-2020.03.20-31543.log  -outfile 'C:\WINDOWS\TEMP\Unified-Hosts-AutoUpdate\ctemp'
invoke-webrequest : Invalid URI: The hostname could not be parsed.
At line:1 char:1
+ invoke-webrequest /log:C:\WINDOWS\TEMP\UAHU-2020.03.20-31543.log -out ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Invoke-WebRequest], UriFormatException
    + FullyQualifiedErrorId : System.UriFormatException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
 
Fri 03/20/2020 @  3:01:34.97: Waiting 40 seconds before retry...
Fri 03/20/2020 @  3:02:54.14: Download Retry 3: hosts...
Fri 03/20/2020 @  3:02:54.17: Executing: powershell invoke-webrequest /log:C:\WINDOWS\TEMP\UAHU-2020.03.20-31543.log  -outfile 'C:\WINDOWS\TEMP\Unified-Hosts-AutoUpdate\ctemp'
invoke-webrequest : Invalid URI: The hostname could not be parsed.
At line:1 char:1
+ invoke-webrequest /log:C:\WINDOWS\TEMP\UAHU-2020.03.20-31543.log -out ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Invoke-WebRequest], UriFormatException
    + FullyQualifiedErrorId : System.UriFormatException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
 
Fri 03/20/2020 @  3:02:55.97: Waiting 80 seconds before retry...
Fri 03/20/2020 @  3:05:35.19: Download Retry 4: hosts...
Fri 03/20/2020 @  3:05:35.22: Executing: powershell invoke-webrequest /log:C:\WINDOWS\TEMP\UAHU-2020.03.20-31543.log  -outfile 'C:\WINDOWS\TEMP\Unified-Hosts-AutoUpdate\ctemp'
invoke-webrequest : Invalid URI: The hostname could not be parsed.
At line:1 char:1
+ invoke-webrequest /log:C:\WINDOWS\TEMP\UAHU-2020.03.20-31543.log -out ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Invoke-WebRequest], UriFormatException
    + FullyQualifiedErrorId : System.UriFormatException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
 
Fri 03/20/2020 @  3:05:37.02: Waiting 160 seconds before retry...
Fri 03/20/2020 @  3:10:57.18: Download Retry 5: hosts...
Fri 03/20/2020 @  3:10:57.21: Executing: powershell invoke-webrequest /log:C:\WINDOWS\TEMP\UAHU-2020.03.20-31543.log  -outfile 'C:\WINDOWS\TEMP\Unified-Hosts-AutoUpdate\ctemp'
invoke-webrequest : Invalid URI: The hostname could not be parsed.
At line:1 char:1
+ invoke-webrequest /log:C:\WINDOWS\TEMP\UAHU-2020.03.20-31543.log -out ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Invoke-WebRequest], UriFormatException
    + FullyQualifiedErrorId : System.UriFormatException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
 
Fri 03/20/2020 @  3:10:59.12: Waiting 320 seconds before retry...
Fri 03/20/2020 @  3:21:39.13: Download Retry 6: hosts...
Fri 03/20/2020 @  3:21:39.16: Executing: powershell invoke-webrequest /log:C:\WINDOWS\TEMP\UAHU-2020.03.20-31543.log  -outfile 'C:\WINDOWS\TEMP\Unified-Hosts-AutoUpdate\ctemp'
invoke-webrequest : Invalid URI: The hostname could not be parsed.
At line:1 char:1
+ invoke-webrequest /log:C:\WINDOWS\TEMP\UAHU-2020.03.20-31543.log -out ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Invoke-WebRequest], UriFormatException
    + FullyQualifiedErrorId : System.UriFormatException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
 
Fri 03/20/2020 @  3:21:41.11: PowerShell failed downloading /log:C:\WINDOWS\TEMP\UAHU-2020.03.20-31543.log!
Fri 03/20/2020 @  3:21:41.14: Cleaning temporary files...
-----

BTW It would be great if the script printed all parsed arguments for easier debugging of such cases.

Script cannot connect to the internet if only "Run As" Administrator

How to reproduce:

  • Run script as administrator
  • Script throws the following error:
This script cannot connect to the Internet!
This script requires an active Internet connection to update your hosts file!

The thing is, there are no Internet issues, and it's not my first time running the script.

Exit with meaningful exit codes

The script currently exits with a success exit code (0), no matter what happens.
For example, if I'm not an admin and the script does nothing, it would still report success.
This should be fixed so whenever the script encounters any failure (e.g. can't create task) it exits with a non-zero exit code (ideally you could even differentiate, e.g. 1 means not admin, 2 means couldn't create task etc).

Use /LOG switch in scheduled task by default?

On several occasions now, I have found that my hosts are not updating, because of the script not updating. Usually, this script "just works" and so I don't really notice it until something is wrong.... and then I go to the scheduled tasks view and get some cryptic MS hex error code.

Then I have to open a command prompt as an admin, change to the correct directory, copypaste the command from three different text fields in the scheduled tasks GUI, change to the correct directory, paste in the two parts of the command, to run it to see what's happening, to try and troubleshoot it. It's no big deal.... but it is a little cumbersome.

Today I noticed that there is an option to output a log file. I've already added that to my scheduled task manually, but it strikes me that it would be nice to have that as a default. Just makes it a lot easier to see what's going on when the script does its magic... and especially, when it doesn't :)

Edit: Ironically, the failure I had, made me miss this update which would have alerted me to the failures XD

No internet after restart for minutes

Ever since I used Unified Hosts AtuoUpdate whenever I turn on my PC for minutes I won't have internet, the only thing that seems to help if I run the windows network troubleshooter, even though it says it didn't find anything wrong when it finishes I finally able to use the internet.

Why is this happening, can I do something about it?

not connecting base?

sometimes the script is stuck in somekind of loop, like this:

Script updates currently disabled
The Unified Hosts is already installed in your local hosts file
Checking for Unified Hosts updates...
Download Retry 1: base...
Download Retry 2: base...
Download Retry 3: base...
Download Retry 4: base...
Download Retry 5: base...
Download Retry 6: base...
Download Retry 7: base...
Download Retry 8: base...
Download Retry 9: base...
Download Retry 10: base...
Download Retry 11: base...
Download Retry 12: base...
Download Retry 13: base...
Download Retry 14: base...
Download Retry 15: base...
Download Retry 16: base...
Download Retry 17: base...
Download Retry 18: base...
Download Retry 19: base...
Download Retry 20: base...
Download Retry 21: base...
Download Retry 22: base...
Download Retry 23: base...
Download Retry 24: base...
Download Retry 25: base...
Download Retry 26: base...
Download Retry 27: base...
Download Retry 28: base...
Download Retry 29: base...
Download Retry 30: base...
Download Retry 31: base...
Download Retry 32: base...
Download Retry 33: base...
Download Retry 34: base...
Download Retry 35: base...
Download Retry 36: base...
Download Retry 37: base...
Download Retry 38: base...
Download Retry 39: base...
Download Retry 40: base...
Download Retry 41: base...
Download Retry 42: base...
Download Retry 43: base...
Download Retry 44: base...
Download Retry 45: base...
Download Retry 46: base...
Download Retry 47: base...
Download Retry 48: base...
Download Retry 49: base...
Download Retry 50: base...
Download Retry 51: base...
Download Retry 52: base...
Download Retry 53: base...
Download Retry 54: base...
Download Retry 55: base...
Download Retry 56: base...
Download Retry 57: base...
Download Retry 58: base...
Download Retry 59: base...
Download Retry 60: base...
Download Retry 61: base...
Download Retry 62: base...
Download Retry 63: base...
Download Retry 64: base...
Download Retry 65: base...```

[question] need little help with small script

@ScriptTiger very sorry to bother you man but i know you are good with scripts so i decided to seek your wisdom on that matter.So long story short.Per your advice to switch to dnsmasq etc. i loaded custom firmware to my router which has dnsmsq installed in it.So far so good but due to small cache(only 8mb) and using ready script(found it in one forum) that is using sim link to load the host file,which loaded once and didn't want to reload it anymore i decided to keep the hosts file in usb connected to the router.This way is working fine and i pointed dnsmasq to use that hosts file instead of the originial. Now i'm working on creating small script just to update the file whenever update is available.I'm using plink.exe which is part of PuTTY to SSH to the router.This is the content of the script so far,any advise on how to make it better will be greatly appreciated:
" @echo off

"D:\Program Files\PuTTY\plink.exe" -ssh [email protected] -pw ****** -m D:\test.sh

pause

start C:\Windows\System32\ipconfig.exe
ipconfig /flushdns

pause"

The content in TEST.SH is:
#!/bin/sh
wget https://raw.githubusercontent.com/StevenBlack/hosts/master/alternates/fakenews-gambling-social/hosts -O /media/AiDisk_a1/Hosts/add.hosts
killall dnsmasq
/usr/sbin/dnsmasq

What i need is after finish updating the file with plink.exe to flush the dns cache without the additional cmd window to flash for a second and overall whatever else you could advise.Thank you.

Hosts marked properly but still receiving error

When I run the cmd file as Admin, command window shows the following messages.

Checking for script updates...
Your script is up to date

Hosts is not properly marked
Please ensure the following lines mark where to insert the blacklist:

#### BEGIN UNIFIED HOSTS ####
#### END UNIFIED HOSTS ####

Notes: You should only have to mark this once
Updates automatically overwite between the above lines
Would you like to open your current hosts file before exiting? [Y,N]?

However, as you can see below, the hosts file begins with the proper marking.

#### BEGIN UNIFIED HOSTS ####
# Managed by ScriptTiger's Unified Hosts AutoUpdate
# https://github.com/ScriptTiger/Unified-Hosts-AutoUpdate
# Ignore list: eaedc45a77913a0410250d589a252468
# Custom list: a639fe0920dd7460ff4564c497116689
#
# This hosts file is a merged collection of hosts from reputable sources,
# with a dash of crowd sourcing via Github
#
# Date: March 03 2018
# Extensions added to this file: fakenews, gambling, porn
# Number of unique domains: 67,718
#
# Fetch the latest version of this file: https://raw.githubusercontent.com/StevenBlack/hosts/master/alternates/fakenews-gambling-porn/hosts
# Project home page: https://github.com/StevenBlack/hosts
#
# ===============================================================
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
# Custom host records are listed here.
# End of custom host records.
#=====================================
# Hosts contributed by Steven Black
# http://stevenblack.com
0.0.0.0 1493361689.rsc.cdn77.org
0.0.0.0 30-day-change.com
0.0.0.0 2468.go2cloud.org

In addition, the latest hosts file from Steven Black is dated March 10 and mine says March 03. However, the script shows that my script is up to date.

IPv4/IPv6 Issues

Does anyone have any issues related to IPv4/IPv6? Some examples might be IPv6 queries not getting blocked, an IPv6 system not successfully querying on an IPv4 network, etc. We have talked about this numerous times on Steven Black's repo, but I just wanted to open an issue here to discuss it as it relates specifically to Windows systems running this script.

If you are encountering an issue related to this, please first try manually installing one of the IPv4_IPv6 hosts files from the below link:
https://scripttiger.github.io/alts/

Whether the above solves your problem or not, please comment as to what your issue was before and what happened after. If this is a relevant issue for us, I will add support for it in the script.

Possibility to complete scheduled task before script detects it has begun

Related to our discussion about the usage of the lock file, I came across another small problem. First, to describe the procedure to replicate the fault:

Have the scheduled task set up and working like the gold that it is.
Run the batch file manually from a command prompt.
Receive the prompt stating that you already have a scheduled task and would you like to run it. Press 'Y' to run the task.
Task runs, exits and cleans up after itself, before the script detects that it is running.
(This occurs because the script waits 5 seconds for the cache directory to be created, and then again because it waits 5 seconds for it to be removed. If the script has nothing to update, on a nice fast NVMe drive and such, 5 seconds is about 4.9 seconds longer than it needs ;) )
Script then sits there indefinitely waiting for the task to run.
CTRL+C to terminate the batch file, leaving temp files (user directory cache, not system cache) in place because it never gets to clean up from the manual run.

Since I feel like I created too much work for you already today, I threw together this attempt which seems to be working nicely. It's quite possibly an abomination to please feel free to discard it for a superior approach. I just wanted to contribute something more productive than "Hey I broke it" for a change ;)

rem Function for running a scheduled task from script before exiting
:Run
rem Lock the running script from being replaced by an update during the triggered task
rem Unlock later and replace running script with update if exists before exit
echo "%LOG:"=%">"%LOCK%"
call :Echo "Activating update task..."
set TIMER=0

schtasks /run /tn "%TN%"
if not exist "%SCACHE%" goto Run_Wait_Start
goto Run_Running

:Run_Wait_Start
timeout /t %TIMER% /nobreak > nul
if not exist "%SCACHE%" (
	set /a !TIMER!+1
	goto Run_Wait_Start
)

:Run_Running
call :Echo "Update task is running..."
set TIMER=0

:Run_Wait_Stop
timeout /t %TIMER% /nobreak > nul
if not exist "%SCACHE%" (
	call :Echo "Update task has completed"
	set TASK=3
	del /q "%LOCK%"
	goto View_Hosts
)
set /a !TIMER!+1
goto Run_Wait_Stop

Edit: I derped and pasted the wrong version, it's no big change

Echo download command to console/log

The task failed with the following log:

Sun 01/26/2020 @  3:00:06.03: Checking for script updates...
Sun 01/26/2020 @  3:00:06.36: The Unified Hosts is already installed in your local hosts file
Sun 01/26/2020 @  3:00:06.37: Checking for Unified Hosts updates...
Sun 01/26/2020 @  3:00:07.19: Waiting 10 seconds before retry...
Sun 01/26/2020 @  3:00:27.20: Download Retry 1: benchmark...
Sun 01/26/2020 @  3:00:27.98: Waiting 20 seconds before retry...
Sun 01/26/2020 @  3:01:08.15: Download Retry 2: benchmark...
Sun 01/26/2020 @  3:01:08.90: Waiting 40 seconds before retry...
Sun 01/26/2020 @  3:02:28.19: Download Retry 3: benchmark...
Sun 01/26/2020 @  3:02:28.92: Waiting 80 seconds before retry...
Sun 01/26/2020 @  3:05:08.20: Download Retry 4: benchmark...
Sun 01/26/2020 @  3:05:13.14: Waiting 160 seconds before retry...
Sun 01/26/2020 @  3:10:33.15: Download Retry 5: benchmark...
Sun 01/26/2020 @  3:10:38.19: Waiting 320 seconds before retry...
Sun 01/26/2020 @  3:21:18.20: Download Retry 6: benchmark...
Sun 01/26/2020 @  3:21:23.18: Failed downloading https://raw.githubusercontent.com/StevenBlack/hosts/master/alternates/fakenews-gambling/hosts!
Sun 01/26/2020 @  3:21:23.23: Cleaning temporary files...

There is nothing wrong with my internet, so I don't know why the download would fail. Rerunning the task manually from the Task Scheduler UI, it succeeds in the first attempt in a total of less than 3 seconds.

I thought a good debugging start would be to echo the exact command used for downloading. Even better, echoing that command's stdout/stderr as well could go a long way to help us understand what went wrong (maybe something like powershell Start-BitsTransfer -source ... > logfilename).

hosts file corrupted, pc bricked

all I know is that I stopped getting internet to work (pinging IP addresses of google worked, but pinging their domain didn't).
Then I opened the hosts file and it had countless markers of **END OF UNIFIED SOMETHING SOMETHING over and over again. Then I left it as it were and ran the script again. To no avail.

Then when I restarted my pc, windows would get stuck on "logging in" forever. Installation dvd startup fix couldn't fix it. resetting cmos either.
I had to boot a windows installation dvd, then use its file browser to delete the hosts file and the cmd script, and then windows logged in with no problem.

Idk what went wrong with this script, but it went wrong in a big way somehow.

Always retry downloads

It looks like some downloads are retried and some aren't. I keep failing on "connectivity issues" but there is nothing wrong with my internet, I'm guessing these are the culprits (but I can't be sure since the error message doesn't specify the URL it was trying to download):

%BITS_FROM% %GH%/master/VERSION %BITS_TO% %Q%%CTEMP%%Q% > nul || call :Connectivity

%BITS_FROM% %URL% %BITS_TO% %Q%%CTEMP%%Q% > nul || call :Connectivity

I think the logical thing to do would be to replace them with Download calls (more consistent too):

Port to PowerShell

The script works great, but really PowerShell would be a much, much better options in any way imaginable (first and foremost ease of maintenance for you). The current ~700 lines would probably be cut in half, while making it more robust.

ignore list ignored

I have in my ignore.txt:
.4chan.org
but somehow, even after updating multiple times, when I open my hosts file, I still see "0.0.0.0 4chan.org"
Am I doing something wrong? should I make two entries in the ignore list, one with and one without the dot at the start?

Write log file

I'm getting intermittent errors where if I re-run manually everything works fine.

It would be great if text logs, detailing the operations taken by the script, were written to somewhere like %TEMP% (the msg error is a good start, but not enough to debug the issue).

DNS Client Cache Issue

Are any users currently experiencing the DNS client cache issue described in StevenBlack/hosts#411? If so, can you manually try the new fix/workaround, StevenBlack/hosts#459, and give your feedback here? I'd like to get some more use cases on it before implementing a similar process in the AutoUpdate script here.

Unbound refuse instead of redirect

Could we set the Unbound list to refuse requests instead of redirect?
This would shrink the file size considerably:
local-zone: "1493361689.rsc.cdn77.org" redirect
local-data: "1493361689.rsc.cdn77.org A 0.0.0.0"
vs
local-zone: "1493361689.rsc.cdn77.org" refuse

Refusing queries also make collecting blocking stats easier (using num.answer.rcode.REFUSED)

Compression Level

Hello,

thank you very much for the amazing script.
Could you tell me more about the compression? What excactly is the advantage to use a highter or lower level?

I am talking about this part:
echo Your hosts file can be from 1 to 9 domains per line. echo 1 is standard, more than 1 is a level of compression. echo If you choose a level of compression, please expect the update to take longer. echo Your current compression level is %OLDCOMP%. choice.exe /m "Would you like to just stick with that?"

Thank you very much

Hosts file does not open if "y" selected but UserChoice registry key has not been set

The last question the script asks is if I want to open my hosts file. I have never right-clicked on a text file and set a user choice for default program for .txt since I use notepad normally and that hapens automatically in Windows 7. Because of that, the logic fails and instead the last program in the list created by the 'ftype' command opens (in my case Zoom Player.) If I right-click on a .txt file and choose notepad as my default, after that it works. The first reg query should probably check errorlevel and bail out if the key doesn't exist and open with notepad there. I do like the logic other than that in case the user uses notepad++ or something like that in place of notepad. I tested with 4 Windows 7 x64 PCs.

'hosts' filters differs when using compression

Compression 9 gives another outcome in 'hosts', then when using no compression (1) in blocklist (hosts).

i.e.
Hosts_Update.cmd https://raw.githubusercontent.com/StevenBlack/hosts/master/alternates/fakenews-gambling-porn/hosts 9 (max compression), then these next lines do NOT show up in 'hosts':

127.0.0.1 localhost
127.0.0.1 localhost.localdomain
127.0.0.1 local
255.255.255.255 broadcasthost
::1 localhost
::1 ip6-localhost
::1 ip6-loopback
fe80::1%lo0 localhost
ff00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
0.0.0.0 0.0.0.0

If I use:
Hosts_Update.cmd https://raw.githubusercontent.com/StevenBlack/hosts/master/alternates/fakenews-gambling-porn/hosts 1 (no compression), then these same lines DO get into the 'hosts' file, as expected.

So somehow these lines get skipped when using compression (9), this is what I get into my 'hosts' when I use compression (9):

# Project releases: https://github.com/StevenBlack/hosts/releases #   # =============================================================== 
# Custom host records are listed here. # End of custom host records. # Start StevenBlack #=====================================   # Hosts contributed by Steven Black # http://stevenblack.com 

Retry improvements

Regarding the Retry function:

  1. It looks like there is no limit to the amount of retries, possibly looping forever. I think at some point (e.g. 10 retries) the script needs to give up.
  2. Ideally, you would implement a simple exponential back-off strategy (or something like it), rather than wait exactly 10 seconds each time (so you would wait 10, 20, 40, 80, and then maybe cap it at 120, 120, 120, etc). This will help overcoming intermittent network issues.

Use Invoke-WebRequest instead of Start-BitsTransfer when PowerShell is available

If PowerShell isn't available BITSAdmin is a great idea, but when it is, relying on the BITS service is counterproductive:

  1. If you have PowerShell installed, Invoke-WebRequest will always be available (it's a core part of PowerShell). However Start-BitsTransfer might not work if the BITS service is not up and running (or even enabled).
  2. Even when BITS is enabled, it is a real potential point of failure - I am 99% sure it is the cause of my jobs failing every other night (will only know for sure now that we have error logs), and I am just as sure that Invoke-WebRequest would have worked 100% of them. Just now, I disabled my network, re-enabled it, and saw this in the logs: Start-BitsTransfer : The job was paused because there is an active Game Mode recipient that isn't the job's owner. The job will resume once the Game Mode recipient loses input focus or is closed. The bottom line is that Invoke-WebRequest is extremely simple, straightforward and far far more mainstream, and thus battle-tested (at least when called from PowerShell) - you would never get such weird error cases there.

Reference: https://superuser.com/questions/362152/native-alternative-to-wget-in-windows-powershell

Alert on task failure

Currently, if the task fails for whatever reason, there is no notification.
This means that I can go on for 20 years having an out of date HOSTS file without knowing, because the script is failing every day and I don't know about it.
I suggest alerting the user somehow so they can't miss a failure. For example - leave a modal dialog open that notifies about the error. Or send an e-mail, or create a toast notification. Something...

More source options

Hi @ScriptTiger have been following you around like a google stalker ๐Ÿ˜ƒ and then came to this script and read your code in the cmd.

As i found this as a good code base for users who runs windows systems and wan't to do blocking via hosts file.

But what came across my hard is it seems one-sided when i came to the choice of source of hosts file, not that there are anything wrong with @StevenBlack project (quite the opposed), but what if a user would like to have the choice of which source[s] they would like to rely on?

So I would give my 2 cents to a suggestion:
have the you're script read it's sources from a file like your white/black list...

Specify log location (or instructions for enabling logging) in error messages

Something that might have gotten lost in the other thread about logging, how about specifying the location of the log file in error messages? I feel it's a fair compromise between automatically enabling logs (which you wish to avoid) and helping the users debug issues themselves.

Something like this:
Error: could not download XYZ (consult the log file for more information: C:\UHA\log.txt)

Or in case logging was not enabled:
Error: could not download XYZ (to enable logging add '/log' as the first parameter to the script)

(I figure this is mostly relevant to quiet/task/msg errors, as a user running the script interactively can see the lines on the screen anyway)

custom.txt issue

(running Unified-Hosts-AutoUpdate with Windows 10 Pro x64)

When I add analytics.ff.avast.com to 'custom.txt' it does not get processed.
I used: 0.0.0.0 analytics.ff.avast.com

all other lines do get processed, for instance: 0.0.0.0 adservice.google.com

With kind regards

reset the hosts script

could you add another script to reset the hosts file to windows default. I found this awesome tool from this page that does that impressively.

choose_2018-07-23_15-29-14

Choosing option 5 resets the hosts to the windows default
Choosing option 4 resets the hosts to the version before updating the hosts file with that tool

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.