spicyjack / app-idgamessync Goto Github PK
View Code? Open in Web Editor NEWCreate/update a mirror of the idgames repository.
Create/update a mirror of the idgames repository.
Script could query the status page in order to get a list of "current mirrors"
The "downloaded file size doesn't match" warning should only be shown after an actual file is downloaded.
Trap Ctrl-C so temp download files can be removed before exiting
The other mirrors may not have all of the files in /newstuff when the script is run, which causes 404 errors
Don't sync files in /incoming, don't list them as well unless --incoming is turned on(?) (don't show in output reports, as well as don't try downloads from incoming unless --incoming is used)
Any file that is generated on the master mirror, such as ls-laR.gz, fullsort.gz, LAST.* should be downloaded only from the master mirror, as the size of those files in the ls-laR.gz file won't be the same on other mirrors until those mirrors are 100% synchronized to the master, which in reality may never happen (because submissions are always being accepted)
Allow the user to only keep a copy of all of the WAD files, not things like graphics
, idstuff
, sounds
, etc. See idGAS/docs.git/idgas_notes.md
for a full list of directories
"unknown" in this case means "not a file, not a directory"
Reading debug logs is a pain with multiple BUILD methods being called. Maybe create new methods with more unique names, or add the full module name to the debug log output
Scan /newstuff on the local system, and compare to /newstuff in the archive, and delete any files that don't exist in the archive copy
When downloading files and the server returns HTTP error 403, the caller to the fetch method should not compare the downloaded file (which wasn't actually downloaded) with what's in the archive.
The number of files that are output as needing to be synced from a mirror and the number of files that are shown in the summary at the end of the filelist as needing to be synced are different when --dry-run
is called. This may be related to #23
So people can add unofficial mirrors in order to spread the load around
Set a maximum amount of threads, say one or two per server. Adding this feature would also require managing threads somehow
When getpwuid/getgrgid return undef, this causes the validation for the local file object to fail for user/group attributes.
Moose may be better than both Mouse or Moo, because it will be more available. Moo would be good because it doesn't have as many dependencies as Mouse or Moose.
When running idgames_mirror.pl
using the --dry-run
switch, keep track of the number of files that are different between the local system and what's shown in ls-laR.gz
, and summarize the differences when the script exits
Change the current files synced/bytes synced
fields to say files to be synced/bytes to be synced
Make the report types "size" and "local" (show files when size differs, show files missing from the local system) the default when running idgames_sync.pl.
This way, it takes less command line switches to get what is most likely the desired behaivor.
Users can get the full report by using --show-all or report type "all" or something like that.
If the ls-laR.gz file has already been downloaded using --update-ls-lar, there's no need to download it a second time. Add a switch like --no-sync or --no-update-ls-lar to cause the script to skip updating this file
Add a 'script' report format, for use by other scripts, up to and including a GTK front end
The 'ls-laR.gz' file is downloaded when the script first starts. If the download fails, the stat call is performed against the undefined filehandle, which throws a warning. Move the call to stat the downloaded file to be after the check for defined-ed-ness of the filehandle for the downloaded file.
Logging already has sub names as part of the output, so having the sub names in the log messages is redundant
Add "strict" checking of file/directory sizes/dates; differing dates and permissions will also fail a test; it's actually hard to check for different size directories, as the directory size between different filesystems (archive filesystem and local filesystem) will only be the same size when both machines have the same OS, OS Version and disk sizes
Windows only, or can you create static binaries for *NIX too?
Use PAR? Pinto?
"filename" is misleading, as it's a path from the archive root directory, not just a filename.
Narrow the scope of debugging output to only show the relevant debug output
This would be used mostly for the /newstuff directory
When a file is downloaded from a mirror, the size of the downloaded file is not checked against what's in ls-laR.gz
It won't be possible to back those changes out, but at least you'll have a log of what happened.
There's not a lot of handling of Windows file paths in the script, most likely things will break badly when the script is run under Windows
Use the original download file's name as part of the tempfile name, it could help identify download issues
Also check to see if this happens in the "full" and "simple" output formats as well. Check in Reports->write_record, verify report_types is doing the correct thing
Split packages into individual files, so that way, packages can be loaded and tested by package, instead of trying to herk around with the entire idgames_sync.pl script. It's easier to test individual files than to try and require a file in order to test packages inside of it. Add a Makefile
rule that can combine all of the different libraries back together into a single script
Example: sub foo {} # sub foo
It's extra noise
Since the mirror server(s) and the local system are most likely in different timezones, use a 72 hour window when comparing dates on files in the archive and files on the local system, in addition to the sizes of the files; if the window on the dates of both files is within 72 hours, and the size of the files are the same, don't sync them
The file to be fetched is already displayed in the output when fetching a file, so the full URL is kind of redundant. Use a regex to just grab the host part of the URL
my $regex = qr!^(ftp|http|https){1}://([\w.-]+)/(.*)$!;
works, but turn off the first capture
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.