spicyjack / app-idgamessync Goto Github PK

Script could query the status page in order to get a list of "current mirrors"

The "downloaded file size doesn't match" warning should only be shown after an actual file is downloaded.

Trap Ctrl-C so temp download files can be removed before exiting

The other mirrors may not have all of the files in /newstuff when the script is run, which causes 404 errors

Don't sync files in /incoming, don't list them as well unless --incoming is turned on(?) (don't show in output reports, as well as don't try downloads from incoming unless --incoming is used)

Any file that is generated on the master mirror, such as ls-laR.gz, fullsort.gz, LAST.* should be downloaded only from the master mirror, as the size of those files in the ls-laR.gz file won't be the same on other mirrors until those mirrors are 100% synchronized to the master, which in reality may never happen (because submissions are always being accepted)

Allow the user to only keep a copy of all of the WAD files, not things like graphics, idstuff, sounds, etc. See idGAS/docs.git/idgas_notes.md for a full list of directories

"unknown" in this case means "not a file, not a directory"

Reading debug logs is a pain with multiple BUILD methods being called. Maybe create new methods with more unique names, or add the full module name to the debug log output

Scan /newstuff on the local system, and compare to /newstuff in the archive, and delete any files that don't exist in the archive copy

When downloading files and the server returns HTTP error 403, the caller to the fetch method should not compare the downloaded file (which wasn't actually downloaded) with what's in the archive.

The number of files that are output as needing to be synced from a mirror and the number of files that are shown in the summary at the end of the filelist as needing to be synced are different when --dry-run is called. This may be related to #23

So people can add unofficial mirrors in order to spread the load around

Set a maximum amount of threads, say one or two per server. Adding this feature would also require managing threads somehow

When getpwuid/getgrgid return undef, this causes the validation for the local file object to fail for user/group attributes.

Moose may be better than both Mouse or Moo, because it will be more available. Moo would be good because it doesn't have as many dependencies as Mouse or Moose.

When running idgames_mirror.pl using the --dry-run switch, keep track of the number of files that are different between the local system and what's shown in ls-laR.gz, and summarize the differences when the script exits

• Number of files that are different
• Approximate size of data to be downloaded

Change the current files synced/bytes synced fields to say files to be synced/bytes to be synced

Make the report types "size" and "local" (show files when size differs, show files missing from the local system) the default when running idgames_sync.pl.

This way, it takes less command line switches to get what is most likely the desired behaivor.

Users can get the full report by using --show-all or report type "all" or something like that.

If the ls-laR.gz file has already been downloaded using --update-ls-lar, there's no need to download it a second time. Add a switch like --no-sync or --no-update-ls-lar to cause the script to skip updating this file

Add a 'script' report format, for use by other scripts, up to and including a GTK front end

The 'ls-laR.gz' file is downloaded when the script first starts. If the download fails, the stat call is performed against the undefined filehandle, which throws a warning. Move the call to stat the downloaded file to be after the check for defined-ed-ness of the filehandle for the downloaded file.

Logging already has sub names as part of the output, so having the sub names in the log messages is redundant

Add "strict" checking of file/directory sizes/dates; differing dates and permissions will also fail a test; it's actually hard to check for different size directories, as the directory size between different filesystems (archive filesystem and local filesystem) will only be the same size when both machines have the same OS, OS Version and disk sizes

Windows only, or can you create static binaries for *NIX too?

Use PAR? Pinto?

"filename" is misleading, as it's a path from the archive root directory, not just a filename.

Narrow the scope of debugging output to only show the relevant debug output

This would be used mostly for the /newstuff directory

When a file is downloaded from a mirror, the size of the downloaded file is not checked against what's in ls-laR.gz

It won't be possible to back those changes out, but at least you'll have a log of what happened.

There's not a lot of handling of Windows file paths in the script, most likely things will break badly when the script is run under Windows

Use the original download file's name as part of the tempfile name, it could help identify download issues

Also check to see if this happens in the "full" and "simple" output formats as well. Check in Reports->write_record, verify report_types is doing the correct thing

Split packages into individual files, so that way, packages can be loaded and tested by package, instead of trying to herk around with the entire idgames_sync.pl script. It's easier to test individual files than to try and require a file in order to test packages inside of it. Add a Makefile rule that can combine all of the different libraries back together into a single script

Example: sub foo {} # sub foo

It's extra noise

Since the mirror server(s) and the local system are most likely in different timezones, use a 72 hour window when comparing dates on files in the archive and files on the local system, in addition to the sizes of the files; if the window on the dates of both files is within 72 hours, and the size of the files are the same, don't sync them

The file to be fetched is already displayed in the output when fetching a file, so the full URL is kind of redundant. Use a regex to just grab the host part of the URL

my $regex = qr!^(ftp|http|https){1}://([\w.-]+)/(.*)$!;

works, but turn off the first capture