corionis / els Goto Github PK

Tools and Jobs are intertwined and Jobs may use other Jobs. To avoid odd exceptions and make defining them easier add some sanity checks and handling in both Jobs and Tools.

• In Tools:

  • Detect Job dependencies on Tool delete
  • Auto-change Job dependencies on Tool rename, w/ prompt
  • Add flag for Origins enabled or disabled

• In Jobs:

  • Detect dependencies on Job delete
  • Auto-change dependencies on Job rename, w/ prompt
  • Detect infinite loops on task add
  • Use Origin enabled or disabled flag
  • Origin paths should be disabled for all Operation tools.

If a Hard Drive in the system goes offline for whatever reason. ELS would try and replace all the files that are on the offline drive.
Since ELS is automated, it could use all the data you have for the month.

Maybe we can have a limit to how much it will transfer in one session.
If the limit gets hit - email the admin.

Abstract the I/O operations and refactor to implement Internet-based I/O operations for one end of mung operations.

This will require reorganization of the code for scanning and other operations that currently expect all media to be locally connected.

The separate targets.json file allows for finer control of where new content is backed-up. But under some circumstances it's just a duplicate of the subscriber.json sources.

When combined with #16 "Add more granular control of target minimum free space" to rearrange how individual drives are handled this idea could simplify ELS configuration with a new option to use the sources as the targets.

Several types of subscriber-side errors are not communicated to the publisher. Can a mechanism be added to provide that information in a manageable fashion?

The current linear search technique was fine with a small collection. However once the collection grows the item look-ups start becoming more of a problem in performance.

Consider refactoring how library items are handled. Possibly use some form of hash table to increase look-up performance.

New ELS 4.0 code for the Navigator is (mostly) internationalized.

Go through code prior to 4.0 and standardize and externalize all remaining user-facing strings.

Currently the back-up process assembles groups of files to be copied, then checks available disk space for that total size. However, as each item is copied additional checks use that same size.

Change the handling to reduce the total size checked by the size of each item completed.

Add a Renamer to Navigator Tools.

Types:

• Case change
• Insert
• Numbering
• Remove
• Replace

A bug is causing communications to fail and hang with a remote subscriber if that subscriber takes longer than 5 minutes to scan its collection. This bug has been addressed but those changes obviously did not fix the issue.

To allow different ports to be used for different purposes - such as one for scheduled operations and another for interactive operations - add a command line parameter to override the port in the JSON file.

This may require a --long-name due to running out of single-letters.

Version 2.1.1 remote mode copy performance is slow. It is about 20% of nominal expectation or more.

The file transfer is handled by Apache Mina. Research Mina library issues and solutions.

Currently ELS only matches against filenames. Now that the "preserve file dates" option works add another to match file dates when deciding what needs to be backed-up. This would make replacing or updating an item easier.

Add an Empty Directory Finder to Navigator Tools.

• Control to delete items

The current implementation of minimum free space is based on the library. However, a library can span multiple storage devices.

Consider changing targets handling for a better approach.

One idea is to add a Devices section something like:

"devices": [
    {
        "device": "/home/plex/Plex/Media/hd01",
        "minimum": 50GB
    },
    {
        "device": "/home/plex/Plex/Media/hd02",
        "minimum": 100GB
    }
]

To support the installers to be built embed the Java Runtime Environment (JRE) for each installer.

• Windows
• Linux Debian
• Linux RedHat/CentOS
• Macintosh

There is already the -l | --library option to specific one or more libraries to back-up.

The opposite might be handy. Add option -L | --exclude to skip one or more libraries during a run.

Implement the Libraries tab in Nabigator.

This tab will allow creation and editing of the JSON libraries file that contains the parameters for an ELS collection of libraries.

There is a minor operational bug with the Navigator when a Hint is created. The Navigator currently marks the Hint for the system it's created on as For instead of Done. That means a separate publisher Hint run is required to update the Hint from For to Done. But that is not necessary if the operation has already been completed.

Add a built-in updater to check a version file on GitHub, perform the download and execute the installer.

• Will need to add the HTTPS protocol to handle the download
• Write update processes to work from command line
• Navigator menu option "Check for updates" runs the same methods

Currently ELS daemons - subscribers, publisher listeners and any Hint Server - must be running before starting remote command line or Navigator operations.

Add a dynamic "jump-up on-demand" launch mechanism for daemons.

• In Linux add support for inetd.
• In Windows add support for its launch technique.

Will require time-out adjustments for the handshake protocol.

When ELS is done a FATAL message is logged with "Process completed normally". FATAL is used so it is always logged regardless of the level parameter.

However, some kinds of exceptions are not caught and it appears to complete normally in the log.

Catch and track exceptions better. Especially basic file I/O exceptions when operating remotely.

The terminal_allowed value controls whether an interactive (manual) terminal is allowed to connect to one of the listeners. However, the value is never used (oops). Add logic to not allow a terminal connection if terminal_allowed is false.

Add support for external tools to Navigator Tools.

• Replaceable parameters of:
  • Browser selection(s)
  • Other ELS and Navigator settings and preferences
• User-defined prompt dialog
  • Gather all input-type replaceable parameters into one dialog
  • Optionally save responses

Both dry run and backup operations fail if the subscriber's collection is empty.

Local with perform an automatic scan. Remote throws an exception.

Rework so an empty (remote) collection does not throw the exception and allows the process to continue.

With release 3.0.0 when using no filename value after a -t | -T option ELS throws an exception.

When a library has several sources, and one is below the defined minimum free space, an error occurred saying there was no available space on any target for that library.

This appeared to be working during tests. Not sure if the one file at issue had a problem itself. Copied that using Filezilla instead and ELS worked from there on.

Requires testing and research.

The back-up process checks disk space prior to performing the operation. Add a similar disk space check for manual operations in the Navigator. Present an appropriate dialog if there is not enough space.

Implement the Operations tab in Navigator.

This tab will allow creating named ELS configurations similar to other Navigator Tools. The tool may also be used as a step in a Job.

• Show all appropriate options for automated back-up operations.
• Save as named configurations in a tool structure.

The free disk space check, when operating locally with publisher and subscriber drives connected to one system, is failing under some unknown condition. This is causing a drive to be filled to zero bytes free.

Research is needed. It seems the physical directory name has to match the library name, but that should not be an issue.

When the Preferences file is missing or has been deleted Navigator dialogs are displayed at 0, 0. Changed the default placement to center the dialog on the Navigator frame.

While testing the Renamer Tool discovered a design issue. When combining multiple tools in a Job, such as Renamer tools, the result of previous step(s) cannot be used by subsequent steps. Because Job tasks require specific origins this breaks the task design paradigm.

Design changes are required.

The Renamer itself may be used manually with the Run button.

Add a Duplicate Finder to Navigator Tools.

• Option for cross-library checking
• Controls to open, copy, move and delete items

During development ELS has always had content on the subscriber's storage devices.
However a new installation will be empty.

Test ELS with a completely empty subscriber and adjust accordingly.

Update all documentation with details of all new capabilities, options, usage information with examples. This includes the wiki and embedded help.

A sleep tool is needed to coordinate timing between multi-step Job operations where the publisher/subscriber roles change. This also helps support time differences between machines.

Simple Tool dialog with a field for length of time to sleep. Minute granularity may be good enough.

It appears a run with Hints will hang if the publisher has not processed the Hints first. This should produce a warning message then stop. Double-check command line fault handling in this case.

Something is causing the final display of summary information after an operation to throw an exception. The operation completes normally but the info display blows-up at the end.

It appears to happen when the -l or --library option is used.

When a Library is selected for a New Folder operation the list of defined Sources for that Library is displayed if there is more than one Source. Add displaying the free disk space for each Source.

ELS uses the concept of file "group" which is simply all the files from a particular content directory, e.g. a movie or TV show. It gathers the list of files in a group, including their sizes, then decides where to copy the group.

Currently when checking for available disk space the total group size is used.

Add a command line option to allow the use of the item size instead of total group size. That way something like a TV show would fill a storage device more closely to its minimum and roll-over to the next storage device accordingly when new episodes are added.

Some people may not want or have difficulty installing the JRE 8 used by ELS.

Consider adding 3 new distributions that include the JRE for the supported operating systems, and add logic to the scripts and batch files to check for the existence of an embedded JRE to execute ELS.

Create a user-oriented web site using GitHub Pages.

Although fancy web sites are nice GitHub Pages was chosen in the hope this project will continue and grow beyond the original authors.

Add the SMTP protocol and an Email tool in Navigator.

Used for sending completion status of Jobs with options for including:

• Runtime logs
• Mismatches file
• What's New file
• Fault messages

Encrypt saved credentials.

There is a problem during the automated handshake when using --remote M for a manual terminal to a subscriber listener.

Add the item size to the logger line when copying a file. For basic visual reference when running.
Write a new method for scaled sizes to return the largest one only - to shorten the logger line from the current long bytes, kb, mb, gb, tb approach.

To promote the use of ELS build installers for all supported environments.

• Windows
• Linux Debian
• Linux RedHat/CentOS
• Macintosh

Adding the Navigator changed some strings that caused path separator handling to be broken due to case differences.

To solve the "odd man out" issue when ELS Hints are used with more than one back-up a new status server is needed to be the one place any publisher or subscriber can query for hint status. This will enable all involved to automatically delete a hint file when all have executed it.

To make ELS and the Navigator easier to install and manage add a command-line option to set the location and name of the configuration directory. Currently the location is fixed at $HOME/.els/ which will remain the default.

This will also facilitate adding test tool and job configurations to the project.

Use -C | --config for this option.

With the addition of the Navigator users have all the power to shoot themselves in the foot :)

To support the various ways of using the Navigator for additions only, managing a subset of collection libraries, etc. add an authorization tool, per user, to control who may read/write/delete items and in which libraries.