Sync [options] /data/local/music dav:https://my.cloud-space.com/data/music

--option1 option1_value --option2 option2_value

Sync --log path/to/log

Sync -l path/to/log

Sync srcUri destUri --help

Sync -h

Sync /data/local/music dav:https://my.cloud-space.com/data/music --help

Sync --file sync_params.txt /path/source /path/dest

Sync --actions actionCreate,actionOverride --filter-create exclude:*.tmp /path/source /path/dest

This is the most basic and "natural" structure type. It can be used for instance to mirror a directory structure on the local hard disk to an external hard disk or a network share.

To specify such a structure, just pass the (OS-specific) path to the root directory without any prefix. The table below lists the additional options that are supported. (Remember that these options need to be prefixed with either src- or dst- to assign them to the source or destination structure.)

It is possible to sync from or to a directory hosted on a WebDav server. To do this, the full URL to the root directory on the server has to be specified with the prefix dav: defining the structure type. The following table lists the additional options supported for WebDav structures. (Remember that these options need to be prefixed with either src- or dst- to assign them to the source or destination structure.)

In addition to these options, the mechanism to authenticate with the server has to be defined. Refer to the Authentication section for more information.

Notes

Using WebDav in sync operations can be problematic as the standard does not define an official way to update a file’s last-modified time. Files have a getlastmodified property, but this is typically set by the server to the time when the file has been uploaded. For sync processes it is, however, crucial to have a correct modification time; otherwise, the file on the server would be considered as changed in the next sync process because its timestamp does not match the one of the file it is compared against.

Concrete WebDav servers provide different options to work around this problem. Stream Sync supports servers that store the modification time of files in a custom property that can be updated. The name of this property can be defined using the modified-property option. As WebDav requests and responses are based on XML, the custom property may use a different namespace than the namespace used for the core WebDav properties. In this case, the modified-namespace option can be set.

When using a WebDav directory as source structure Stream Sync will read the modification times of files from the configured modified-property property; if this is undefined, the standard property getlastmodified is used instead.

When a WebDav directory acts as destination structure, after each file upload another request is sent to update the file’s modification time to match the one of the source structure. Here again the configured property (with the optional namespace) is used or the standard property if unspecified.

Most Windows users will have a Microsoft account and thus access to a free cloud storage area referred to as OneDrive. For Windows there is an integrated OneDrive client that automatically syncs this storage area to the local machine. For Linux, however, no official client exists.

Stream Sync supports a OneDrive storage as both source or destination structure of a sync process. The storage is identified by using a URL of the form onedrive:<driveID> where driveID is a string referencing a specific Microsoft OneDrive account. In addition, the following special command line options are supported:

OneDrive uses OAuth 2 as authentication mechanism with a special identity provider from Microsoft. Therefore, the corresponding credentials have to be setup (refer to the OAuth 2 section for further information). This requires a bunch of preparation steps before sync processes can be run successfully. The example Sync from a local directory to Microsoft OneDrive contains a full description of the steps necessary.

Another popular cloud storage offering is available from Google: On a Google Drive account users can store information up to a certain limit. Most users of Android will have such an account. As is true for Microsoft OneDrive, official sync clients are not available for all operation systems.

Stream Sync can handle a Google Drive account as both source and destination of a sync process. To access such an account, use a URL of the form googledrive:<path>, where path is the optional root path of the sync process. If it is missing, the special root folder of the Google Drive account is used; otherwise, only the path specified here is taken into account by sync operations. Note that there is no such thing like an account ID in the URL; the account to be accessed is encoded in the OAuth 2 access token, which is used for authentication (the OAuth 2 section contains more information about this topic).

One speciality of Google Drive is that this file system is not strictly hierarchical. A single file or folder can have multiple parents, and it is possible that a folder can have multiple children with the same name. Thus, a path like documents/private/MyText.doc does not necessarily uniquely identify a single element. Even cycles in folder structures are possible. Stream Sync does not handle such scenarios. It treats Google Drive like any other folder structure and assumes the same properties. So when using Stream Sync together with Google Drive, you should make sure that at least the sub path to be synced follows the conventions of a strictly hierarchical file system.

Other than the root path to be synced in the target Google Drive account - which is part of the structure URL - you typically do not have to specify any further configuration options.

You can find a complete example how to set up Stream Sync for accessing a Google Drive account in the section Sync from a local directory to Google Drive.

The easiest authentication mechanism is Basic Auth, which requires that a user name and password are provided. This information is then passed to the server in the Authorization header. (Therefore, this mechanism makes only sense when HTTPS is used for the server communication.)

To make use of Basic Auth, just define the command line options user and password. Note that these options have to be prefixed with src- or dst- to assign them to either the source or destination structure. Examples how to use these options can be found in the Examples section, for instance under Sync from a local directory to a WebDav directory.

OAuth 2 is another popular way for authentication. Stream Sync supports the Authorization code flow. In this flow the authentication is done by an external server, a so-called identity provider (IDP). In a first step, an authorization code is retrieved. In this step, the user basically grants Stream Sync the permission to access her account with a set of pre-defined rights. This is done by opening a Web page at a URL specific to the IDP in the user’s Web browser. The user then authenticates against the IDP, e.g. by filling out a login form or using another means. If login is successful, the IDP invokes a so-called redirect URL and passes the authorization code as a query parameter.

In a second step, the authorization code has to be exchanged against an access token. This is done by calling another endpoint provided by the IDP and passing the authorization code as a form parameter. If everything goes well, the IDP replies with a document that contains both an access token and a refresh token. The access token must be passed in the Authorization header for all requests sent to the target server. Its validity period is limited; when it expires, the refresh token can be used to obtain a new access token. The refresh token is typically valid for a longer time; so the user has to do the login (i.e. the first step) only once, and then Stream Sync can access the target server as long as the refresh token stays valid.

The authorization code flow is interactive; it requires that the user executes some actions in a Web browser. This is not a great fit for a command line tool like Stream Sync. To close this gap, in addition to the main class of Stream Sync, there is a second CLI class responsible for the configuration and management of OAuth identity providers: com.github.sync.cli.oauth.OAuth.

What this class basically does is updating a storage with information about known IDPs: First, an IDP has to be added to the system. In this step a number of properties for this IDP has to be provided, such as the URLs to specific endpoints or the client ID and secret to be used for the interaction with the IDP. For this purpose, the init command is used. An example invocation could look as follows:

The command supports the following options:

After the execution of this command, the IDP-related information is stored under the path specified, but no access token is retrieved yet. This is done using the login command as follows:

The parameters correspond to the ones of the init command; encryption is supported in the same way. (If an encryption password has been specified to the init command, the same password must be entered here as well.)

The login command does the actual interaction with the IDP as required by the authorization code flow. It tries to open the standard Web browser at the authorization URL configured for the IDP in question. If this fails for some reason, a message is printed asking the user to open the browser manually and navigate to this URL. The Web page served at this URL is under the control of the IDP; it should give the relevant instructions to do a successful authentication, e.g. by filling out a login form. If this is the first login attempt, the user is typically asked whether she wants to grant the access rights defined by the scope parameter to this client application. If authentication is successful, the IDP then redirects the user’s browser to the redirect URL. Depending on the configured redirect URL, there are two options:

• If the redirect URL is of the form http://localhost:<port>;, the command opens a small HTTP server at the configured port and waits for the redirect. It can then obtain the authorization code automatically without any further user interaction.

• For other types of redirect URLs, the user is responsible to extract the code; for instance from the URL displayed in the browser’s address bar. The command opens a prompt on the console where the code can be entered.

If everything goes well, the command creates a new file in the specified storage path with the access and refresh tokens obtained from the IDP; the file is optionally encrypted.

With this information in place, Stream Sync can now be directed to use this IDP for authentication. To do this, the user and password options used for basic auth have to be replaced by ones pointing to the desired IDP:

Note how, analogous to the OAuth commands, the IDP is referenced by its name and the path where its data is stored; the encrypt-idp-data and idp-password options are supported as well.

With one final OAuth command the data of a specific IDP can be removed again:

This command deletes all files for the selected IDP in the path specified. As the files are just deleted, no encryption password is required here.

As is true for the main Sync application, the OAuth application offers the switch --help (or its short form -h) to explicitly request usage information. To get a general help screen, just enter:

To request help information specific to a concrete command, also provide this command, for instance:

Sync --throttle 1 ...

Sync --throttle 45 --throttle-unit minute ...

-Dakka.http.host-connection-pool.max-connections=1 -Dakka.http.host-connection-pool.max-open-requests=2

Sync C:\data\work dav:https://sd2dav.1und1.de/backup/work --timeout 600

Section Log files described the usage of the --log option to produce a protocol of the operations executed during a sync/mirror process. While such a log file is meaningful on its own, for mirror processes it can serve an additional purpose:

It is possible to use such a log file as input for another mirror process. Then the sync operations to be executed are not calculated as the delta between two structures, but are directly read from the log file. This is achieved by specifying the --sync-log option whose value is the path to the log file to be read. Note that in this mode still the URIs for both the source and destination structure need to be specified; log files contain only relative URIs, and in order to resolve them correctly the root URIs of the original structures must be provided.

If the structures to be synced are pretty complex and/or large files need to be transferred over a slow network connection, sync processes can take a while. With the support for log files this problem can be dealt with by running multiple incremental mirror processes. This works as follows:

1. An initial mirror process is run for the structures in question that has the --log option set and enables Dry-run mode. This does not execute any actions, but creates a log file with the operations that need to be done.

2. Now further mirror processes can be started to process the sync log written in the first step. For such operations the following options must be set:

   • --sync-log is set to the path of the log file written in the first step.

   • --log is set to a file keeping track on the progress of the overall operation. This file is continuously updated with the sync operations that have been executed.

The mirror processes can now be interrupted at any time and resumed again later. When restarted with these options the process ignores all sync operations listed in the progress log and only executes those that are still pending. This is further outlined in the Examples and use cases section.

In the incremental mode, as described above, the error log file has no further function than reporting errors. Sync operations that appear in the error log are not written to the normal log and are not considered to be completed. So when running another mirror process from the sync log, these operations are retried (and if they fail again, they are written anew to the error log).

The typical use case for Stream Sync in mirror mode is transferring data from one system - the leading system - to another data structure; the destination structure gets modified to become a clone of the original system. From time to time you may need to run a mirror process in the inverse direction.

Consider for example that you use Stream Sync as a backup tool. If you mess up with your original data, you will probably want to replace it from the backup storage. This is of course easily possible: you just have to rewrite the sync command you use for your backup to work in the opposite direction. This can be done rather mechanically; the source and destination URIs have to be exchanged, as well as the src- and dst- prefixes of the parameters that configure your data structures.

Sync commands tend to be become complex; you often need a bunch of parameters to configure authentication and fine-tune the transfer process. Maybe you have therefore written shell scripts that contain your sync commands. In the backup scenario, you would have a shell script that triggers your backup. To restore your data from the backup structure, you could create a restore script using the replacements outlined above. This solution is not ideal, however, because you now have to maintain two scripts that need to be kept in sync.

For such use cases, Stream Sync offers an easier solution: it supports the --switch parameter, which swaps the source and destination structures, effectively reversing the sync direction. This means, you do not have to duplicate your commands or scripts, but simply add a parameter to switch the sync direction.

If you use shell scripts to store your sync commands, you should write them in a way that they support additional parameters. For instance, if your backup script looks as follows:

Add the special parameter "$@" at the end, which represents all the parameters entered by the user:

You can now transform your backup script to a restore script by simply adding the --switch parameter:

As has been shortly mentioned in the Sync mode section, Stream Sync manages a file with information about the local state for each sync stream. Based on this file, it can detect local updates and compute the changes to be applied to the remote structure (or recognize conflicting changes). A sync stream is identified by its local and remote sources; for each combination of a local and a remote source, a separate state file is created.

Per default Stream Sync can manage these state files transparently without user interaction. They are stored in a subfolder named .stream-sync in the current user’s home directory and have a (non-readable) name derived from the URIs of the local and remote structures. (Actually, the name is computed by concatenating the local and the remote URI, calculating a SHA-1 hash on this string, and applying a Base64-encoding on the result; but this is merely an implementation detail.)

While these defaults should work well in most cases, they can be overridden with some command line options:

• --state-path allows specifying the path in which the state file is created. Here the user can provide an arbitrary directory. The path will be created if it does not exist.

• --stream-name can be used to set a name for the sync stream. The state file is then given this name instead of the cryptic auto-generated one.

The following fragment shows a usage example of these options:

Every run of a sync process updates the local state file associated with the stream. For the initial execution of the stream, a state file does not exist yet. This is no problem if one of the structures taking part in the sync process is empty and will be initialized from the other side. Then the initial sync run actually becomes a mirror: Stream Sync copies all the files found in the existing structure to the empty one and writes an up-to-date local state file automatically.

If there is already data on both sides, however, there should better be a valid local state file available before running a first sync process. Otherwise, Stream Sync considers all local files as newly created and will treat changes on remote files as conflicts. To avoid this, you should create a clean local state file that reflects the current state of the local structure. This is achieved by adding the --import-state switch to the command line. The switch enables a special mode, in which only the local structure is iterated over, and all files encountered are recorded in the state file. Afterwards, a fresh and up-to-date state file exists. A (re-)import of the local state can also be done if the state file got corrupted for whatever reason.

For the example sync stream from the previous section, an import command could look as follows:

This should be a frequent use case, in which some local work is saved on an external hard disk. The command line is pretty straight-forward, as the target drive can be accessed like a local drive; e.g. under Windows it is assigned a drive letter. The only problem is that if the file system on the external drive is FAT32, it may be necessary to explicitly specify a time zone in which last-modified timestamps are interpreted (refer to the description of local directories for more information). For this purpose, the time-zone option needs to be provided. In addition, the ignore-time-delta option is set to a value of 2 seconds to make sure that small differences in timestamps with a granularity below seconds do not cause unnecessary copy operations.

Consider the case that a directory structure stores the data of different projects: the top-level folder contains a sub folder for each project; all files of this project are then stored in this sub folder and in further sub sub folders.

On your local hard-disk you only have a subset of all existing projects, the ones you are currently working on. On a backup medium all project folders should be saved.

Default sync processes are not suitable for this scenario because they would remove all project folders from the backup medium that are not present in the source structure. This can be avoided by using the min-level filter as follows:

This filter statement says that on the top-level of the destination structure no remove operations are executed. For the example at hand the effect is that folders for projects not available in the source structure will not be removed. In the existing folders, however, (which are on level 1 and greater) full sync operations are applied; so all changes done on a specific project folder are transferred to the backup medium.

As described under Sync log files, with the correct options mirror processes can be stopped at any time and resumed at a later point in time. The first step is to generate a so-called sync log, i.e. a file containing the operations to be executed to sync the structures in question:

This command does not change anything in the destination structure, but only creates a file /data/sync.log with a textual description of the operations to execute. (Such files have a pretty straight-forward structure. Each line represents an operation including an action and the element affected.)

Now another mirror process can be started that takes this log file as input. To keep track on the progress that is made, a second log file has to be written - the progress log:

This process can be interrupted and later started again with the same command line. It will execute the operations listed in the sync log, but ignore the ones contained in the progress log. Therefore, the whole sync process can be split in a number of incremental sync processes.

The following command can be used to mirror a local directory structure to an online storage:

Here all options supported by the WebDav structure type are configured. The server (which really exists) does not allow modifications of the standard WebDav getlastmodified property, but uses a custom property named Win32LastModifiedTime with the namespace urn:schemas-microsoft-com: to hold a modified time different from the upload time. This property will be set correctly for each file that is uploaded during a sync process.

Note that the --dst-password parameter could have been omitted. Then the user would have been prompted for the password.

Building upon the previous example, with some additional options it is possible to protect the data on the WebDav server using encryption:

This command specifies that both the content and the names of files are encrypted using the password "s3cr3t" when copied onto the WebDav server. With an encryption mode of files only the files' content would be encrypted, but the file names would remain in plain text. The size of the cache for encrypted names is increased to avoid unnecessary crypt operations. In the example the number of sync operations per second is limited to 2 to avoid that the server rejects requests because its load is too high. Also, a larger timeout has been set (600 seconds = 10 minutes), so that uploads of larger files will not cause operations to fail.

As described in the Microsoft OneDrive section, some preparations are necessary before OneDrive can be used as source or destination structure of a sync process. These are mainly related to authentication because an OAuth client for the Microsoft Identity Provider (IDP) has to be registered and integrated with Stream Sync.

As a first step, the OAuth client application has to be created in the Azure Portal. The application is assigned a client ID and a client secret and is then able to interact with the Microsoft IDP to obtain valid access tokens. Note that if Stream Sync was a closed source application, it could have been registered as a client application and be shipped with its client secret. But because the full source is available in a public repository, such a registration cannot be done; the client secret would not be very secret, would it?

The steps necessary to create a client application are described in detail in the official Microsoft documentation under OneDrive authentication and sign-in. Here we will give a short outline.

Log into the Microsoft Azure Portal and navigate to the page for App registrations. Here you can create a new application. You are then presented a form where you can enter some data about the new application. Choose a name and select the type of accounts to be supported. You also have to enter a redirect URI, which will be invoked by the Microsoft IDP as part of the code authorization flow. It is up to you, which redirect URI you choose; if you intend to run sync processes on your personal machine, it is recommended to use a URI pointing to localhost with a port number that is not in use on your computer, such as http://localhost:8080. This simplifies the integration with Stream Sync as described below.

After all information has been entered, the app can be registered. The app is then assigned an ID that is displayed in the overview page. On the certificates and secrets page, you can request a new client secret. Copy this secret, it is required later on.

Next you have to add the information about your OAuth client application to Stream Sync. This is done with some command line operations. For the following steps we assume that you have defined some environment variables that are referenced in the commands below:

With a first command, basic properties of the client application are specified:

Here we use the name microsoft to reference this IDP and a localhost redirect URI. The other options, the URLs and the scope values, are defined by the OneDrive API and must have exactly these values. This command will prompt you for a password for the IDP; sensitive data in the token directory is encrypted with this password. (If you do not want the files to be encrypted, add the option --encrypt-idp-data false.)

Now we can do a login against the Microsoft IDP and obtain an initial pair of an access and refresh token:

This command will open your standard Web browser and point it to the authorization URL of the Microsoft IDP. You are presented a form to enter the credentials of your Microsoft account. You are then asked whether you want to grant access to your client application. Confirm this.

Because we have used a redirect URI of the form http://localhost:<port>; the authorization code can be obtained automatically, and the command should finish with a message that the login was successful. (For other redirect URIs you have to determine the code yourself and enter it at the prompt in the console.)

After completion of these steps, Stream Sync has all the information to authenticate against your OneDrive account. So you can run a sync process. One piece of information you still need is the ID of your OneDrive account. This can be obtained by signing in into the OneDrive Web application. The browser’s address bar shows a URL of the form https://onedrive.live.com/?id=root&cid=xxxxxx. The ID in question is the alphanumeric string after the cid parameter. We assume that you create an environment variable DRIVE_ID with this value.

The following command shows how the local work directory can be synced against the data folder of your OneDrive account:

Of course, you can use other standard options as well, for instance for setting timeouts, configuring encryption or set filters. The following example uses the same options as the one in the section about WebDav and encryption:

The steps to set up Stream Sync for an integration with Google Drive are very similar to the ones described in the OneDrive example. Specifically, an application needs to be created in the Google Cloud Platform Console, in order to obtain the credentials (the OAuth client ID and secret) required for the authentication with Google’s OAuth identity provider. As the OneDrive example covers the basics in detail, this section will focus mainly on the differences between these cloud storage providers.

Documentation about the process can be found in the Official Google documentation. Here is a short summary:

At first, a new project has to be created in the Google Cloud Platform Console. With this new project selected, under Credentials click CREATE CREDENTIALS and select OAuth Client ID. Set the Application type to Desktop app and enter a name for the new client. After the successful creation of the OAuth client, the web application will present its client ID and secret. In contrast to Microsoft’s OAuth implementation, no redirect URL needs to be specified when selecting Desktop app as client type. We can use a local redirect URL later when interacting with the identity provider.

With the OAuth client ID and secret available, Stream Sync can now be configured with the details of this client. This can be done using the following command:

The next step is a login against the Google identity provider. It can be triggered with the command below:

The command opens a web browser and navigates to a login page served by the Google OAuth identity provider. The account you select for the login will be the one that is later accessed by Stream Sync. You have to confirm that you grant access to the application you have created before. After a successful login, Stream Sync should be able to obtain the OAuth tokens and store them locally in the configured path.

You can now run sync processes using your Google Drive account as source or destination structure. For instance, the following command syncs the folder /data/google against the full content stored in your Google Drive:

The destination URI googledrive: refers to the root folder of your Google Drive. It is possible to specify a path after the googledrive: prefix; so you could sync only the sub folder music as follows:

Of course, all other options provided by Stream Sync, like encryption or filters, are available as well.

This section discusses the initialization of a sync process over already existing data on a concrete example. It assumes that the mirror mode of Stream Sync has already been used to keep a backup of a local folder with music files on a Google Drive account. (For simplicity, we use the example from the previous section about Google Drive.) Now another device comes into play that should have read and write access to the music collection. The challenge here lies in the correct setup of the local state file.

The first step is to make sure that the local folder contains the most recent data and is in sync with the content of the Google Drive folder. The straight-forward way to achieve this is by running again a mirror process that applies all local changes to the Cloud folder:

This assumes that modifications were done only locally, since all changes on the Google Drive Folder are overridden. If this was not the case, you would have to manually ensure that both structures contain the same, up-to-date data.

After the local folder has the correct content, the local state can now be imported using the command below. We use the standard name and location for the local state file:

This should finish rather fast, since only the local file system is processed. The command yields a file with local state information in the .stream-sync subfolder of the user’s home directory.

The second device that should have access to the music collection can be initialized in a similar way. Probably, we want to run a mirror process first, but this time using the Google Drive folder as source and the local folder as destination structure - after the steps performed on the first computer, the Google Drive should contain the most recent data. After this is done, the local state can be imported as described before; execute an equivalent command, maybe the path to the local folder has to be adapted.

In the future, manipulations can be done on the data on both devices. Start a sync process when appropriate using a command like this:

(This is basically the same command as for importing the local state, just without the --import-state flag.) Stream Sync will sync the changes from both devices or issue warnings if it detects conflicting changes.