mattijs / node-rsync Goto Github PK

Filenames with special characters break the final command output.
This is what I'm using in the meantime to get around it: rsync.source(filename.replace(/(["\s'$])/g,'$1'))`

Hi,
I want to calculate the time I need to rysnc an image of 50-60KB.
Can I know it?
Should I rely on the response from which we get rsync function?

you have an bug in the rsync.js line 734

I believe this is a typo on:

this._outputHandlers.stderr = stdout should be this._outputHandlers.stderr = stderr

Thanks for this nice wrapper, but using it on OS X it seems I can't get a handler on stderr output to be called. The following snippet:

function runRsync(src, dst) {
    // Build the command
    var rsync = new Rsync()
        .shell('ssh')
        .flags('az')
        .set('v')
        .set('timeout', 5) // exit code 30 "io timeout after 5 seconds -- exiting" when reached
        .set('out-format', "ITEM %f") // tag and filename with full path
        .progress()
        .source(src)
        .destination(dst);

    rsync.execute(function done(error, code, cmd) {
            console.log('CMD: ' + cmd);
            if (error) {
                console.log(error.message);
            }
            console.log('Exit Code: ' + code);
        },
        function hStdOut(data) {
            console.log('OUT: ' + data);
            // every file gets its own line acc. to out-format option
            // every progress gets a line with 4 fields:
            // - bytes, percent, throughput, est. time remaining (h:mm:ss)
            // summary starts with a blank line, followed by totals:
            // - "sent xx bytes", "received xx bytes", "xxx bytes/sec",
            //  followed by a line with "total size is <bytes>  speedup is <factor>"
        },
        function hStdErr(data) {
            console.log('ERR: ' + data);
            // never called?
        }
    );
}
runRsync(src, dst);

leads to this console output (personal data replaced by [...]):

  OUT: io timeout after 5 seconds -- exiting
  rsync error: timeout in data send/receive (code 30) at /SourceCache/rsync/rsync-42/rsync/io.c(164)    [sender=2.6.9]
  CMD: rsync -azv --rsh="ssh" --timeout="5" --out-format="ITEM %f" --progress [base dir removed]/output/ [user]@[host]:[basedir]/import/
  rsync exited with code 30
  Exit Code: 30

showing "rsync exited with code 30" without the here intended prefix "ERR: ".
Am I missing something, e. g. the handler being called only when rsync's not exiting, or is this an issue?
Well, in this timeout case, I can pretty much live without an error output handler, because a restart will be needed anyway, which is dictated by the exit code.
Thanks in advance if you find some time to give me clue.
Cheers
[ofi]

Is there a known pattern for reporting the output of --stats from rsync back to one of the handlers?

Just letting you know, currently all --excludes comes before --includes. Looking into it.

Is possible to have some configurations per environment?

Regards, Nicholls

When Rsync finishes running using gulp, the console get's freezed on the "Finished 'x' after 'x'ms ( [13:22:17] Finished 'sync' after 5.38 ms ), without getting back to allow me type in the command line;

This is what I have:

gulp.task('sync', function () {

// Build the command
var rsync = new Rsync()
    .flags('va')
    .source('./')
    .destination('../../xxx');

// Execute the command
rsync.execute(function(error, code, cmd) {
    console.log('Rsync is complete!');
});

});

Can it be used on windows systems that don't support rsync natively?

Rsync = require('rsync');

cmd = new Rsync()
    .flags('avz')
    .source('src/')
    .destination('dst/');

c = cmd.execute(
    function(error,code,cmd) { console.log("Done"); },
    function(data) { console.log(data.toString()); },
    function(data) { console.log(data.toString()); },
);

setTimeout(function() {
    console.log("!!!!!!!Killing!!!!!!");
    c.kill();
},4000);

Should kill the sync after 4 seconds. It does not. rsync does seem to receive the signal.

Tried to set the password，.set('password-file',pwdfile)，but failed

Allow overriding of environment variables being passed to child_process.spawn & rsync.

This will allow us to be more secure in passing around environment variables while also using the --password-file flag.

When using a cygwin path with a space in it as a source, rsync will error out with error code 23.
I tried the following things:

• escape space with backslash
• modify rsync.js and put the source arguments in quotes (i tried single and double quotes)

I will try to resolve this issue but I am currently out of ideas.

Add tests to check the command that is going to be executed. The scope it to test until executing ,not if the command is executed properly by rsync.

This involves collecting commands that are known to execute properly and test the command output against these commands that are known to execute successfully.

For example the command rsync -avz --exclude='.git' dir_a/ dir_b is known to execute properly. A test could be made to see if setting up this command with Rsync results in the same command being produced.

Example: https://github.com/mattijs/node-rsync/blob/master/tests/escape.test.js

This issue has been generated on-behalf of Mik317 (https://huntr.dev/app/users/Mik317)

Vulnerability Description

Affected versions execute arbitrary commands remotely inside the victim's PC. The issue occurs because user input is formatted inside a command that will be executed without any checks. The spawn function receives the _executableShell variable, which is the /bin/sh command. This could result in any command, even if the function is written correctly, leading to RCE.

The issue arises here:

https://github.com/mattijs/node-rsync/blob/master/rsync.js#L506

Bug Bounty

We have opened up a bounty for this issue on our bug bounty platform. Want to solve this vulnerability and get rewarded 💰? Go to https://huntr.dev/

I enabled progress: var rsync = new Rsync().progress()

Added stdout handler in rsync.execute, and all I see in output is:

<Buffer 20 20 20 20 20 20 36 35 38 38 36 36 20 20 35 35 25 20 20 36 34 33 2e 34 32 6b 42 2f 73 20 20 20 20 30 3a 30 30 3a 30 30 0d>

Thanks all..

When have many long arguments in

line 362 | if (long.length > 0) args.push(long.join(' '));

this line create an argument with spaces, for some reason child_process.spawn don't support arguments with space in it so i suggest this change:
line 362 | for (var i in long) args.push(long[i]);

it resolve the problem to me.

Thanks for your great work!

Support for --rsync-path

Is there a way to have an event handeler hooked on the progress ?
I need to log the current progress/ speed of a given job every seconds

Thanks for this wrapper, seems like a better way than to manage a shell script instead. But I'm having difficulties trying to get it set up. I'm running this rsync command:

rsync -avzn ./ [email protected]:/var/www/blog

Which translates to this using node-rsync:

var rsync = new Rsync()
  .flags('avzn')
  .source('./')
  .destination('[email protected]:/var/www/blog');

rsync.execute(function(error, code, cmd) {
    // we're done
    console.log('Done.');
});

The problem is that the command just hangs. I think it's an issue with the spawn command on this line: https://github.com/mattijs/node-rsync/blob/master/rsync.js#L448 - if I try running this command:

/bin/sh rsync -avzn ./ [email protected]:/var/www/blog

I get this:

/usr/bin/rsync: /usr/bin/rsync: cannot execute binary file

And if specifying the -c parameter:

/bin/sh -c rsync -avzn ./ [email protected]:/var/www/blog

I just get the rsync help screen.

I wonder if it would be better to spawn the rsync command directly?

I'm using the latest version of node-rsync on node v0.10.26.

See https://github.com/jerrysu/gulp-rsync for "inspiration".
Basically you want to be able to call rsync from an arbitrary location, but tell it to use another location as its "root".

env:
process.platform: win32
node version: v10.x.x

my code

const workDir = path.resolve('.')


console.log(chalk.red(`This platform is ${process.platform}\n`))

const stdoutHandler = function(data) {
	console.log(chalk.green('stdoutHandler: '))
	console.log(data.toString())
	console.log(chalk.green('================\n'))
}
const stderrHandler = function() {
	console.log(chalk.red('stderrHandler: '))
	console.log(data.toString())
	console.log(chalk.red('================\n'))
}

module.exports = async (txt) => {
	
	const outFile = path.resolve(workDir, `testSync.txt`)
	await fse.outputFile(outFile, txt)
	
	const cmd = new Rsync()
		.flags('avz')
		.shell('ssh')
		.source(outFile)
		.destination('name@server:/home/appadmin/cli');
	
	cmd.execute(function(error, code, cmd) {
		console.log('code: ', code)
		if (error) {
			console.error(error)
			console.log('error: ', cmd)
			process.exit(1)
		}
		console.log('All done executing: ', cmd);
	}, stdoutHandler, stderrHandler);
}

cmd output content:

$ retailo2o-cli test 000
This platform is win32

stdoutHandler:
'rsync' �����ڲ����ⲿ���Ҳ���ǿ����еij���
���������ļ���

================

code:  1
Error: rsync exited with code 1
    at Rsync.<anonymous> (D:\nvm\v10.15.1\node_global\node_modules\retailo2o-cli\node_modules\rsync\rsync.js:525:21)
    at ChildProcess.emit (events.js:189:13)
    at maybeClose (internal/child_process.js:970:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:259:5)
error:  rsync -avz --rsh=ssh /workspace/coupon/testSync.txt name@server:/home/appadmin/cli

It is now only possible to add include and exclude statements. This is a very inflexible implementation because include statements will always come before exclude statements, order is not perserved.

Rsync offers a lot of filtering rules and is very flexible in building them: as single statement, from files and applying them to directories.

The module should provide more ways to specify filtering rules where all rules convert to filters and order is preserved.

Hello,

I'm trying to run this simple code in Windows:

var Rsync = require('rsync');

// Build the command
var rsync = new Rsync()
  .dry()
  .flags('az')
  .source("a.txt")
  .destination('b.txt');

// Execute the command
rsync.execute(function(error, code, cmd) {
    console.log(error);
    console.log(code);
    console.log(cmd);
});

This is what I get:

[Error: rsync exited with code 1]
1
rsync -naz a.txt b.txt

What is wrong?

Thank you

# Running rsync command to send file from one server to another using npm rsync module.

With following options set :

{ _executable: 'rsync',
_executableShell: '/bin/sh',
_sources: [ ----"source server"---- ],
_destination: '--------destination server-----',
_patterns: [],
_options: { rsh: 'ssh', a: null, v: null, z: null, h: null },
_outputHandlers: { stdout: null, stderr: null },
_debug: false }

It hangs for unexpected time & without any error.

In my case my directory size is more than 200 MB.

Hi there,

I'm trying to use a custom port for my rsync script.
Here's what I use :

const rsyncArgs = new Rsync()
            .flags('avz')
            .set('e', `ssh -p 5000`)
            .set('progress')
            .set('no-motd')
            .source(`${directory}/`)
            .destination(`${stage.server.user}@${stage.server.host}:${stage.server.directory}`)

const rsync = spawn('rsync', rsyncArgs.args());

And here the command generated by calling rsyncArgs.command():

rsync -avz -e "ssh -p 5000" --progress --no-motd /private/var/www/dir user@host:public_html

But my script exits with:

rsync: Failed to exec ssh -p 5000: No such file or directory (2)
rsync error: error in IPC code (code 14) at pipe.c(85) [sender=3.1.3]
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [sender=3.1.3]

When I run the command in the terminal, everything works smoothly. So the request is generating fine, but isn't running in with the spawn methods...

if I set rsync.shell() to "ssh -i SOMEKEY.pem" when I call rsync.command() I get:

rsync --rsh="ssh\ -i\ SOMEKEY.pem".

This should be either "ssh -i SOMEKEY.pem" or ssh\ -i\ SOMEKEY.pem

I noticed that $ is not escaped as it probably should in Linux/OS X. For instance, this:

Rsync = require('rsync')
var rsync = new Rsync()
  .flags('az')
  .source('/source')
  .destination('server:/destination$test')
  .command()

will yield

"rsync -az /source server:/destination$test"

So when the rsync command is run, $test will be interpreted as an environment variable.

such as 777 ;

Please publish to npm version with cwd support.

Could you please tell me how I can set RSYNC_PASSWORD environment variable in the child process?

just like rsync -e 'ssh -p 1234'?

Hey there,

i run into some trubble running rsync with exec():

• stdout buffer-overflow with --info=progress2
• rsync child process gets not killed on SIGTERM cause of running in a subshell
  so i reworked it to use spawn()
  Buffering the whole stdout/stderr stream is not a good option

moreover ive added events and overallProgress parsing

perhaps you wanna pick my commits: https://github.com/psi-4ward/node-rsync

I would like to rsync some directories from a server to local with --delete-after and --delete-excluded options.
To use --delete-excluded option, I need two filter rules.
According to the man page, "You may use as many --filter options on the command line as you like to build up the list of files to exclude."
However, I've failed to add two filter rules with set() function. Only the last filter option works.

I cannot use merge file due to the limitation of my filesystem. I can use only command-line filter rule.

Could you add the feature that multiple --filter options work?

How would one specify the ssh key using this tool?

On line 408, the first line of the function Rsync.prototype.output makes a call to Array.isaArray instead of Array.isArray.

Would you be willing to entertain support for custom streams? This will give users more control over handling output. For example, I'd like to pipe the --progress output to another stream.

Probably custom stdout and stderr options would work fine

I will probably go ahead and implement this in a fork, but just wondering if you would consider accepting a PR.

I'm running into this issue and can't seem to find a solution. When setting a source or destination that has spaces in it, it causes all kinds of issues and for a variety of reasons:

1. When entering a name with spaces, the spaces are escaped but the entire string isn't put into quotes. Simply escaping the name doesn't work. There's an SO question on this
2. When entering a name with spaces and trying to manually quote it, the quotes get escaped (which makes sense but doesn't offer a quick workaround)

Hi,

I'm testing this extension and after executing a command, I'm getting this error:
[Error: rsync exited with code 14]

The command executed looks like this:
rsync -av --rsh="ssh -i /home/tomasz/.ssh/rs/rsync" /tmp/logs/KMJzx9fzb/*.gz [email protected]:~/logs

If I copy paste it and run it in shell, it works fine.
rsync version 3.1.0 protocol version 31

I'm using a key with no passphrase to connect there.

Any ideas ?

Thanks in advance

The current tests are lacking a proper way to describe and test the full command that will be outputted to the shell. There should be a proper test harness that is easy to expand to test output cases.

If you add a filter with, for instance, .exclude('.git'), it's ignored although if you copy and paste the command as reported by node-rsync into the shell it works perfectly.

I have a local host A connected to server B and server B is connected to server C. Now I wanted to rsync files from local host A to C, will this library support it? This is not an issue rather wanted to know how this can be achieved using your library.