gulp exits with 'Error in plugin' when gulp-sftp used with gulp-concat and gulp.watch about gulp-sftp HOT 21 CLOSED

The easiest solution is to switch to streams. The closure is an issue with ssh2 - please check out the known issues section of the readme.

from gulp-sftp.

I have already read your readme and tried using buffer: false but that just caused more errors.

I don't quite understand what the problem is and what I should be changing my task to do? I know roughly it has something to do with buffers and streams.

Should I try using one of the plugins Lars Kappert suggests such as gulp-streamify?

from gulp-sftp.

Streamify should take care of it. I would encourage you to bring the issue up with the author of ssh2 and cite the sudden disconnect when having large buffers.

from gulp-sftp.

https://github.com/gtg092x/gulp-sftp/blob/master/package.json#L36

@gtg092x: @mscdex has suggested updating this project to the latest version of ssh2 (v0.3.3) but that you may face some issues with backwards compatibility. His comment is here: mscdex/ssh2#153 (comment)

Do you think you could make this update? Let me know if you want me to contribute a pull request and look into this

from gulp-sftp.

Sure thing - I'll make the change tomorrow evening - thank you!

from gulp-sftp.

@mscdex @u01jmg3 - I've updated my ssh2 and my tests are still getting abrupt connection closings. I've got a branch ready with the changes in place, but it looks like that issue still exists and can be recreated using large buffers via large transfers. I haven't dug into it, but my guess is that it's a memory management error via pagenet.

I could be wrong though. If this issue is on my end, I'll gladly accept pull requests that can upload large files without relying on streamBuffers. I just can't find anywhere that I'm forcing a sftp disconnect.

In the mean time, refer to gulp-streamify if we can't solve your issue.

from gulp-sftp.

Refer to https://github.com/gtg092x/gulp-sftp/tree/sshbump for expected changes

from gulp-sftp.

Do you have an actual minimal reproducible test somewhere?

from gulp-sftp.

Yes - though you might have different memory limitations because you're not on the same machine.
I always get an error uploading the attached file. If you need more details - auth information, etc. Let me know.

from gulp-sftp.

I think I see what the problem is. OpenSSH limits the size of its SFTP packets to 256kb. So sending the whole contents at one time (~3mb) surpasses this and OpenSSH rejects the data, despite the SFTP spec allowing for a full, unsigned 32-bit packet length. I'm assuming that means OpenSSH always buffers entire SFTP packets, no matter what the actual request is (which is really silly).

I'm not sure what the other implementations do, but I guess I could detect OpenSSH and enforce the same limit in that case.

from gulp-sftp.

@mscdex - I appreciate it; I'm dodging some work by acting like it's not my problem, but your module does far more work than mine and supports way more use cases. Thanks for stepping to make that change. Let me know when that's in and I'll update my branch with the latest version.

from gulp-sftp.

@gtg092x @u01jmg3 This should be fixed now as of mscdex/ssh2@2603c4dc8. Please give the master branch a try.

from gulp-sftp.

@gtg092x: will you be updating gulp-sftp to use the latest version of ssh2?

from gulp-sftp.

@u01jmg3 I haven't published a new version that includes the previously mentioned change yet, I will publish a new version once others have also tested it to make sure it works for them as well.

from gulp-sftp.

Okay, no problem

from gulp-sftp.

You should be able to test it by changing to the gulp-sftp module directory and then doing npm install https://github.com/mscdex/ssh2/tarball/master. Then re-run your script that you were previously having issues with.

from gulp-sftp.

I re-worked my script to get around this issue so will have to revert back for testing - will let you know how it goes. Thanks for the tip

from gulp-sftp.

@mscdex The version on your master branch works without disconnecting me. Thank you for your help!
Let me know when you've bumped your version with npm and I'll do the same on this end.

from gulp-sftp.

All good for me too - thanks again to you both @mscdex @gtg092x

I'll await the version bump

from gulp-sftp.

I've now published ssh2 v0.3.4. Thanks all.

from gulp-sftp.

Updated on this end and on npm - closing

from gulp-sftp.