Hello,

Currently I use this project to send data to my Swift infrastructure. We encounter some performance issue when reading/writing big files (> dozen MB) from/to swift. We did serveral tests, one of those for instance using rclone shows that when uploading a big file (2GB) into swift using rclone swift backend type we reach a speed of 200MB/s (both ways : upload/download). Then, using rclone sftp backend pointing to sftpcloudfs (running locally), we only reach 6MB/s (local to swift) and 3MB/s (Swift to local).
It looks like there's a huge overhead reading remote file by small chunks. Do you think there's a way to improve this, if so, could finger point where to look at in the code ? :-)

I am trying to make sftpcloudfs run on startup to ensure the desired behaviour is running after a reboot.

How to do this on my Plesk install on CentOS 6.4 (Final)?

Also, should it run using a specific user account name or just root?

Hello,

I typed in sftpcloudfs.conf memcache server with port, which is running, but files being uploaded to SFTP are not being cached. I checked it using memcache stats. I'm using sftpcloudfs from GitHub with Ubuntu 14.10 (package in Ubuntu is broken, can't get files from OpenSwift).

Caching in ftp-cloudfs is working with memcache.

Are there any plans to support keystone's v3 authentication? If not would you be open to a patch?

When uploading sub-directories containing more than 1 file with scp, an extra \x00 is inserted in channel, causing error while parsing record in recv_inner().

How to reproduce :

--- a/sftpcloudfs/scp.py
+++ b/sftpcloudfs/scp.py
@@ -143,9 +143,15 @@ class SCPHandler(threading.Thread):
         if '\n' not in self.buffer:
             while True:
                 chunk = self.channel.recv(1024)
+                self.log.info('[recv_line] chunk=%s', chunk)
+                hex_chunk = "".join(x.encode('hex') for x in chunk)
+                self.log.info('[recv_line] hex chunk=%s', hex_chunk)
                 self.buffer += chunk
+                self.log.info('[recv_line] buffer=%s', self.buffer)
+                hex_buffer = "".join(x.encode('hex') for x in self.buffer)
+                self.log.info('[recv_line] hex buffer=%s', hex_buffer)
                 if '\n' in chunk:
                     break

Logs :

Jan  6 14:56:17 node-xxx paramiko[18892]: INFO: [recv_line] chunk=D0755 0 in
Jan  6 14:56:17 node-xxx paramiko[18892]: INFO: [recv_line] hex chunk=4430373535203020696e0a
Jan  6 14:56:17 node-xxx paramiko[18892]: INFO: [recv_line] buffer=D0755 0 in
Jan  6 14:56:17 node-xxx paramiko[18892]: INFO: [recv_line] hex buffer=4430373535203020696e0a
Jan  6 14:56:17 node-xxx paramiko[18892]: INFO: [recv_line] chunk=D0755 0 sub1
Jan  6 14:56:17 node-xxx paramiko[18892]: INFO: [recv_line] hex chunk=4430373535203020737562310a
Jan  6 14:56:17 node-xxx paramiko[18892]: INFO: [recv_line] buffer=D0755 0 sub1
Jan  6 14:56:17 node-xxx paramiko[18892]: INFO: [recv_line] hex buffer=4430373535203020737562310a
Jan  6 14:56:18 node-xxx paramiko[18892]: INFO: [recv_line] chunk=C0644 65536 file2
Jan  6 14:56:18 node-xxx paramiko[18892]: INFO: [recv_line] hex chunk=43303634342036353533362066696c65320a
Jan  6 14:56:18 node-xxx paramiko[18892]: INFO: [recv_line] buffer=C0644 65536 file2
Jan  6 14:56:18 node-xxx paramiko[18892]: INFO: [recv_line] hex buffer=43303634342036353533362066696c65320a
Jan  6 14:56:18 node-xxx paramiko[18892]: INFO: [recv_line] chunk=
Jan  6 14:56:18 node-xxx paramiko[18892]: INFO: [recv_line] hex chunk=00
Jan  6 14:56:18 node-xxx paramiko[18892]: INFO: [recv_line] buffer=
Jan  6 14:56:18 node-xxx paramiko[18892]: INFO: [recv_line] hex buffer=00
Jan  6 14:56:18 node-xxx paramiko[18892]: INFO: [recv_line] chunk=C0644 32768 file1
Jan  6 14:56:18 node-xxx paramiko[18892]: INFO: [recv_line] hex chunk=43303634342033323736382066696c65310a
Jan  6 14:56:18 node-xxx paramiko[18892]: INFO: [recv_line] buffer=
Jan  6 14:56:18 node-xxx paramiko[18892]: INFO: [recv_line] hex buffer=0043303634342033323736382066696c65310a

My (dirty) workaround :

--- a/sftpcloudfs/scp.py
+++ b/sftpcloudfs/scp.py
@@ -143,6 +143,8 @@ class SCPHandler(threading.Thread):
         if '\n' not in self.buffer:
             while True:
                 chunk = self.channel.recv(1024)
+                if chunk == '\x00':
+                    continue
                 self.buffer += chunk
                 if '\n' in chunk:
                     break

Any hints on what's causing this behaviour ?

Update the dependencies to the latest available version with support, test and fix any incompatibilities.

Even though it is suggested that sftpcloudfs is running on a high port, by a non-root user, it will most likely be accessible to clients on *:22 - via a rewrite, or similar, to prevent the user from having to connect to a non-standard port.

On that basis it will get hit by dictionary attacks which should ideally be rate-limited.

My initial though was that we could use the memcached instance to count failing login attempts and drop clients who made too many, or implement some backoff.

However sftpcloudfs doesn't directly use memcaced. Instead it merely passes over the details to the ftp-cloudfs library.

Would you accept a patch which implemented rate-limiting by actually trying to keep track of connection failures in memcached, instantiating a connection to a memcached instanc if the memcached options were supplied?

Or failing that would you have an alternative preferred approach to handling rate-limiting?

Python 2.7 will end of life in 2020 so we should ensure that the project works with Python 3.

Supporting Python 2 is optional.

The docs and the server help shows it as an option processed by OptionParser, but we're pre-processing it before it gets to the RawConfigParser.

https://github.com/Memset/sftpcloudfs/blob/master/sftpcloudfs/main.py#L92

In that way only works with --config FILE (space, two different positional arguments) and not with --config=FILE (one single positional argument).

I am trying to run sftpcoudfs on a production server to send backup files over to CloudFiles.

What are the benefits of using memcache?

How to set it up correctly?

Hi,

I have followed the instructions on your website as it is and installed sftpcloudfs along with memcache.

I am able to login to the sftp server with winscp / filezilla. I am also able to download files from my container on openstack swift.

The only issue I am facing is, I am not able to PUT a file in the container. The error I am getting is as below in syslog.

May 13 17:20:22 Storage-1 paramiko[3]: DEBUG: open((<sftpcloudfs.server.SFTPServerInterface object at 0x23c30d0>, u'/ftp-cloudfs/404error.html', 705, <SFTPAttributes: [ size=30 ]>),{}): enter

May 13 17:20:22 Storage-1 paramiko[3]: DEBUG: SFTPHandle(path=u'/ftp-cloudfs/404error.html', flags=705)

May 13 17:20:23 Storage-1 paramiko[3]: INFO: open(u'/ftp-cloudfs/404error.html', 705, <SFTPAttributes: [ size=30 ]>) from ('124.153.69.242', 23710): HTTPConnectionPool(host='simplidrivedel.netmagicsolutions.comPUT', port=80): Max retries exceeded with url: / (Caused by <class 'socket.gaierror'>: [Errno -2] Name or service not known)

May 13 17:20:23 Storage-1 paramiko[3]: DEBUG: open: returns 4

May 13 17:20:24 Storage-1 paramiko[3]: DEBUG: stat((<sftpcloudfs.server.SFTPServerInterface object at 0x23c30d0>, u'/ftp-cloudfs/404error.html'),{}): enter

May 13 17:20:24 Storage-1 paramiko[3]: INFO: stat(u'/ftp-cloudfs/404error.html',) from ('124.153.69.242', 23710): [Errno 2] No such file or directory 404error.html

May 13 17:20:24 Storage-1 paramiko[3]: DEBUG: stat: returns 2

May 13 17:20:26 Storage-1 object-auditor Begin object audit "forever" mode (ZBF).

ftp-cloudfs uses Rackspace Cloud Files auth url as defautl auth url.

We should probably be doing the same (makes sense, this is a Cloud Files proxy).

Currently it raises a ConfigParser.ParsingError exception (or derived exception).

It could catch the exception an provide feedback.

The same problem was fixed here: https://github.com/cloudfs/ftp-cloudfs/issues/55

When the sftpcloudfs is started with --foreground, the pidfile is still checked for existence, and possibly created.

We're using sftpcloudfs with ubuntu's upstart, which doesn't use pidfiles. It found it quite counterintuitive that a pidfile is checked and created, even though it is not configured or required.

I see two solutions:

• Don't create a pidfile when it is not configured (my preference)
• Don't create a pidfile when --foreground is set.

I'm willing to implement either of these solutions, once a decision has been made.

Perhaps using boto library directly or another s3 specific helper library like:
https://pypi.python.org/pypi/s3/0.1.3

I am installing sftp-cludfiles via pip which seems to install fine according to this log http://pastie.org/private/6yiqvbsq9ipdtxvp6uchha

when trying to use run pidlock i am getting :

chmouel@swift-all-in-one:~$ sftpcloudfs 
Traceback (most recent call last):
  File "/usr/local/bin/sftpcloudfs", line 4, in <module>
    from sftpcloudfs.main import Main
  File "/usr/local/lib/python2.6/dist-packages/sftpcloudfs/main.py", line 35, in <module>
    from daemon.pidlockfile import PIDLockFile
ImportError: No module named pidlockfile

This is because Ubuntu has a old version of python-pidlock by default which doesn't have a daemon.pidlockfile and take precendence in python path :

http://pastie.org/private/r7ca5l0f7xmk2d47ogcaga

I have work-arounded in ftpcloudfs with using my own in utils.PidFile

I am trying to use your code to securely send files to my Rackspace Cloud files via SFTP.

However, after following your instructions from QuickStart.md on how to generate an RSA key, I am unable to connect to the sftp proxy.
This is what I use to test if it works:
curl sftp://127.0.0.1:8022 -u username
This is the error message I get:
curl: (51) SSL peer certificate or SSH remote key was not OK

It only works when I specify --insecure, which is not good.
curl sftp://127.0.0.1:8022 --insecure -u username

Would be able to help me fix this issue?

PS: I already have created a valid SSL certificate which I've paid for vie RapidSSL, could this be of any use?

We have sftpcloudfs installed and running on CentOS 6.4, but are having an issue when configuring for Keystone authentication. The following error is being logged:

Errno 5] Endpoint for 'object-store' not found - have you specified a region?:

The following config file is being used:

# sftpcloudfs configuration file
#

[sftpcloudfs]

# Authentication URL (eg Memstore, RackSpace Cloud Files, OpenStack)
auth-url = http://192.168.20.50:5000/v2.0

# Host RSA key used by the server.
host-key-file = /etc/sftpcloudfs/rsa_key

# Address to bind.
bind-address = 192.168.20.50

# Port to bind.
port = 8022

# SCP support.
# scp-support = yes

# Memcache server(s) for external cache (eg 127.0.0.1:11211)
# Can be a comma-separated list.
memcache = 192.168.20.50:11211

# Maximum number of workers.
# max-children = 20

# Large file support.
# Specify a size in MB to split large files.
# split-large-files = (empty)

# Hide .part directory from large files
# hide-part-dir = no

# Log file location.
# log-file = (empty)

# Enable logging to system logger (daemon facility).
syslog = yes

# Show detailed information on logging.
verbose = yes

# Pid file location when in daemon mode.
# pid-file = (empty)

# UID to drop privileges when in daemon mode.
# uid = (empty)

# GID to drop privileges when in daemon mode.
# gid = (empty)

# Auth 2.0 (Keystone), requires keystoneclient
keystone-auth = yes

# Region name to be used with Auth 2.0 (optional)
keystone-region-name = regionOne

# Tenant separator to be used with Auth 2.0 (eg. TENANT.USERNAME)
keystone-tenant-separator = ':'

# Service type to be used with Auth 2.0.
keystone-service-type = 'object-store'

# Endpoint type to be used with Auth 2.0.
keystone-endpoint-type = 'publicurl'

# EOF

The pidfile is always created and defaults to $TEMPDIR/$PACKAGENAME.pid.

This also affects the command line help.

Related: #20

It can be like:

https://github.com/chmouel/ftp-cloudfs/wiki/QuickStart

This could be in the wiki or just a quickstart.md file distributed with the source code.

The communication between the SFTP daemon and Swift is over HTTP, and it means that when uploading a big file without write permissions we need to upload the whole file before getting an error.

It would be a good idea testing the permissions first (ie. create a zero bytes file), so the SFTP service returns the error ASAP.