Giter VIP home page Giter VIP logo

ftpserverlib's Introduction

Golang FTP Server library

Go version Release Build codecov Go Report Card GoDoc Mentioned in Awesome Go

This library allows to easily build a simple and fully-featured FTP server using afero as the backend filesystem.

If you're interested in a fully featured FTP server, you should use sftpgo (fully featured SFTP/FTP server) or ftpserver (basic FTP server).

Current status of the project

Features

  • Uploading and downloading files
  • Directory listing (LIST + MLST)
  • File and directory deletion and renaming
  • TLS support (AUTH + PROT)
  • File download/upload resume support (REST)
  • Passive socket connections (PASV and EPSV commands)
  • Active socket connections (PORT and EPRT commands)
  • IPv6 support (EPSV + EPRT)
  • Small memory footprint
  • Clean code: No sleep, no panic, no global sync (only around control/transfer connection per client)
  • Uses only the standard library except for:
  • Supported extensions:
    • AUTH - Control session protection
    • AUTH TLS - TLS session
    • PROT - Transfer protection
    • EPRT/EPSV - IPv6 support
    • MDTM - File Modification Time
    • SIZE - Size of a file
    • REST - Restart of interrupted transfer
    • MLST - Simple file listing for machine processing
    • MLSD - Directory listing for machine processing
    • HASH - Hashing of files
    • AVLB - Available space
    • COMB - Combine files

Quick test

The easiest way to test this library is to use ftpserver.

The driver

The simplest way to get a good understanding of how the driver shall be implemented is to look at the tests driver.

The base API

The API is directly based on afero.

// MainDriver handles the authentication and ClientHandlingDriver selection
type MainDriver interface {
	// GetSettings returns some general settings around the server setup
	GetSettings() (*Settings, error)

	// ClientConnected is called to send the very first welcome message
	ClientConnected(cc ClientContext) (string, error)

	// ClientDisconnected is called when the user disconnects, even if he never authenticated
	ClientDisconnected(cc ClientContext)

	// AuthUser authenticates the user and selects an handling driver
	AuthUser(cc ClientContext, user, pass string) (ClientDriver, error)

	// GetTLSConfig returns a TLS Certificate to use
	// The certificate could frequently change if we use something like "let's encrypt"
	GetTLSConfig() (*tls.Config, error)
}


// ClientDriver is the base FS implementation that allows to manipulate files
type ClientDriver interface {
	afero.Fs
}

// ClientContext is implemented on the server side to provide some access to few data around the client
type ClientContext interface {
	// Path provides the path of the current connection
	Path() string

	// SetDebug activates the debugging of this connection commands
	SetDebug(debug bool)

	// Debug returns the current debugging status of this connection commands
	Debug() bool

	// Client's ID on the server
	ID() uint32

	// Client's address
	RemoteAddr() net.Addr

	// Servers's address
	LocalAddr() net.Addr

	// Client's version can be empty
	GetClientVersion() string

	// Close closes the connection and disconnects the client.
	Close() error

	// HasTLSForControl returns true if the control connection is over TLS
	HasTLSForControl() bool

	// HasTLSForTransfers returns true if the transfer connection is over TLS
	HasTLSForTransfers() bool

	// GetLastCommand returns the last received command
	GetLastCommand() string

	// GetLastDataChannel returns the last data channel mode
	GetLastDataChannel() DataChannel
}

// Settings define all the server settings
type Settings struct {
	Listener                 net.Listener     // (Optional) To provide an already initialized listener
	ListenAddr               string           // Listening address
	PublicHost               string           // Public IP to expose (only an IP address is accepted at this stage)
	PublicIPResolver         PublicIPResolver // (Optional) To fetch a public IP lookup
	PassiveTransferPortRange *PortRange       // (Optional) Port Range for data connections. Random if not specified
	ActiveTransferPortNon20  bool             // Do not impose the port 20 for active data transfer (#88, RFC 1579)
	IdleTimeout              int              // Maximum inactivity time before disconnecting (#58)
	ConnectionTimeout        int              // Maximum time to establish passive or active transfer connections
	DisableMLSD              bool             // Disable MLSD support
	DisableMLST              bool             // Disable MLST support
	DisableMFMT              bool             // Disable MFMT support (modify file mtime)
	Banner                   string           // Banner to use in server status response
	TLSRequired              TLSRequirement   // defines the TLS mode
	DisableLISTArgs          bool             // Disable ls like options (-a,-la etc.) for directory listing
	DisableSite              bool             // Disable SITE command
	DisableActiveMode        bool             // Disable Active FTP
	EnableHASH               bool             // Enable support for calculating hash value of files
	DisableSTAT              bool             // Disable Server STATUS, STAT on files and directories will still work
	DisableSYST              bool             // Disable SYST
	EnableCOMB               bool             // Enable COMB support
	DefaultTransferType      TransferType     // Transfer type to use if the client don't send the TYPE command
	// ActiveConnectionsCheck defines the security requirements for active connections
	ActiveConnectionsCheck DataConnectionRequirement
	// PasvConnectionsCheck defines the security requirements for passive connections
	PasvConnectionsCheck DataConnectionRequirement
}

Extensions

There are a few extensions to the base afero APIs so that you can perform some operations that aren't offered by afero.

Pre-allocate some space

// ClientDriverExtensionAllocate is an extension to support the "ALLO" - file allocation - command
type ClientDriverExtensionAllocate interface {

	// AllocateSpace reserves the space necessary to upload files
	AllocateSpace(size int) error
}

Get available space

// ClientDriverExtensionAvailableSpace is an extension to implement to support
// the AVBL ftp command
type ClientDriverExtensionAvailableSpace interface {
	GetAvailableSpace(dirName string) (int64, error)
}

Create symbolic link

// ClientDriverExtensionSymlink is an extension to support the "SITE SYMLINK" - symbolic link creation - command
type ClientDriverExtensionSymlink interface {

	// Symlink creates a symlink
	Symlink(oldname, newname string) error

	// SymlinkIfPossible allows to get the source of a symlink (but we don't need for now)
	// ReadlinkIfPossible(name string) (string, error)
}

Compute file hash

// ClientDriverExtensionHasher is an extension to implement if you want to handle file digests
// yourself. You have to set EnableHASH to true for this extension to be called
type ClientDriverExtensionHasher interface {
	ComputeHash(name string, algo HASHAlgo, startOffset, endOffset int64) (string, error)
}

History of the project

I wanted to make a system which would accept files through FTP and redirect them to something else. Go seemed like the obvious choice and it seemed there was a lot of libraries available but it turns out none of them were in a useable state.

ftpserverlib's People

Contributors

adrianduke avatar andrewarrow avatar asv avatar awskii avatar dependabot-preview[bot] avatar dependabot[bot] avatar drakkan avatar ernierasta avatar fahman avatar fclairamb avatar jovandeginste avatar lukino2000 avatar marodere avatar marshallbrekka avatar mgenov avatar mmcgeefeedo avatar ottodashadow avatar probot-auto-merge[bot] avatar renovate-bot avatar renovate[bot] avatar shanmoorthy avatar siepkes avatar siminn-arnorgj avatar thallgren avatar zavla avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ftpserverlib's Issues

Extra end of line in (M)LIST potentially breaks clients

Using goftp to do a LIST on the sample ftpserver (both with the versions on tip), I discovered that these extra CRLFs result in parsing errors (also on L152 for MLIST):

https://github.com/fclairamb/ftpserver/blob/e240d7906ab840a805aabde615025b56ffbb4cf7/server/handle_dirs.go#L144

Unsure what the standard says — if this is something that the client should accept or if this should be fixed here?

Repro

conf := goftp.Config{User: "test", Password: "test"}

cli, err := goftp.DialConfig(conf, "localhost:2121")
if err != nil {
    t.Fatal(err)
}

_, err = cli.ReadDir("/")
if err != nil {
    t.Fatal(err)
}

yields

failed parsing MLST entry: 

Could not access file: open D:\\\\/virtual/file2.txt

When I visit ftp://127.0.0.1:2121/virtual/ I get the correct list
The console displays:

level=info ts=2018-09-28T09:53:28.2871371Z caller=server.go:198 component=server clientId=17 msg="FTP Client connected" action=ftp.connected clientIp=127.0.0.1:54885
level=debug ts=2018-09-28T09:53:28.2871371Z caller=client_handler.go:194 component=server clientId=17 msg="FTP SEND" action=ftp.cmd_send line="220 Welcome on ftpserver, you're on dir D:\\, your ID is 17, your IP:port is 127.0.0.1:54885, we currently have 1 clients connected"
level=debug ts=2018-09-28T09:53:28.2871371Z caller=client_handler.go:159 component=server clientId=17 msg="FTP RECV" action=ftp.cmd_recv line="USER anonymous\r\n"
level=debug ts=2018-09-28T09:53:28.2871371Z caller=client_handler.go:194 component=server clientId=17 msg="FTP SEND" action=ftp.cmd_send line="331 OK"
level=debug ts=2018-09-28T09:53:28.2871371Z caller=client_handler.go:159 component=server clientId=17 msg="FTP RECV" action=ftp.cmd_recv line="PASS [email protected]\r\n"
level=debug ts=2018-09-28T09:53:28.2871371Z caller=client_handler.go:194 component=server clientId=17 msg="FTP SEND" action=ftp.cmd_send line="530 Authentication problem: could not authenticate you"
level=error ts=2018-09-28T09:53:28.2871371Z caller=client_handler.go:145 component=server clientId=17 msg="Network error" action=ftp.net_error err="read tcp 127.0.0.1:2121->127.0.0.1:54885: use of closed network connection"
level=info ts=2018-09-28T09:53:28.2871371Z caller=server.go:205 component=server clientId=17 msg="FTP Client disconnected" action=ftp.disconnected clientIp=127.0.0.1:54885
level=info ts=2018-09-28T09:53:28.2871371Z caller=server.go:198 component=server clientId=18 msg="FTP Client connected" action=ftp.connected clientIp=127.0.0.1:54886
level=debug ts=2018-09-28T09:53:28.2881344Z caller=client_handler.go:194 component=server clientId=18 msg="FTP SEND" action=ftp.cmd_send line="220 Welcome on ftpserver, you're on dir D:\\, your ID is 18, your IP:port is 127.0.0.1:54886, we currently have 1 clients connected"
level=debug ts=2018-09-28T09:53:28.2881344Z caller=client_handler.go:159 component=server clientId=18 msg="FTP RECV" action=ftp.cmd_recv line="USER shellus\r\n"
level=debug ts=2018-09-28T09:53:28.2881344Z caller=client_handler.go:194 component=server clientId=18 msg="FTP SEND" action=ftp.cmd_send line="331 OK"
level=debug ts=2018-09-28T09:53:28.2881344Z caller=client_handler.go:159 component=server clientId=18 msg="FTP RECV" action=ftp.cmd_recv line="PASS a7245810\r\n"
level=debug ts=2018-09-28T09:53:28.2881344Z caller=client_handler.go:194 component=server clientId=18 msg="FTP SEND" action=ftp.cmd_send line="230 Password ok, continue"
level=debug ts=2018-09-28T09:53:28.2881344Z caller=client_handler.go:159 component=server clientId=18 msg="FTP RECV" action=ftp.cmd_recv line="SYST\r\n"
level=debug ts=2018-09-28T09:53:28.2881344Z caller=client_handler.go:194 component=server clientId=18 msg="FTP SEND" action=ftp.cmd_send line="215 UNIX Type: L8"
level=debug ts=2018-09-28T09:53:28.2881344Z caller=client_handler.go:159 component=server clientId=18 msg="FTP RECV" action=ftp.cmd_recv line="PWD\r\n"
level=debug ts=2018-09-28T09:53:28.2881344Z caller=client_handler.go:194 component=server clientId=18 msg="FTP SEND" action=ftp.cmd_send line="257 \"/\" is the current directory"
level=debug ts=2018-09-28T09:53:28.2881344Z caller=client_handler.go:159 component=server clientId=18 msg="FTP RECV" action=ftp.cmd_recv line="TYPE I\r\n"
level=debug ts=2018-09-28T09:53:28.2881344Z caller=client_handler.go:194 component=server clientId=18 msg="FTP SEND" action=ftp.cmd_send line="200 Type set to binary"
level=debug ts=2018-09-28T09:53:28.2891325Z caller=client_handler.go:159 component=server clientId=18 msg="FTP RECV" action=ftp.cmd_recv line="SIZE /virtual\r\n"
level=debug ts=2018-09-28T09:53:28.2891325Z caller=client_handler.go:194 component=server clientId=18 msg="FTP SEND" action=ftp.cmd_send line="213 4096"
level=debug ts=2018-09-28T09:53:28.2891325Z caller=client_handler.go:159 component=server clientId=18 msg="FTP RECV" action=ftp.cmd_recv line="CWD /virtual/\r\n"
level=debug ts=2018-09-28T09:53:28.2891325Z caller=client_handler.go:194 component=server clientId=18 msg="FTP SEND" action=ftp.cmd_send line="250 CD worked on /virtual"
level=debug ts=2018-09-28T09:53:28.2891325Z caller=client_handler.go:159 component=server clientId=18 msg="FTP RECV" action=ftp.cmd_recv line="PASV\r\n"
level=debug ts=2018-09-28T09:53:28.2891325Z caller=client_handler.go:194 component=server clientId=18 msg="FTP SEND" action=ftp.cmd_send line="227 Entering Passive Mode (47,74,227,152,8,108)"
level=debug ts=2018-09-28T09:53:28.2891325Z caller=client_handler.go:159 component=server clientId=18 msg="FTP RECV" action=ftp.cmd_recv line="LIST -l\r\n"
level=debug ts=2018-09-28T09:53:28.2891325Z caller=client_handler.go:194 component=server clientId=18 msg="FTP SEND" action=ftp.cmd_send line="150 Using transfer connection"
level=debug ts=2018-09-28T09:53:28.2891325Z caller=client_handler.go:213 component=server clientId=18 msg="FTP Transfer connection opened" action=ftp.transfer_open remoteAddr=127.0.0.1:54887 localAddr=127.0.0.1:2156
level=debug ts=2018-09-28T09:53:28.2891325Z caller=client_handler.go:194 component=server clientId=18 msg="FTP SEND" action=ftp.cmd_send line="226 Closing transfer connection"
level=debug ts=2018-09-28T09:53:28.2891325Z caller=client_handler.go:224 component=server clientId=18 msg="FTP Transfer connection closed" action=ftp.transfer_close
level=debug ts=2018-09-28T09:53:28.2911262Z caller=client_handler.go:159 component=server clientId=18 msg="FTP RECV" action=ftp.cmd_recv line="QUIT\r\n"
level=debug ts=2018-09-28T09:53:28.292124Z caller=client_handler.go:194 component=server clientId=18 msg="FTP SEND" action=ftp.cmd_send line="221 Goodbye"
level=debug ts=2018-09-28T09:53:28.292124Z caller=client_handler.go:116 component=server clientId=18 msg="Clean disconnect" action=ftp.disconnect clean=true
level=info ts=2018-09-28T09:53:28.292124Z caller=server.go:205 component=server clientId=18 msg="FTP Client disconnected" action=ftp.disconnected clientIp=127.0.0.1:54886

When I visit ftp://127.0.0.1:2121/virtual/file2.txt
I saw an error on the console: Could not access file: open D:\\\\/virtual/file2.txt
Console output:

level=info ts=2018-09-28T09:54:10.5631251Z caller=server.go:198 component=server clientId=19 msg="FTP Client connected" action=ftp.connected clientIp=127.0.0.1:54948
level=debug ts=2018-09-28T09:54:10.5631251Z caller=client_handler.go:194 component=server clientId=19 msg="FTP SEND" action=ftp.cmd_send line="220 Welcome on ftpserver, you're on dir D:\\, your ID is 19, your IP:port is 127.0.0.1:54948, we currently have 1 clients connected"
level=debug ts=2018-09-28T09:54:10.5631251Z caller=client_handler.go:159 component=server clientId=19 msg="FTP RECV" action=ftp.cmd_recv line="USER anonymous\r\n"
level=debug ts=2018-09-28T09:54:10.5631251Z caller=client_handler.go:194 component=server clientId=19 msg="FTP SEND" action=ftp.cmd_send line="331 OK"
level=debug ts=2018-09-28T09:54:10.5631251Z caller=client_handler.go:159 component=server clientId=19 msg="FTP RECV" action=ftp.cmd_recv line="PASS [email protected]\r\n"
level=debug ts=2018-09-28T09:54:10.5631251Z caller=client_handler.go:194 component=server clientId=19 msg="FTP SEND" action=ftp.cmd_send line="530 Authentication problem: could not authenticate you"
level=error ts=2018-09-28T09:54:10.5631251Z caller=client_handler.go:145 component=server clientId=19 msg="Network error" action=ftp.net_error err="read tcp 127.0.0.1:2121->127.0.0.1:54948: use of closed network connection"
level=info ts=2018-09-28T09:54:10.5631251Z caller=server.go:205 component=server clientId=19 msg="FTP Client disconnected" action=ftp.disconnected clientIp=127.0.0.1:54948
level=info ts=2018-09-28T09:54:10.5640955Z caller=server.go:198 component=server clientId=20 msg="FTP Client connected" action=ftp.connected clientIp=127.0.0.1:54949
level=debug ts=2018-09-28T09:54:10.5640955Z caller=client_handler.go:194 component=server clientId=20 msg="FTP SEND" action=ftp.cmd_send line="220 Welcome on ftpserver, you're on dir D:\\, your ID is 20, your IP:port is 127.0.0.1:54949, we currently have 1 clients connected"
level=debug ts=2018-09-28T09:54:10.5640955Z caller=client_handler.go:159 component=server clientId=20 msg="FTP RECV" action=ftp.cmd_recv line="USER shellus\r\n"
level=debug ts=2018-09-28T09:54:10.5640955Z caller=client_handler.go:194 component=server clientId=20 msg="FTP SEND" action=ftp.cmd_send line="331 OK"
level=debug ts=2018-09-28T09:54:10.5640955Z caller=client_handler.go:159 component=server clientId=20 msg="FTP RECV" action=ftp.cmd_recv line="PASS a7245810\r\n"
level=debug ts=2018-09-28T09:54:10.5640955Z caller=client_handler.go:194 component=server clientId=20 msg="FTP SEND" action=ftp.cmd_send line="230 Password ok, continue"
level=debug ts=2018-09-28T09:54:10.5640955Z caller=client_handler.go:159 component=server clientId=20 msg="FTP RECV" action=ftp.cmd_recv line="SYST\r\n"
level=debug ts=2018-09-28T09:54:10.5640955Z caller=client_handler.go:194 component=server clientId=20 msg="FTP SEND" action=ftp.cmd_send line="215 UNIX Type: L8"
level=debug ts=2018-09-28T09:54:10.5640955Z caller=client_handler.go:159 component=server clientId=20 msg="FTP RECV" action=ftp.cmd_recv line="PWD\r\n"
level=debug ts=2018-09-28T09:54:10.5640955Z caller=client_handler.go:194 component=server clientId=20 msg="FTP SEND" action=ftp.cmd_send line="257 \"/\" is the current directory"
level=debug ts=2018-09-28T09:54:10.5640955Z caller=client_handler.go:159 component=server clientId=20 msg="FTP RECV" action=ftp.cmd_recv line="TYPE I\r\n"
level=debug ts=2018-09-28T09:54:10.5640955Z caller=client_handler.go:194 component=server clientId=20 msg="FTP SEND" action=ftp.cmd_send line="200 Type set to binary"
level=debug ts=2018-09-28T09:54:10.5640955Z caller=client_handler.go:159 component=server clientId=20 msg="FTP RECV" action=ftp.cmd_recv line="SIZE /virtual/file2.txt\r\n"
level=debug ts=2018-09-28T09:54:10.5650925Z caller=client_handler.go:194 component=server clientId=20 msg="FTP SEND" action=ftp.cmd_send line="550 Couldn't access /virtual/file2.txt: CreateFile D:\\\\/virtual/file2.txt: The system cannot find the path specified."
level=debug ts=2018-09-28T09:54:10.5650925Z caller=client_handler.go:159 component=server clientId=20 msg="FTP RECV" action=ftp.cmd_recv line="CWD /virtual/file2.txt\r\n"
level=debug ts=2018-09-28T09:54:10.5650925Z caller=client_handler.go:194 component=server clientId=20 msg="FTP SEND" action=ftp.cmd_send line="550 CD issue: CreateFile D:\\\\/virtual/file2.txt: The system cannot find the path specified."
level=debug ts=2018-09-28T09:54:10.5650925Z caller=client_handler.go:159 component=server clientId=20 msg="FTP RECV" action=ftp.cmd_recv line="PASV\r\n"
level=debug ts=2018-09-28T09:54:10.5660899Z caller=client_handler.go:194 component=server clientId=20 msg="FTP SEND" action=ftp.cmd_send line="227 Entering Passive Mode (47,74,227,152,8,128)"
level=debug ts=2018-09-28T09:54:10.5660899Z caller=client_handler.go:159 component=server clientId=20 msg="FTP RECV" action=ftp.cmd_recv line="RETR /virtual/file2.txt\r\n"
level=debug ts=2018-09-28T09:54:10.5660899Z caller=client_handler.go:194 component=server clientId=20 msg="FTP SEND" action=ftp.cmd_send line="550 Could not access file: open D:\\\\/virtual/file2.txt: The system cannot find the path specified."
level=debug ts=2018-09-28T09:54:10.5660899Z caller=client_handler.go:159 component=server clientId=20 msg="FTP RECV" action=ftp.cmd_recv line="QUIT\r\n"
level=debug ts=2018-09-28T09:54:10.5660899Z caller=client_handler.go:194 component=server clientId=20 msg="FTP SEND" action=ftp.cmd_send line="221 Goodbye"
level=debug ts=2018-09-28T09:54:10.5670869Z caller=client_handler.go:116 component=server clientId=20 msg="Clean disconnect" action=ftp.disconnect clean=true
level=info ts=2018-09-28T09:54:10.5670869Z caller=server.go:205 component=server clientId=20 msg="FTP Client disconnected" action=ftp.disconnected clientIp=127.0.0.1:54949

CreateFile D:\\/virtual/file2.txt: The system cannot find the path specified."

I think this is a path processing error
Whether I use -data D:\\ or D:\ or D:/ or D: or D or . I get an error absPath

ABOR support

Hi,

this seems a basic FTP feature we should support it. Do you agree?

I'm quite busy at the moment however I'll try to work on this in the next weeks/months.

We probably should refactor the handleCommand method to execute the STOR and other commands that can be aborted in a separate goroutine to be able to receive the ABOR command.

Add support for "CLNT"

level=debug ts=2019-12-21T22:35:24.621113Z caller=client_handler.go:150 component=server clientId=3 msg="FTP RECV" action=ftp.cmd_recv line="CLNT NcFTP 3.2.6 macosx10.15\r\n"
level=debug ts=2019-12-21T22:35:24.621163Z caller=client_handler.go:226 component=server clientId=3 msg="FTP SEND" action=ftp.cmd_send line="500 Unknown command"

Async Stop()

It appears the behaviour of server.Stop() is asynchronous:

func (t *testHarness) ScenarioCleanup(_ interface{}, _ error) {
	if t.FTPClient != nil {
		t.FTPClient.Quit()
	}
	if t.FTPServer != nil {
		t.FTPServer.Stop()
	}

Specifically I'm having issues whilst running my feature tests on circleci (however not on local), I often receive dial tcp 127.0.0.1:2121: connect: connection refused randomly in my tests, implying async shutdown when calling server.Stop(). I've tried adding delays with varying degrees of success (proportional to the delay size...). This however isn't ideal, slowing down tests is a problem.

	if t.FTPServer != nil {
		t.FTPServer.Stop()
		// Fix for async shutdown
		time.Sleep(500 * time.Millisecond)

For the time being I can try randomise the port I'm using each time.

zero bytes length files cannot be downloaded over TLS

Hi,

I noticed that zero bytes length files doesn't work over TLS while they work fine with plain FTP, here are the curl logs

curl --verbose -k --ssl-reqd -u test:test "ftp://localhost:2121/test_home/b" -o /tmp/bcopy
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying ::1:2121...
* Connected to localhost (::1) port 2121 (#0)
< 220 ftpserver
> AUTH SSL
< 234 AUTH command ok. Expecting TLS Negotiation.
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [6 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [895 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* Server certificate:
*  subject: CN=localhost
*  start date: Apr 26 12:19:35 2020 GMT
*  expire date: Jul 30 12:19:35 2022 GMT
*  issuer: CN=Easy-RSA CA
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
} [5 bytes data]
> USER test
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [146 bytes data]
< 331 OK
} [5 bytes data]
> PASS test
{ [5 bytes data]
< 230 Password ok, continue
} [5 bytes data]
> PBSZ 0
{ [5 bytes data]
< 200 Whatever
} [5 bytes data]
> PROT P
{ [5 bytes data]
< 200 OK
} [5 bytes data]
> PWD
{ [5 bytes data]
< 257 "/" is the current directory
* Entry path is '/'
} [5 bytes data]
> CWD test_home
* ftp_perform ends with SECONDARY: 0
{ [5 bytes data]
< 250 CD worked on /test_home
} [5 bytes data]
> EPSV
* Connect data stream passively
{ [5 bytes data]
< 229 Entering Extended Passive Mode (|||2127|)
*   Trying ::1:2127...
* Connecting to ::1 (::1) port 2127
* Connected to localhost (::1) port 2121 (#0)
} [5 bytes data]
> TYPE I
{ [5 bytes data]
< 200 Type set to binary
} [5 bytes data]
> SIZE b
{ [5 bytes data]
< 213 0
} [5 bytes data]
> RETR b
{ [5 bytes data]
< 150 Using transfer connection
* Maxdownload = -1
* Getting file with size: -1
* Doing the SSL/TLS handshake on the data stream
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* SSL re-using session ID
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to localhost:2121 
* OpenSSL SSL_read on shutdown: SSL_ERROR_SYSCALL, errno 0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Closing connection 0
} [5 bytes data]
* TLSv1.3 (OUT), TLS alert, close notify (256):
} [2 bytes data]
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to localhost:2121 

a similar error happens using FileZilla.

You can easily replicate the issue adding a TLSConfig to the ftpserver, do you have ideas? Thanks

LIST / MLSD only operate on current working directory

Expected Behaviour

Running a MLSD {path} or LIST {path} the {path} parameter should be provided to ClientHandlingDriver.ListFiles(). Ideally it should accept an additional argument path string to make: ClientHandlingDriver.ListFiles(cc ClientContext, path string). This would allow the driver to list directories other than the the current working dir on behalf of the client. {path} is specified (at least for MLSD) in RFC 3659.

Current Behaviour

Running a MLSD {path} or LIST {path} command the {path} argument is ignored as best I can tell. The only path you can infer the client is referring to is the ClientContext.Path() which is only set on a CWD or CDUP command. |

Example: if a client logins and immediately issues a MLSD /some/path/other/than/pwd the ClientHandlerDriver.ListFiles() method only can infer a path from the ClientContext which will be /, which is not what the client asked.

Happy to provide a PR if you consider this an issue as I do.

TLS and active FTP transfer

Hello,

I use sftpgo. I noticed, that TLS is working for login and PASV transfers, but then data transfer is unencrypted.
As sftpgo is using ftpserverlib, I suspect bug may be in ftpserverlib.

Filezilla is by default using PASV mode, then one should set Active mode in configuration, before trying to connect.

Summary:

  • login using AUTH TLS works
  • active FTP connection is established (from port 20 to high port of the Filezilla client)
  • active FTP data transfer is unencrypted, then fails, as client expect it to be encrypted the same way as control FTP channel.

Result in Filezilla:

Status: Connecting to 172.31.SER.VER:21...
Status: Connection established, waiting for welcome message...
Response: 220 SFTPGo 1.0.0-dev ready
Command: AUTH TLS
Response: 234 AUTH command ok. Expecting TLS Negotiation.
Status: Initializing TLS...
Status: Verifying certificate...
Status: TLS connection established.
Command: USER test
Response: 331 OK
Command: PASS ********
Response: 230 Password ok, continue
Command: CLNT FileZilla
Response: 200 Good to know
Command: OPTS UTF8 ON
Response: 200 I'm in UTF8 only anyway
Command: PBSZ 0
Response: 200 Whatever
Command: PROT P
Response: 200 OK
Status: Logged in
Status: Retrieving directory listing...
Command: PWD
Response: 257 "/" is the current directory
Command: TYPE I
Response: 200 Type set to binary
Command: PORT 172,31,CLI,ENT,50,172
Response: 200 PORT command successful
Command: MLSD
Response: 150 Using transfer connection
Error: GnuTLS error -15: An unexpected TLS packet was received.
Error: Transfer connection interrupted: ECONNABORTED - Connection aborted
Response: 226 Closing transfer connection
Error: Failed to retrieve directory listing

Unencrypted data (directory listing) can be seen by looking on data transfer from port 20 of the FTP server.

If my OpenFile returns error, second 'STOR' command in sequence fails

If I send multiple files, and the first returns an error I get
229 Entering Extended Passive Mode (|||41111|)
back as an error from my client. Is this an issue with the client or the server?
Is 229 Entering Extended Passive Mode (|||41111|) valid for a store response?
logs:

t=2017-09-21T17:08:20+0200 lvl=dbug msg="FTP SEND" action=ftp.cmd_send id=1 line="220 Welcome, you're on dir /tmp/doc-service-ftp-test"
t=2017-09-21T17:08:20+0200 lvl=dbug msg="FTP RECV" action=ftp.cmd_recv id=1 line="FEAT\r\n"
t=2017-09-21T17:08:20+0200 lvl=dbug msg="FTP SEND" action=ftp.cmd_send id=1 line="211- These are my features"
t=2017-09-21T17:08:20+0200 lvl=dbug msg="FTP SEND" action=ftp.cmd_send id=1 line=" UTF8"
t=2017-09-21T17:08:20+0200 lvl=dbug msg="FTP SEND" action=ftp.cmd_send id=1 line=" SIZE"
t=2017-09-21T17:08:20+0200 lvl=dbug msg="FTP SEND" action=ftp.cmd_send id=1 line=" MDTM"
t=2017-09-21T17:08:20+0200 lvl=dbug msg="FTP SEND" action=ftp.cmd_send id=1 line=" REST STREAM"
t=2017-09-21T17:08:20+0200 lvl=dbug msg="FTP SEND" action=ftp.cmd_send id=1 line="211 end"
t=2017-09-21T17:08:20+0200 lvl=dbug msg="FTP RECV" action=ftp.cmd_recv id=1 line="USER foo\r\n"
t=2017-09-21T17:08:20+0200 lvl=dbug msg="FTP SEND" action=ftp.cmd_send id=1 line="331 OK"
t=2017-09-21T17:08:20+0200 lvl=dbug msg="FTP RECV" action=ftp.cmd_recv id=1 line="PASS bar\r\n"
t=2017-09-21T17:08:20+0200 lvl=dbug msg="FTP SEND" action=ftp.cmd_send id=1 line="230 Password ok, continue"
t=2017-09-21T17:08:20+0200 lvl=dbug msg="FTP RECV" action=ftp.cmd_recv id=1 line="TYPE I\r\n"
t=2017-09-21T17:08:20+0200 lvl=dbug msg="FTP SEND" action=ftp.cmd_send id=1 line="200 Type set to binary"
t=2017-09-21T17:08:20+0200 lvl=dbug msg="FTP RECV" action=ftp.cmd_recv id=1 line="EPSV\r\n"
t=2017-09-21T17:08:20+0200 lvl=dbug msg="FTP SEND" action=ftp.cmd_send id=1 line="229 Entering Extended Passive Mode (|||41077|)"
t=2017-09-21T17:08:20+0200 lvl=dbug msg="FTP RECV" action=ftp.cmd_recv id=1 line="STOR ../create1.txt\r\n"
t=2017-09-21T17:08:20+0200 lvl=dbug msg="FTP SEND" action=ftp.cmd_send id=1 line="150 Using transfer connection"
t=2017-09-21T17:08:20+0200 lvl=dbug msg="FTP Transfer connection opened" action=ftp.transfer_open id=1 remoteAddr=127.0.0.1:47174 localAddr=127.0.0.1:41077
t=2017-09-21T17:08:20+0200 lvl=dbug msg="FTP SEND" action=ftp.cmd_send id=1 line="550 Requested path outside base directory"
t=2017-09-21T17:08:20+0200 lvl=dbug msg="FTP SEND" action=ftp.cmd_send id=1 line="226 Closing transfer connection"
t=2017-09-21T17:08:20+0200 lvl=dbug msg="FTP Transfer connection closed" action=ftp.transfer_close id=1
t=2017-09-21T17:08:20+0200 lvl=dbug msg="FTP RECV" action=ftp.cmd_recv id=1 line="EPSV\r\n"
t=2017-09-21T17:08:20+0200 lvl=dbug msg="FTP SEND" action=ftp.cmd_send id=1 line="229 Entering Extended Passive Mode (|||41111|)"
t=2017-09-21T17:08:20+0200 lvl=dbug msg="FTP RECV" action=ftp.cmd_recv id=1 line="PASV\r\n"
t=2017-09-21T17:08:20+0200 lvl=dbug msg="FTP SEND" action=ftp.cmd_send id=1 line="227 Entering Passive Mode (127,0,1,1,180,93)"
	driver_test.go:93: Failed to save simple file: 229 Entering Extended Passive Mode (|||41111|)

Dependabot couldn't find a Dockerfile for this project

Dependabot couldn't find a Dockerfile for this project.

Dependabot requires a Dockerfile to evaluate your project's current Docker dependencies. It had expected to find one at the path: /Dockerfile.

If this isn't a Docker project, or if it is a library, you may wish to disable updates for it in the .dependabot/config.yml file in this repo.

View the update logs.

reload-config file

Whenever you change the configuration (eg: change a password, add or delete an account) you must recompile everything, stop the server and start it! All files being sent will be lost when we stop the server :(

  1. How can we reload the server-config (eg: a task to reload-config every 10 minutes)
  2. Add the possibility to start / stop / restart / reload the server

Eg: ftpserver -start

if ServerIsStarted() {
  fmt.Println("Server is already started")
} elseif err := FtpStart(); err != nil{
  fmt.Errorf("Error! %v", err)
} else {
  fmt.Println("OK server started successful")
}

Same thing with -Stop, -Restart, -Reload

Drop log15 ?

I saw a fork ( https://github.com/AntiPaste/ftpserver ) that dropped log15 logging.

I don't think this is a good idea, I find structured logging to be a very important concept of any modern software but I wonder what you think about it.

Please vote:

  • 👎 for keeping log15
  • 👍 for dropping log15

how to add IDLE timeout

it's not an issue but a request

how to add IDLE timeout. Example a client connects but does not execute any command. After x seconds it is disconnected automatically

Callbacks on File events

I am fairly new to Golang so not sure if this is possible at all. My question is it possible to define/invoke callback functions once certain file related events complete - such as upload, delete, rename etc.

LIST -a option

Hi,

I tested the library with curlftpfs it issue a LIST command passing the -a option and ftpserverlib search for a non-existing -a directory. Based on the doc here:

"""
High-quality clients do not attempt to use these features. File selection and sorting are user-interface features that can and should be handled by the client.
"""

this simple patch fixes the issue

diff --git a/handle_dirs.go b/handle_dirs.go
index 16187e7..13f2f15 100644
--- a/handle_dirs.go
+++ b/handle_dirs.go
@@ -88,6 +88,10 @@ func (c *clientHandler) handlePWD() error {
 }
 
 func (c *clientHandler) handleLIST() error {
+	param := strings.ToLower(c.param)
+	if param == "-a" || param == "-l" || param == "-al" || param == "-la" {
+		c.param = ""
+	}
 	if files, err := c.getFileList(); err == nil || err == io.EOF {
 		if tr, errTr := c.TransferOpen(); errTr == nil {
 			err = c.dirTransferLIST(tr, files)
-- 
2.28.0

this is basically the same way that pyftpdlib handle this.

I'm unsure if add support for this feature and, if we decide to include this patch, if add a check to verify that a directory named -a, -l etc. exists before discarding the param. What do you think about? Thank you

Support for running behind load balancers / Kubernetes

As you are obviously aware, the PASV command must return a public ip back to the client as part of its response.

This is currently supported by the PublicHost setting, or the fallback to the LocalAddr() on net.Conn object.

However if running in Kubernetes cluster behind a load balancer on AWS (either an ELB or NLB) there will be multiple public IPs for a given hostname.
This can be problematic as the IP the ftp server resolves the hostname to may not be the same one the client resolved to.
This is a real problem as we've seen clients in the wild error out if the returned PASV IP did not match the IP it had connected to.

Fortunately both the ELB and NLB support the proxy protocol, which can be used to retrieve the private ip of the load balancer, and then lookup its corresponding public ip.

Rather than building all of that functionality into the ftp server its self, I think it would make more sense to allow the user to provide an existing net.Listener, and a resolver function to return a public ip.

I've submitted a PR that modifies the ClientContext to expose the LocalAddr, and a PublicIpResolver that takes said context and can return a public ip.

I'm open to other ways of addressing this, but I figured this was the smallest change possible that kept the specifics of my use case out of scope of this project.

What is the variable "NonStandardActiveDataPort"?

in file ftpserver/server/driver.go

What is the variable "NonStandardActiveDataPort"?

type Settings struct {
...
	**NonStandardActiveDataPort** bool _// Allow to use a non-standard active data port_
...
}

An example for "... non-standard active data port" ?

Thank

An error occurs when connecting to the server

Google Translation: French-> English

An error occurs when connecting to the server with "Active" mode and SSL or TLS encrypted connection

Example: Configure FileZilla Ftp Client with:

-Transfer mode: Active
-Encryption: SSL or TLS

=> Error detected: Unable to connect to the server (with a login and password) with this configuration.

Thanks

Your .dependabot/config.yml contained invalid details

Dependabot encountered the following error when parsing your .dependabot/config.yml:

Automerging is not enabled for this account. You can enable it from the [account settings](https://app.dependabot.com/accounts/fclairamb/settings) screen in your Dependabot dashboard.

Please update the config file to conform with Dependabot's specification using our docs and online validator.

support active mode

It would be good if the ftpserver library supports active mode for clients which are not allowed to use passive mode.

Flow:

FTP server's port 21 from anywhere (Client initiates connection)
FTP server's port 21 to ports > 1023 (Server responds to client's control port)
FTP server's port 20 to ports > 1023 (Server initiates data connection to client's data port)
FTP server's port 20 from ports > 1023 (Client sends ACKs to server's data port)

Command: PORT 192,168,150,80,14,178

Memory leak: file is not closed if TransferOpen() fails

https://github.com/fclairamb/ftpserver/blob/d06e1a0d4a20bf8a311961952b15c550a1acbfd9/server/handle_files.go#L22-L35

The file opened on line 22 is never closed if the TransferOpen() call fails.
You could of course call file.Close() within that else block, my only concern is that still results in an empty file being created.

Another option is to call TransferOpen() first, and then attempt to open the file.

I can submit a PR once a solution is agreed upon.

[Discussion] Handle client sudden disconnections

Following up on #70 ...

If a client disconnects halfway through the upload, the current driver doesn't allow to detect anything.

Subjects to address:

  • Do we have a correct way to handle that in the first place? What is the difference between a successful upload and one that isn't? If a TCP connection stops halfway through, how can we detect it?

Provide correct error codes

Right now this library returns always returns StatusSyntaxErrorNotRecognised (500) if the command returns an error.

If a file is not found then for example a 550 should be returned.

Maybe this can be guessed by the returned error (for example os.ErrNotExist)?

For custom filesystems it might be good to have a custom error type that can include an error code.

Various improvements from #63

Various improvements from #63:

  • Map initialization
  • daddy --> server renaming
  • Dial timeout for active connections
  • Constants declaration
  • Comments fixes
  • SetDeadLine improvement
  • Variable shadowing
  •  Clean shutdown of server and the associated clients

configurable port ranges for passive mode

As FTP server could be packed into a docker container which requires exposed ports to be specified, it would be nice if the ftpserver allow setting of port range in the Settings.

This is how passive mode port is taken in the current implementation In transfer_pasv.go:

	addr, _ := net.ResolveTCPAddr("tcp", ":0")
	tcpListener, err := net.ListenTCP("tcp", addr)
	if err != nil {
		log15.Error("Could not listen", "err", err)
		return
	}

This is how pure-ftpd is looking for free port for data in the provided range:

    on = 1;
    if (setsockopt(datafd, SOL_SOCKET, SO_REUSEADDR,
                   (char *) &on, sizeof on) < 0) {
        error(421, "setsockopt");
        return;
    }
    dataconn = ctrlconn;
    for (;;) {
        if (STORAGE_FAMILY(dataconn) == AF_INET6) {
            STORAGE_PORT6(dataconn) = htons(p);
        } else {
            STORAGE_PORT(dataconn) = htons(p);
        }
        if (bind(datafd, (struct sockaddr *) &dataconn,
                 STORAGE_LEN(dataconn)) == 0) {
            break;
        }
        p--;
        if (p < firstport) {
            p = lastport;
        }
        if (p == firstporttried) {
            (void) close(datafd);
            datafd = -1;
            addreply_noformat(425, MSG_PORTS_BUSY);
            return;
        }
    }

Wrong response code for MKD

I've been tracking down a small issue. I'm using this ftp server in a new project and using a different ftp client to test it: github.com/jlaffaye/ftp.

Calling mkdir on the FTP server (MKD) replies with 250 which the ftp client assumes is an error since it expects 257 as defined in RFC 959 (https://www.w3.org/Protocols/rfc959/5_Declarative.html).

I'm unsure if it's best to modify the response from the server to be 257 or for the ftp client to accept either 250 and 257. What's your preference?

Thanks, this is a nice little ftp server package.

Add support for "HELP SITE"

level=debug ts=2019-12-21T22:35:24.620081Z caller=client_handler.go:150 component=server clientId=3 msg="FTP RECV" action=ftp.cmd_recv line="HELP SITE\r\n"
level=debug ts=2019-12-21T22:35:24.620131Z caller=client_handler.go:226 component=server clientId=3 msg="FTP SEND" action=ftp.cmd_send line="500 Unknown command"

Data dir setting not working

Hi,
on current master there is problem with -data cli parameter. It is passed, but at the end always / (main directory) is served.

This is quite huge problem. ;-)

File listing adds some space

As @mgenov noted, file listing adds some spaces. And it also doesn't seem to respect the formating with the year for more than 6 months old files.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.