WebDAV directory listing is too slow about cells HOT 5 OPEN

I tried the same operation with fusion s3fs connected as local folder, the result are more worst, about 57 minutes for get the listing.

I found project sftpgo, which support s3 and wabdav (written on go), and for this folder result - 1 second:

curl -X PROPFIND --user 'user:pass' 'https://example.com/dav/e2-store/backup' --basic -o backup_listing
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 4564k    0 4564k    0     0  2798k      0 --:--:--  0:00:01 --:--:-- 2797k

I believe you can compare they're webdav implementation and yours and improve your version.

PS Environment is the same.

from cells.

Hi @archekb

Thanks for your reports.
As a disclaimer, DAV is a crappy API and heavily depends on the DAV client performing many requests. Same for fuse-based connectors that may trigger multiple "stat" requests on each files to build their listing info. Furthermore, listing 200 files vs. 7500 is a very different operation, even page by page. And I am not even pointing your "sftpgo" example, that is probably listing a local folder, where as Cells is listing an index, loading metadata from various services, and checking permissions (ACL) during the same flow. So you may compare apple and oranges here.

That said, the DAV endpoint may need some performance optimisation, as we applied to the main API used by the frontend (heavy use of caching).

In between we would recommend using the cells-client if you want to interact programmatically with the server (uses the same APIs as frontend)

-c

from cells.

Hi, @cdujeu , thanks for your reply.

heavily depends on the DAV client

In all cases I used curl for getting listing, the conditions was equal s3 store and get listing for folder with 7500 files and we are talking about one operation aka one request PROPFIND. I believe we can skip discussion about clients. (I was test a few clients and behavior was the same.)

Furthermore, listing 200 files vs. 7500 is a very different operation, even page by page

I believe you can optimize current mechanism, for webdav, and make 38 requests to function which prepare listing for html frontend (I believe it do the same loading metadata from various services, and checking permissions (ACL)), merge it together and convert results to webdav xml response. And I believe it will more faster then 25 minutes.

And this is real problem, when curl can wait 25 minutes for listing, but other webdav clients can't wait 25 minutes for listing, usually it's timed out after 600 sec, maybe earlier.

"sftpgo" example, that is probably listing a local folder

No, my bad, I do not describe it clearly, but it's still S3-compatible iDrive E2 storage. "sftpgo" prepare listing for folder with 7500 files from s3 storage less than 1 sec. Even if it were a local folder, Cells do the same for 2 min 40 seconds seconds, it is 160x times slower. But actually it was s3 storage, and Cells was for 1500x times slower.

So you may compare apple and oranges here.

I compare two application which allows me to get access to my s3 storage via webdav. I believe it's reasonable compare.

we would recommend using the cells-client

Thanks, but my task is to take a backup the phone, customers, many times ask you implement this simple feature, which allows automatically sync files from phone with cloud by schedule here, here, here and here. But who's care?

That's why I should use third party software and webdav for sync my files with cloud.

from cells.

Correct config NginX proxy for WebDAV:

 location /dav/ {
        proxy_buffering off;
        proxy_request_buffering off;
        proxy_pass_request_headers on;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;

        # Proxy timeouts between successive read/write operations, not the whole request.
        proxy_connect_timeout              10m;
        proxy_send_timeout                 10m;
        proxy_read_timeout                 10m;
        proxy_pass https://localhost:8443;
    }

This mandatory, if you don't wanna have a randoms files rejects and errors in NginX logs:

        proxy_buffering off;
        proxy_request_buffering off;
        proxy_set_header Host $host;

@cdujeu you can add it in to NginX proxy doc section, because I spend a few days for proxy_request_buffering, because error from Nginx was [error] 98#98: *15 SSL_write() failed (32: Broken pipe) while sending request to upstream, request: "PUT /dav/... HTTP/2.0", upstream:...

from cells.

hi @archekb we'll dig deeper for the performances - it's probable that the current implementation (based on golang standard dav library that wraps our internal datasources are virtual filesystem) does make unnecessary operations (like OpenFile when it's all about getting stats).

from cells.