Comments (5)
I tried the same operation with fusion s3fs connected as local folder, the result are more worst, about 57 minutes for get the listing.
I found project sftpgo, which support s3 and wabdav (written on go), and for this folder result - 1 second:
curl -X PROPFIND --user 'user:pass' 'https://example.com/dav/e2-store/backup' --basic -o backup_listing
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 4564k 0 4564k 0 0 2798k 0 --:--:-- 0:00:01 --:--:-- 2797k
I believe you can compare they're webdav implementation and yours and improve your version.
PS Environment is the same.
from cells.
Hi @archekb
Thanks for your reports.
As a disclaimer, DAV is a crappy API and heavily depends on the DAV client performing many requests. Same for fuse-based connectors that may trigger multiple "stat" requests on each files to build their listing info. Furthermore, listing 200 files vs. 7500 is a very different operation, even page by page. And I am not even pointing your "sftpgo" example, that is probably listing a local folder, where as Cells is listing an index, loading metadata from various services, and checking permissions (ACL) during the same flow. So you may compare apple and oranges here.
That said, the DAV endpoint may need some performance optimisation, as we applied to the main API used by the frontend (heavy use of caching).
In between we would recommend using the cells-client if you want to interact programmatically with the server (uses the same APIs as frontend)
-c
from cells.
Hi, @cdujeu , thanks for your reply.
heavily depends on the DAV client
In all cases I used curl for getting listing, the conditions was equal s3 store and get listing for folder with 7500 files and we are talking about one operation aka one request PROPFIND
. I believe we can skip discussion about clients. (I was test a few clients and behavior was the same.)
Furthermore, listing 200 files vs. 7500 is a very different operation, even page by page
I believe you can optimize current mechanism, for webdav, and make 38 requests to function which prepare listing for html frontend (I believe it do the same loading metadata from various services, and checking permissions (ACL)
), merge it together and convert results to webdav xml response. And I believe it will more faster then 25 minutes.
And this is real problem, when curl can wait 25 minutes for listing, but other webdav clients can't wait 25 minutes for listing, usually it's timed out after 600 sec, maybe earlier.
"sftpgo" example, that is probably listing a local folder
No, my bad, I do not describe it clearly, but it's still S3-compatible iDrive E2 storage. "sftpgo" prepare listing for folder with 7500 files from s3 storage less than 1 sec. Even if it were a local folder, Cells do the same for 2 min 40 seconds seconds, it is 160x times slower. But actually it was s3 storage, and Cells was for 1500x times slower.
So you may compare apple and oranges here.
I compare two application which allows me to get access to my s3 storage via webdav. I believe it's reasonable compare.
we would recommend using the cells-client
Thanks, but my task is to take a backup the phone, customers, many times ask you implement this simple feature, which allows automatically sync files from phone with cloud by schedule here, here, here and here. But who's care?
That's why I should use third party software and webdav for sync my files with cloud.
from cells.
Correct config NginX proxy for WebDAV:
location /dav/ {
proxy_buffering off;
proxy_request_buffering off;
proxy_pass_request_headers on;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
# Proxy timeouts between successive read/write operations, not the whole request.
proxy_connect_timeout 10m;
proxy_send_timeout 10m;
proxy_read_timeout 10m;
proxy_pass https://localhost:8443;
}
This mandatory, if you don't wanna have a randoms files rejects and errors in NginX logs:
proxy_buffering off;
proxy_request_buffering off;
proxy_set_header Host $host;
@cdujeu you can add it in to NginX proxy doc section, because I spend a few days for proxy_request_buffering
, because error from Nginx was [error] 98#98: *15 SSL_write() failed (32: Broken pipe) while sending request to upstream, request: "PUT /dav/... HTTP/2.0", upstream:...
from cells.
hi @archekb we'll dig deeper for the performances - it's probable that the current implementation (based on golang standard dav library that wraps our internal datasources are virtual filesystem) does make unnecessary operations (like OpenFile when it's all about getting stats).
from cells.
Related Issues (20)
- Error when sharing a folder HOT 1
- Cells ( v4.2.0) does not build on Go 1.20 HOT 2
- [4.2] Public Link for directory is broken if "enable search in minisite is turned on" HOT 7
- Pydio Cells на russian
- Error “Pydio is not defined” when run build-core command in gui.ajax folder HOT 9
- Cannot login on `dev-next` docker image
- Error when Install via Helm HOT 2
- Charset / utf8mb4 related trouble running cells configure
- Password change fails when forcing MFA at the same time HOT 1
- Enhancement: language support for collabaro (editor libreoffice) HOT 1
- docker image: variable for database location and database passord
- cannot install (cells.exe configure) HOT 2
- Docs for k8s installs HOT 12
- writing files to webdav share / mountpoint of public links creates empty files (locking enabled)
- Remember me feature is not available
- docker image does not recognise timezone appropriately HOT 2
- Policy applied on shared cell is also affecting own cell HOT 1
- Web Interface Menu point of return to main page from cells console are missed if turn off home section HOT 2
- The circle that allows the user to navigate through an audio file during preview disappear HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cells.