ezhov-evgeny / webdav-client-python-3 Goto Github PK
View Code? Open in Web Editor NEWEasy to use WebDAV Client for Python 3.x
License: Other
Easy to use WebDAV Client for Python 3.x
License: Other
Are these two attributes ignored? They don't seem to be in the code base in any functional way unless I've missed something obvious.
When listing directories, it looks like connection pooling isn't being used.
Since I am listing a lot of directories, I am cycling through the ephemeral port list around 20 times, before doing anything else.
I would probably significantly reduce time and load on both ends, if I was able to reuse a single connection, instead of spawning 200000.
I believe the issue why connection pooling isn't being used automatically (as advertised by requests/urllib3) is that the connections created are always streaming, even if it's just a PROPFIND, which doesn't really benefit from the streaming.
My proposed solution would be to make the streaming flag in the request conditional upon the method being called. stream=True
for actual up/downloads, and stream=False
for list
Hi,
I have a question regarding the download method which calls def download_file(self, remote_path, local_path, progress=None):
. When I download a tar.gz file from a WebDAV server, the file is much bigger than on the WebDAV server. I realized that the file getting extracted during the download process. I read that this can happen if you are not using the .read method:
f.write(response.raw) instead of f.write(response.raw.read())
Is this a normal behavior? Is it possible to download a file without getting extracted?
Thanks in advance!
Currently, when calling list()
or info()
the client does not attempt to parse the <d:getcontenttype>
property.
This is odd, because it is part of the standard.
It woud be very useful if it could be parsed and returned in the result.
Is there any other way I can access this property in the response?
Situation:
Need to put single file into multiple accounts using webdav.
Implementation:
Connecting to accounts using threads (3-4). Each thread create client = Client(options)
and running small download to test connections. There are many clients so each
thread is doing same operation but for different client (limit on 3-4 threads not saturate network link.)
Problem:
CopyThread::__copy_file [Thread-2]!(['Test_Client01_2020-01.zip'])
CopyThread::__copy_file [Thread-1]!(['Test_Client02_2020-01.zip'])
CopyThread::__copy_file [Thread-3]!(['Test_Client05_2020-01.zip'])
CopyThread::__copy_file [Thread-1]!(['Test_Client03_2020-01.zip'])
CopyThread::__copy_file [Thread-2]!(['Test_Client04_2020-01.zip'])
except CopyThread::__process_package [Thread-2]! Remote resource: / not found
CopyThread::__copy_file [Thread-3]!(['Test_Client07_2020-01.zip'])
CopyThread::__copy_file [Thread-1]!(['Test_Client08_2020-01.zip'])
except CopyThread::__process_package [Thread-1]! Remote resource: / not found
.......
My quick solution:
Move {root, session, timeout verify} from static class Client variables into local object variables
so they in init function {self.root, self.session, self.timeout, self.verify}
This seems to solve my problem.
If I want to upload or download massive files, typically 100k.
And I would like to make sure it only spawn at most 5 concurrent requests at the same time(from the server side aspect).
What is the recommendation of this implementation?
Can I set maximum concurrent requests number?
And if I use python Threading to multi-thread it,
should they share one client instance(and use download_async) or spawn for each thread worker(and use download_sync)?
Thank you
Hey this is a great library and been using it extensively in scripts, but I would like to propose the option to authorise using a header, Authorization: Bearer $token
instead of Authorization: Basic $base64auth
.
Would this be feasible?
webdav-client-python-3/webdav3/client.py
Line 710 in 96aebf2
Return sentence should be outside the for loop, it's causing the method to push only one file at a time.
I am using this module along with pythongssapi/requests-gssapi
, unfortunately the module is not threadsafe and I have to perform my requests with a ThreadPoolExecutor
. Interleaving threads break requests. My basic code passes auth
always to the session request now. Unfortunately, this module does not allow that. Or if you want to use alternative credentials this is not possible.
My request is to pass this through and if auth
is None
use the one from options/session else this one.
При работе с сервером NextCloud, если в названии файла или папки есть кириллица, то Client.move() или Сlient.clean() падают с ошибкой UnicodeEncodeError: 'latin-1' codec can't encode characters in position 52-55: ordinal not in range(256). При этом другие функции (.mkdir(), .info(), .list(), .download_file(), .upload_file()) работают нормально
This snippet:
webdav-client-python-3/webdav3/client.py
Lines 206 to 207 in 4fcfcbc
This spills logs w/o any reason:
2021-04-30 11:42:06,358 [documents-job-worker_2] ERROR requests_gssapi.gssapi_: handle_other(): Mutual authentication unavailable on 403 response
2021-04-30 11:42:06,417 [documents-job-worker_2] ERROR requests_gssapi.gssapi_: handle_other(): Mutual authentication unavailable on 403 response
2021-04-30 11:42:06,547 [documents-job-worker_2] ERROR requests_gssapi.gssapi_: handle_other(): Mutual authentication unavailable on 403 response
2021-04-30 11:42:06,661 [documents-job-worker_2] ERROR requests_gssapi.gssapi_: handle_other(): Mutual authentication unavailable on 403 response
2021-04-30 11:42:06,800 [documents-job-worker_2] ERROR requests_gssapi.gssapi_: handle_other(): Mutual authentication unavailable on 403 response
2021-04-30 11:42:06,859 [documents-job-worker_2] ERROR requests_gssapi.gssapi_: handle_other(): Mutual authentication unavailable on 403 response
2021-04-30 11:42:06,997 [documents-job-worker_2] ERROR requests_gssapi.gssapi_: handle_other(): Mutual authentication unavailable on 403 response
2021-04-30 11:42:07,055 [documents-job-worker_2] ERROR requests_gssapi.gssapi_: handle_other(): Mutual authentication unavailable on 403 response
2021-04-30 11:42:07,188 [documents-job-worker_2] ERROR requests_gssapi.gssapi_: handle_other(): Mutual authentication unavailable on 403 response
Hi,
the verify option does not work because and option request is sent in the constructor thus there is no way to set the verify to false before that.
Few weeks ago I developed an app using the module and worked flawless, but from a few days ago, any time I try to download any .vcf file from contacts, it shows that error.
The weird thing is that I can list the files perfectly.
This is the client initialization method:
` def get_client(self):
options = {
'webdav_hostname': self.env.user.company_id.nc_url,
'webdav_login': self.env.user.company_id.nc_user,
'webdav_password': self.env.user.company_id.nc_passwd
}
client = Client(options)
client.verify = False
return client`
Since this commit, the Client
constructor performs a request to the WebDAV server on construction.
This is pretty annoying as it now breaks a lot of unit tests on my side where I used dumb URLs for wrapper object, and I am sure it will impact other users as well.
Would it be possible to add a constructor argument to bypass this?
I am curious why methods like upload_to, upload, etc only return None instead of returning the response. I am happy to write a patch to return the response if it would be acceptable.
My use case: I want to check the return code to ensure it is a clean 200 after doing an upload.
Thanks for the library!
To reproduce, try creating an empty file on a server that supports gzip encoding, then download it using download_from
:
from io import BytesIO
buf = BytesIO()
client.download_from(buf, 'http://example.com/empty-file')
print(buf.getvalue())
Instead of empty content (b''
), I'm getting b'\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00'
, which is gzipped empty content.
It looks like the method uses response.raw
, which does not decode gzip. A demonstration (https://httpbin.org/gzip should reply with a JSON body):
>>> import requests
>>> response = requests.get('https://httpbin.org/gzip', stream=True)
>>> response.raw.read()
b'\x1f\x8b\x08\x00\xf7\xa7\xa1^\x02\xff= ...
would be nice to have an option in the function to get the file list recursively should be as simple as adding a header
Depth = infinity
just a thought..
Aaron
Hi, I am new to webdav, and I just spent hours to get client.list() to work, then I just find out that I have a tailing slash in config['webdav_hostname'] that makes me failed. It has a tailing slash because the URL from my service provider has it and I didn't think it matters until now. Should the library have the ability to handle tailing slash?
Hello! I am using webdav-client-python-3
to talk to a WebDAV server - it's working really great! However, the hardcoded timeout
value of 30 sec used in the HTTP requests is too short for my use case. Would it be possible to add the time out value as an optional setting to initialise the client? Thanks in advance!
Hi,
When I tried to download a file using the code below, I got a RemoteResourceNotFound
error:
from webdav3.client import Client
webdav = Client({
"webdav_hostname": "XXXX",
"webdav_login": "XXXX",
"webdav_password": "XXXX"
})
webdav.download_sync(remote_path="some filename", local_path="some path")
Traceback (most recent call last):
[..]
File "/home/bauer/conda/lib/python3.7/site-packages/webdav3/client.py", line 409, in download_sync
self.download(local_path=local_path, remote_path=remote_path)
File "/home/bauer/conda/lib/python3.7/site-packages/webdav3/client.py", line 349, in download
if self.is_dir(urn.path()):
File "/home/bauer/conda/lib/python3.7/site-packages/webdav3/client.py", line 66, in _wrapper
res = fn(self, *args, **kw)
File "/home/bauer/conda/lib/python3.7/site-packages/webdav3/client.py", line 622, in is_dir
self._check_remote_resource(remote_path, urn)
File "/home/bauer/conda/lib/python3.7/site-packages/webdav3/client.py", line 610, in _check_remote_resource
raise RemoteResourceNotFound(remote_path)
webdav3.exceptions.RemoteResourceNotFound: Remote resource: /HUUD55DU.zip not found
After some debugging I realized that the source of this error was not a missing file, but wrong credentials. More specifically the password was not correct. I think the error is misleading here, since it does not point to an authentication problem. Additionally, I wonder if there is a way to verify login credentials before making any other request? I didn't find one.
I find that the last element of the path is appended and causes a Remote resource error
.
Example code and response:
# Imports, requires `pip install webdavclient3`
from webdav3.client import Client
# Set options
options = {
'webdav_hostname':
'https://owncloud.local/remote.php/dav/files/username',
'webdav_login' : 'username',
'webdav_password': 'token'
}
# Set client
client = Client(options)
# List root
client.list("/")
[
'username/',
'BOXER/',
'Documents/'
]
# Download
client.download_sync(remote_path="BOXER/", local_path="~/BOXER-local")
---------------------------------------------------------------------------
RemoteResourceNotFound Traceback (most recent call last)
<ipython-input-47-2d7b3e02ed57> in <module>
21
22 # Download
---> 23 client.download_sync(remote_path="BOXER/",
local_path="~/BOXER-local")
/opt/conda/lib/python3.7/site-packages/webdav3/client.py in
download_sync(self, remote_path, local_path, callback)
395 :param callback: the callback which will be invoked when
downloading is complete.
396 """
--> 397 self.download(local_path=local_path,
remote_path=remote_path)
398 if callback:
399 callback()
/opt/conda/lib/python3.7/site-packages/webdav3/client.py in download(self,
remote_path, local_path, progress)
338 urn = Urn(remote_path)
339 if self.is_dir(urn.path()):
--> 340 self.download_directory(local_path=local_path,
remote_path=remote_path, progress=progress)
341 else:
342 self.download_file(local_path=local_path,
remote_path=remote_path, progress=progress)
/opt/conda/lib/python3.7/site-packages/webdav3/client.py in
download_directory(self, remote_path, local_path, progress)
362 _remote_path =
"{parent}{name}".format(parent=urn.path(), name=resource_name)
363 _local_path = os.path.join(local_path, resource_name)
--> 364 self.download(local_path=_local_path,
remote_path=_remote_path, progress=progress)
365
366 @wrap_connection_error
/opt/conda/lib/python3.7/site-packages/webdav3/client.py in download(self,
remote_path, local_path, progress)
337 """
338 urn = Urn(remote_path)
--> 339 if self.is_dir(urn.path()):
340 self.download_directory(local_path=local_path,
remote_path=remote_path, progress=progress)
341 else:
/opt/conda/lib/python3.7/site-packages/webdav3/client.py in _wrapper(self,
*args, **kw)
68 log.debug("Requesting %s(%s, %s)", fn, args, kw)
69 try:
---> 70 res = fn(self, *args, **kw)
71 except requests.ConnectionError:
72 raise NoConnection(self.webdav.hostname)
/opt/conda/lib/python3.7/site-packages/webdav3/client.py in is_dir(self,
remote_path)
607 urn = Urn(remote_path)
608 parent_urn = Urn(urn.parent())
--> 609 self._check_remote_resource(remote_path, urn)
610
611 response = self.execute_request(action='info',
path=parent_urn.quote())
/opt/conda/lib/python3.7/site-packages/webdav3/client.py in
_check_remote_resource(self, remote_path, urn)
595 def _check_remote_resource(self, remote_path, urn):
596 if not self.check(urn.path()) and not
self.check(Urn(remote_path, directory=True).path()):
--> 597 raise RemoteResourceNotFound(remote_path)
598
599 @wrap_connection_error
RemoteResourceNotFound: Remote resource: /BOXER/BOXER/ not found
Should be included to be complete
When parent directory has too many files more than one thousand, Client.is_dir will be very slowly.
I check the source code, is_dir execute request with action info and parent path and depth 1, and response will return all files in parent directory.
I try execute request with remote path and depth 0, and it works.
Methods of Client
, .list, .info, .download*, .pull, .push, ...
, etc., fail when the webdav_root
is actually needed to quote. For example, we have a webdav_root='/remote.php/webdav/steph(2021)'
, in which the parentheses are actually needed to quote, so we get a quoted webdav_root, '/remote.php/webdav/steph%282021%29', and next, Client.list(path)
fails to exclude the target path itself, Client.info
fails to find resources, ....
This issue can be easily reproduced with any version (until now latest 3.14.6) of client, a nextcloud server, the webdav_root mentioned above.
Here is a screenshot about the issue:
The true structure of files in the remote is:
steph(2021)/
└── file1.txt
Is this library compatiple with nextcloud? I'm experiencing very odd+wrong behaviour.
In the code below I first check for existing and non-existing directories and files. Afterwards I create a new directory and sync a single file.
Based on that test, every tested function doesn't work as I expect it to based on the readme,
check
returns always true for files and always false for directories. Except if the directory doesn' exist. In that case it returns true.mkdir
returns true, indicating that the directory was created, but thats not the case (check with the nextcloud webview)upload_sync
doesn't upload the file but also raises no error.`
def issue():
c = {
'webdav_hostname': config.nextcloud_url,
'webdav_login': config.username,
'webdav_password': config.password
}
client = Client(c)
# exists
print('Check existing: expect 3x True')
print(client.check('Photos'))
print(client.check('Nextcloud.png'))
print(client.check('Photos/Frog.jpg'))
#doesn't exist
print('Check non-existing: expect 3x False')
print(client.check('test_Documents'))
print(client.check('test3.pdf'))
print(client.check('test_Photos/Frog.jpg'))
# create
r = client.mkdir('test_Documents')
r2 = client.check('test_Documents')
print('Add new dir: expect 2x True')
print(r, r2)
client.upload_sync(remote_path='test3.pdf', local_path='../test3.pdf')
r3 = client.check('test3.pdf')
print('Check sync file: expect True')
print(r3)
print('Check new dir, synced file')
client = Client(c)
r = client.check('test_Documents')
r2 = client.check('test3.pdf')
print('expect 2x True')
print(r, r2)
Output:
Check existing: expect 3x True
False
True
True
Check non-existing: expect 3x False
True
True
True
Add new dir: expect 2x True
True True
Check sync file: expect True
True
Check new dir + synced file: expect 2x True
False True
I am trying to list the files of another computer and I am not able to do that. Do I need to install NGINX?
Hi,
I have a djangodav webdav server
https://github.com/MnogoByte/djangodav
I upload files with this client.
That results in the following console log:
[19/Feb/2021 14:27:24] "HEAD /auth/webdav/24/input/ HTTP/1.1" 200 0
[19/Feb/2021 14:27:26] "PUT /auth/webdav/24/input/Freilauf.CATPart HTTP/1.1" 201 0
[19/Feb/2021 14:27:26] code 400, message Bad request syntax ('0')
[19/Feb/2021 14:27:26] "0" 400 -
the upload works perfectly. But what does the Bad request message mean?
Does this clients is sending a bad request in not the corryt syntax, or are the requests correct and the problem is, that djangodav is just expecting something different?
Stackoverflow is not in unison about the question if webdav servers must implement the HEAD request on folders: https://stackoverflow.com/questions/16578345/should-webdav-server-support-head-on-folders
Anyway, the webdav server in my Synology NAS does not implement HEAD on folders. It answers with status code 404 on the root folder and status code 403 on nested folders. If I do a PROPFIND on the same path it gives me information about the path.
(See https://gist.github.com/Phaiax/1b186336eb3c6d457705a1564ed112a4 for some curl tests
Edit: I tried it all with trailing /
and without, the results were the same.)
This has the effect that I can't list folders because list()
first checks if the folder exists:
if directory_urn.path() != Client.root and not self.check(directory_urn.path()):
raise RemoteResourceNotFound(directory_urn.path())
Check uses the HEAD
method on the folder, and that fails.
I propose that list()
should call is_dir()
instead of check()
. Also, the info
action should also set Depth=0
instead of Depth=1
in the header because the directory listing is not needed in the cases where the info
action is used. Thirdly, document that not all servers support check()
on folders, even if the RFC says otherwise.
Thanks, Daniel
I have the following problem, I'm connecting to a propietary WebDAV service, but the service is in a path that wasn't root and root directory answers a 404 error.
In the 0.14 release the client in the __init__
function make a test connection to the root
webdav-client-python-3/webdav3/client.py
Line 156 in 7db1cfd
In my special case, it raises a NotFound error when connect, so maybe is possibly to give a root path or a config to not test the connection at start.
Thanks!
while using the webdav-client for python 3 I experience an issue very similar to CloudPolis#18 and CloudPolis#22.
I can list contents of a directory and check if a file exists but cannot get info or perform any resource operations (rename, move, etc). Uploading goes without any issues.
I have tried setting the root to /remote.php/webdav as suggested in the CloudPolis solutions but that seems to limit the number of folders I can see. It does reveal the info.
The webdav server I am trying to connect to called Stack and is a service of TransIP stack
The version of the client is webdavclient3==0.12.
The push method does not remove the trailing '' from the end of windows directroy names returned by listdir before formatting the remote path. This results in an invalid character error being returned by the webdav server.
To reporduce:
client.push
with local_directory
as your new top level directory.This will return something like:
webdav3.exceptions.ResponseErrorCode: Request to https://your-webdav-domain.example/remote.php/dav/files/your-user/test_folder1/test_folder2%5C/ failed with code 400 and message: b'<?xml version="1.0" encoding="utf-8"?>\n<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns" xmlns:o="http://owncloud.org/ns">\n <s:exception>OCA\\DAV\\Connector\\Sabre\\Exception\\InvalidPath</s:exception>\n <s:message>File name contains at least one invalid character</s:message>\n <o:retry xmlns:o="o:">false</o:retry>\n <o:reason xmlns:o="o:">File name contains at least one invalid character</o:reason>\n</d:error>\n'
client.mkdir('test1/test2/test3')
webdav3.exceptions.RemoteParentNotFound: Remote parent for: test1/test2/test3 not found
It looks like only basic auth is allowed form https://github.com/ezhov-evgeny/webdav-client-python-3/blob/develop/webdav3/client.py#L217
Is it possible to allow other auth methods like NTLM, e.g.:
client.session.auth = HttpNtlmAuth(username, password)
HI, when I download file from webdav server with, an empty file downloaded by download_sync
.
It is very difficult to reappear. And I have a question about https://github.com/ezhov-evgeny/webdav-client-python-3/blob/09ae1414ddd331775686ebe55a625e732ef78b14/webdav3/client.py#L195 , Is it only when response.status_code == 507, == 404 , == 405, >= 400 raise exceptions? Is it possible response other status_code when download file(GET)? If other status_code come, https://github.com/ezhov-evgeny/webdav-client-python-3/blob/09ae1414ddd331775686ebe55a625e732ef78b14/webdav3/client.py#L399 it may create an empty file but not write anything. I don't know if that's the reason why I get an empty file.
Hi,
it seems with version 3.14.5, I can't import WebDavException anymore. It worked with 3.14.4.
Anything changed in this area?
Thanks!
Traceback (most recent call last):
File "/Users/jojo/.venvs/discodos/bin/discosync", line 6, in <module>
from discodos.cmd.sync import main
File "/Users/jojo/git/discodos/discodos/cmd/sync.py", line 21, in <module>
from webdav3.client import WebDavException
ImportError: cannot import name 'WebDavException' from 'webdav3.client' (/Users/jojo/.venvs/discodos/lib/python3.7/site-packages/webdav3/client.py)
Is there a possibility to set minimal upload speed and return exception, when speed it too low
Currently, Urn
marks itself as a directory if and only if its href ends with a '/'. However, this may not apply to all webdav implementations. For example, the webdav provider I'm using currently (link, chinese website) returns <d:response><d:href>
never ends with a /
. Instead, it marks an item a directory using <d:response><d:propstat><d:prop><d:getcontenttype>
with text httpd/unix-directory
. See demo response below.
<d:response>
<d:href>/dav/(censored)</d:href>
<d:propstat>
<d:prop>
<d:getlastmodified>Mon, 24 Feb 2020 02:29:02 GMT</d:getlastmodified>
<d:getcontentlength>0</d:getcontentlength>
<d:getetag />
<d:owner>[email protected]</d:owner>
<d:current-user-privilege-set>
<d:privilege>
<d:read />
</d:privilege>
<d:privilege>
<d:write />
</d:privilege>
<d:privilege>
<d:all />
</d:privilege>
<d:privilege>
<d:read_acl />
</d:privilege>
<d:privilege>
<d:write_acl />
</d:privilege>
</d:current-user-privilege-set>
<d:getcontenttype>httpd/unix-directory</d:getcontenttype>
<d:displayname>(censored)</d:displayname>
<d:resourcetype>
<d:collection />
</d:resourcetype>
</d:prop>
<d:status>HTTP/1.1 200 OK</d:status>
</d:propstat>
</d:response>
I'm new to webdav protocol and not familier with its standard implementation. If it is a wrong implementation from the webdav provider, please let me known and I'll contact the site's custom. Many thanks.
Hi,
Problem:
I used this library to upload files into a Nextcloud instance, but was unhappy with the fact, that the "creation-date" and the "last modified-date" was set by Nextcloud to the "current date". In my case I lost a great deal of information.
After some research I found that this is not a problem of the webdav client, but the server and as there is no "generic" way, all servers use slightly different approaches to overcome this problem.
Solution
For Own/NextCloud it's simply an additional header added to the "upload" request ("X-OC-CTime: " plus int unix timestamp for "creation time" and "X-OC-MTime: " plus int unix timestamp for "modification time").
As the "execute_request" function has the ability to "take" additional headers ("headers_ext"), it was quite simple to provide the required "header" via the "upload" function to "execute_request".
Question
Is it possible to extend the API for all (or at least for "upload") to contain the "header_ext" parameter, as this would give users a better flexability?
For my use-case only uploading of files requires this extension, but for other use-cases it might be required in other api calls as well.
Regards
Angus
Directories are not created on any service via mkdir. I managed to create a directory when using upload_dir on yandex, but I got an error and the files did not load.
>>> client.mkdir("fdvfdv") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Program Files (x86)\Python38-32\lib\site-packages\webdav3\client.py", line 69, in _wrapper res = fn(self, *args, **kw) File "C:\Program Files (x86)\Python38-32\lib\site-packages\webdav3\client.py", line 306, in mkdir raise RemoteParentNotFound(directory_urn.path()) webdav3.exceptions.RemoteParentNotFound: Remote parent for: /fdvfdv/ not found
>>> client.execute_request("mkdir", "33") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Program Files (x86)\Python38-32\lib\site-packages\webdav3\client.py", line 233, in execute_request raise ResponseErrorCode(url=self.get_url(path), code=response.status_code, message=response.content) webdav3.exceptions.ResponseErrorCode: Request to https://dav.box.com/dav33 failed with code 403 and message: b'<?xml version="1.0" encoding="utf-8"?>\n<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">\n <s:exception>Sabre_DAV_Exception_Forbidden</s:exception>\n <s:message>Permission denied to create node</s:message>\n</d:error>\n'
I just upgrade to version 0.13 with no code modified.
raise error
ERROR - name 'hostname' is not defined
When I downgrade to 0.12. everything is okay. I am using webdavclient3 in multiprocessing.pool. Is webdavclient3 v0.13 multiprocess safe ?
Python 3.8.0 (tags/v3.8.0:fa919fd, Oct 14 2019, 19:21:23) [MSC v.1916 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from webdav3.client import Client
>>> options = {
... 'webdav_hostname': "https://webdav.yandex.ru",
... 'webdav_login': "my_login",
... 'webdav_password': "my_pass"
... }
>>> client = Client(options)
>>> client.free()
10737418240
>>> client.list()
['33/', '55.txt']
>>> client.upload_sync(remote_path="33/123.txt", local_path="C:\\Users\\USER\\Desktop\\123.txt")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Program Files (x86)\Python38-32\lib\site-packages\webdav3\client.py", line 497, in upload_sync
self.upload(local_path=local_path, remote_path=remote_path)
File "C:\Program Files (x86)\Python38-32\lib\site-packages\webdav3\client.py", line 433, in upload
self.upload_file(local_path=local_path, remote_path=remote_path)
File "C:\Program Files (x86)\Python38-32\lib\site-packages\webdav3\client.py", line 68, in _wrapper
res = fn(self, *args, **kw)
File "C:\Program Files (x86)\Python38-32\lib\site-packages\webdav3\client.py", line 484, in upload_file
raise RemoteParentNotFound(urn.path())
webdav3.exceptions.RemoteParentNotFound: Remote parent for: /33/123.txt not found
>>> client.upload_sync(remote_path="33\\123.txt", local_path="C:\\Users\\USER\\Desktop\\123.txt")
Traceback (most recent call last):
File "C:\Program Files (x86)\Python38-32\lib\site-packages\urllib3\connectionpool.py", line 665, in urlopen
httplib_response = self._make_request(
File "C:\Program Files (x86)\Python38-32\lib\site-packages\urllib3\connectionpool.py", line 376, in _make_request
self._validate_conn(conn)
File "C:\Program Files (x86)\Python38-32\lib\site-packages\urllib3\connectionpool.py", line 994, in _validate_conn
conn.connect()
File "C:\Program Files (x86)\Python38-32\lib\site-packages\urllib3\connection.py", line 386, in connect
self.sock = ssl_wrap_socket(
File "C:\Program Files (x86)\Python38-32\lib\site-packages\urllib3\util\ssl_.py", line 370, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "C:\Program Files (x86)\Python38-32\lib\ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "C:\Program Files (x86)\Python38-32\lib\ssl.py", line 1040, in _create
self.do_handshake()
File "C:\Program Files (x86)\Python38-32\lib\ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ConnectionResetError: [WinError 10054] Удаленный хост принудительно разорвал существующее подключение
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Program Files (x86)\Python38-32\lib\site-packages\requests\adapters.py", line 439, in send
resp = conn.urlopen(
File "C:\Program Files (x86)\Python38-32\lib\site-packages\urllib3\connectionpool.py", line 719, in urlopen
retries = retries.increment(
File "C:\Program Files (x86)\Python38-32\lib\site-packages\urllib3\util\retry.py", line 400, in increment
raise six.reraise(type(error), error, _stacktrace)
File "C:\Program Files (x86)\Python38-32\lib\site-packages\urllib3\packages\six.py", line 734, in reraise
raise value.with_traceback(tb)
File "C:\Program Files (x86)\Python38-32\lib\site-packages\urllib3\connectionpool.py", line 665, in urlopen
httplib_response = self._make_request(
File "C:\Program Files (x86)\Python38-32\lib\site-packages\urllib3\connectionpool.py", line 376, in _make_request
self._validate_conn(conn)
File "C:\Program Files (x86)\Python38-32\lib\site-packages\urllib3\connectionpool.py", line 994, in _validate_conn
conn.connect()
File "C:\Program Files (x86)\Python38-32\lib\site-packages\urllib3\connection.py", line 386, in connect
self.sock = ssl_wrap_socket(
File "C:\Program Files (x86)\Python38-32\lib\site-packages\urllib3\util\ssl_.py", line 370, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "C:\Program Files (x86)\Python38-32\lib\ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "C:\Program Files (x86)\Python38-32\lib\ssl.py", line 1040, in _create
self.do_handshake()
File "C:\Program Files (x86)\Python38-32\lib\ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(10054, 'Удаленный хост принудительно разорвал существующее подключение', None, 10054, None))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Program Files (x86)\Python38-32\lib\site-packages\webdav3\client.py", line 68, in _wrapper
res = fn(self, *args, **kw)
File "C:\Program Files (x86)\Python38-32\lib\site-packages\webdav3\client.py", line 280, in check
response = self.execute_request(action='check', path=urn.quote())
File "C:\Program Files (x86)\Python38-32\lib\site-packages\webdav3\client.py", line 162, in execute_request
response = self.session.request(
File "C:\Program Files (x86)\Python38-32\lib\site-packages\requests\sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "C:\Program Files (x86)\Python38-32\lib\site-packages\requests\sessions.py", line 668, in send
history = [resp for resp in gen] if allow_redirects else []
File "C:\Program Files (x86)\Python38-32\lib\site-packages\requests\sessions.py", line 668, in <listcomp>
history = [resp for resp in gen] if allow_redirects else []
File "C:\Program Files (x86)\Python38-32\lib\site-packages\requests\sessions.py", line 239, in resolve_redirects
resp = self.send(
File "C:\Program Files (x86)\Python38-32\lib\site-packages\requests\sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "C:\Program Files (x86)\Python38-32\lib\site-packages\requests\adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(10054, 'Удаленный хост принудительно разорвал существующее подключение', None, 10054, None))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Program Files (x86)\Python38-32\lib\site-packages\webdav3\client.py", line 497, in upload_sync
self.upload(local_path=local_path, remote_path=remote_path)
File "C:\Program Files (x86)\Python38-32\lib\site-packages\webdav3\client.py", line 433, in upload
self.upload_file(local_path=local_path, remote_path=remote_path)
File "C:\Program Files (x86)\Python38-32\lib\site-packages\webdav3\client.py", line 68, in _wrapper
res = fn(self, *args, **kw)
File "C:\Program Files (x86)\Python38-32\lib\site-packages\webdav3\client.py", line 483, in upload_file
if not self.check(urn.parent()):
File "C:\Program Files (x86)\Python38-32\lib\site-packages\webdav3\client.py", line 70, in _wrapper
raise NoConnection(self.webdav.hostname)
webdav3.exceptions.NoConnection: Not connection with https://webdav.yandex.ru
>>> client.upload_sync(remote_path="123.txt", local_path="C:\\Users\\USER\\Desktop\\123.txt")
Traceback (most recent call last):
File "C:\Program Files (x86)\Python38-32\lib\site-packages\urllib3\connectionpool.py", line 665, in urlopen
httplib_response = self._make_request(
File "C:\Program Files (x86)\Python38-32\lib\site-packages\urllib3\connectionpool.py", line 376, in _make_request
self._validate_conn(conn)
File "C:\Program Files (x86)\Python38-32\lib\site-packages\urllib3\connectionpool.py", line 994, in _validate_conn
conn.connect()
File "C:\Program Files (x86)\Python38-32\lib\site-packages\urllib3\connection.py", line 386, in connect
self.sock = ssl_wrap_socket(
File "C:\Program Files (x86)\Python38-32\lib\site-packages\urllib3\util\ssl_.py", line 370, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "C:\Program Files (x86)\Python38-32\lib\ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "C:\Program Files (x86)\Python38-32\lib\ssl.py", line 1040, in _create
self.do_handshake()
File "C:\Program Files (x86)\Python38-32\lib\ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ConnectionResetError: [WinError 10054] Удаленный хост принудительно разорвал существующее подключение
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Program Files (x86)\Python38-32\lib\site-packages\requests\adapters.py", line 439, in send
resp = conn.urlopen(
File "C:\Program Files (x86)\Python38-32\lib\site-packages\urllib3\connectionpool.py", line 719, in urlopen
retries = retries.increment(
File "C:\Program Files (x86)\Python38-32\lib\site-packages\urllib3\util\retry.py", line 400, in increment
raise six.reraise(type(error), error, _stacktrace)
File "C:\Program Files (x86)\Python38-32\lib\site-packages\urllib3\packages\six.py", line 734, in reraise
raise value.with_traceback(tb)
File "C:\Program Files (x86)\Python38-32\lib\site-packages\urllib3\connectionpool.py", line 665, in urlopen
httplib_response = self._make_request(
File "C:\Program Files (x86)\Python38-32\lib\site-packages\urllib3\connectionpool.py", line 376, in _make_request
self._validate_conn(conn)
File "C:\Program Files (x86)\Python38-32\lib\site-packages\urllib3\connectionpool.py", line 994, in _validate_conn
conn.connect()
File "C:\Program Files (x86)\Python38-32\lib\site-packages\urllib3\connection.py", line 386, in connect
self.sock = ssl_wrap_socket(
File "C:\Program Files (x86)\Python38-32\lib\site-packages\urllib3\util\ssl_.py", line 370, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "C:\Program Files (x86)\Python38-32\lib\ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "C:\Program Files (x86)\Python38-32\lib\ssl.py", line 1040, in _create
self.do_handshake()
File "C:\Program Files (x86)\Python38-32\lib\ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(10054, 'Удаленный хост принудительно разорвал существующее подключение', None, 10054, None))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Program Files (x86)\Python38-32\lib\site-packages\webdav3\client.py", line 68, in _wrapper
res = fn(self, *args, **kw)
File "C:\Program Files (x86)\Python38-32\lib\site-packages\webdav3\client.py", line 280, in check
response = self.execute_request(action='check', path=urn.quote())
File "C:\Program Files (x86)\Python38-32\lib\site-packages\webdav3\client.py", line 162, in execute_request
response = self.session.request(
File "C:\Program Files (x86)\Python38-32\lib\site-packages\requests\sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "C:\Program Files (x86)\Python38-32\lib\site-packages\requests\sessions.py", line 668, in send
history = [resp for resp in gen] if allow_redirects else []
File "C:\Program Files (x86)\Python38-32\lib\site-packages\requests\sessions.py", line 668, in <listcomp>
history = [resp for resp in gen] if allow_redirects else []
File "C:\Program Files (x86)\Python38-32\lib\site-packages\requests\sessions.py", line 239, in resolve_redirects
resp = self.send(
File "C:\Program Files (x86)\Python38-32\lib\site-packages\requests\sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "C:\Program Files (x86)\Python38-32\lib\site-packages\requests\adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(10054, 'Удаленный хост принудительно разорвал существующее подключение', None, 10054, None))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Program Files (x86)\Python38-32\lib\site-packages\webdav3\client.py", line 497, in upload_sync
self.upload(local_path=local_path, remote_path=remote_path)
File "C:\Program Files (x86)\Python38-32\lib\site-packages\webdav3\client.py", line 433, in upload
self.upload_file(local_path=local_path, remote_path=remote_path)
File "C:\Program Files (x86)\Python38-32\lib\site-packages\webdav3\client.py", line 68, in _wrapper
res = fn(self, *args, **kw)
File "C:\Program Files (x86)\Python38-32\lib\site-packages\webdav3\client.py", line 483, in upload_file
if not self.check(urn.parent()):
File "C:\Program Files (x86)\Python38-32\lib\site-packages\webdav3\client.py", line 70, in _wrapper
raise NoConnection(self.webdav.hostname)
webdav3.exceptions.NoConnection: Not connection with https://webdav.yandex.ru
When files is:
a
-b
-c
list('a') will return ['a', 'b', 'c']
When files is:
a
-a
-b
-c
list('a') will return ['a', 'b', 'c', 'a']
It makes me confused. i expect that it don't return the parent dir name.
Thanks for everyone's work on this project, it's very helpful.
Simple test case:
from webdav3.exceptions import *
from webdav3.client import Client
server = ""
login_name = ""
password = ""
root = "/"
_webdav_client = Client(options={
"webdav_hostname": server,
"webdav_login": login_name,
"webdav_password": password,
"proxy_hostname": "localhost:8080"
})
# _webdav_client.session.proxies = 'localhost:8080' <<-- This also doesn't work
_webdav_client.verify = False
root_dir = _webdav_client.list(str(root), get_info=True)
print(root_dir)
# There is a successful response -
[{'created': None, 'name': None, 'size': '', 'modified': 'Sun, 24 Jan 2021 20:24:17 GMT', 'etag': '"600dd771d443e"', 'isdir': True, 'path': '/somestuff'
....}}
Rerun the test with the environment variables that the Requests library uses. This should fail because the proxy does not exist. It does fail.
import os
from webdav3.exceptions import *
from webdav3.client import Client
server = ""
login_name = ""
password = ""
root = "/"
os.environ["HTTP_PROXY"] = "localhost:8080"
os.environ["HTTPS_PROXY"] = "localhost:8080"
_webdav_client = Client(options={
"webdav_hostname": server,
"webdav_login": login_name,
"webdav_password": password,
"proxy_hostname": "localhost:8080"
})
_webdav_client.verify = False
root_dir = _webdav_client.list(str(root), get_info=True)
print(root_dir)
# An exception result -
...
raise NoConnection(self.webdav.hostname)
webdav3.exceptions.NoConnection: No connection with https://myserver
To test the success case start a proxy -
$ mitmproxy
Rerun the two tests above. You would expect to see traffic through the proxy in both cases but there won't be in the first test, there will be in the second.
It looks to me like the Client class is ignoring the proxy options as is the WebDAVSettings class..
I also don't see a way for the requests session to include a proxy option,
webdav-client-python-3/webdav3/client.py
Lines 206 to 218 in 4fcfcbc
self.session.proxies.update(proxy_hostname)
The problem is here:
webdav-client-python-3/webdav3/client.py
Line 711 in 4fcfcbc
Steps to reproduce
from webdav3.client import Client
options = { 'webdav_hostname': HOSTNAME, 'webdav_login': LOGIN, 'webdav_password': PASSWORD, 'webdav_root': DAVROOT, 'disable_check': True }
client = Client(options)
client.download(remote_path=TESTFILE,local_path="./test.txt")
Error
Traceback (most recent call last):
File "//webdav-test.py", line 13, in
client.download(remote_path=TESTFILE,local_path="./test.txt")
File "/usr/local/lib/python3.9/site-packages/webdav3/client.py", line 397, in download
self.download_file(local_path=local_path, remote_path=remote_path, progress=progress,
File "/usr/local/lib/python3.9/site-packages/webdav3/client.py", line 67, in _wrapper
res = fn(self, *args, **kw)
File "/usr/local/lib/python3.9/site-packages/webdav3/client.py", line 458, in download_file
total = int(response.headers['content-length'])
File "/usr/local/lib/python3.9/site-packages/requests/structures.py", line 54, in getitem
return self._store[key.lower()][1]
KeyError: 'content-length'
Current workaround
Force downgrade to 3.14.5: pip3 install webdavclient3==3.14.5
When we calls client.push
, the client compare the local time and remote time.
Sometimes, the remote server's timezone is not the same with local timezone. It maybe cause some misunderstood.
webdav-client-python-3/webdav3/client.py
Line 706 in 96aebf2
Maybe we can introduce a new variable, specify the remote timezone params.
options = {
'webdav_hostname': "https://webdav.server.ru",
'webdav_login': "login",
'webdav_password': "password",
'remote_timezone': 'Asia/Shanghai', # new variable
}
client = Client(options)
In the self.is_local_more_recent
function, we use the timezone param:
webdav-client-python-3/webdav3/client.py
Line 748 in 96aebf2
try:
remote_info = self.info(remote_path)
remote_last_mod_date = remote_info['modified']
remote_last_mod_date = dateutil_parser.parse(remote_last_mod_date)
# change the timezone
if self.remote_timezone:
remote_last_mod_date = remote_last_mod_date.astimezone(tz.gettz(self.remote_timezone))
remote_last_mod_date_unix_ts = int(remote_last_mod_date.timestamp())
except Exception:
pass
We had an issue due to a strict mod_security that suddenly started to drop all requests. Changing the user-agent seems to fix the issue.
Current webdav-client-python-3 implementation support custom headers for all commands with the headers_ext parameter, however this is not used by the main methods as list, info and so on. From our point of view, the better would be to set headers_ext at client connection.
Similar to #67 : I want to be able to set the retries used in the low level libraries (=0 is a hassle to work around in my own code)
Hi again,
If I do something like this
client = Client(options)
all_files = client.list()
and then iterate through it, the ordering seems random. Is there a way to have the list method return it in a specific order (timestamp, filename, etc.) or is this not implemented (or even out of scope of your module)?
thanks!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.