Comments (1)
Hello,
This is a great proposal and it targets a problem we hit very often in FTS/Gfal2 transfers.
The proposed solution needs to be adapted, as Davix cannot resolve the passive host. Instead, it will need to be told this value.
Some theory
In an HTTP-TPC, two parties are involved, named active and passive. You are correct, Davix resolves one of them before the copy operation starts, namely the active party. However, Davix has no way of resolving the passive party and in fact, it shouldn't. It's best to let the active party resolve the transfer location. Davix will simply pass the "passive party" location to the active party, which becomes responsible for doing the host resolution and then the file transfer.
Once the HTTP-TPC starts, the active party will continuously send transfer updates back to the initiating client, in this case Davix. These updates are called HTTP-TPC Performance Markers, and one such field is the RemoteConnections
field (best documentation I know on this topic is here). The RemoteConnections
field contains the IP address of the passive host.
Back to the problem
Knowing the actual hosts involved in the transfer is very valuable. For this reason, I've created the following issue DMC-1358. The change introduces two new getter methods in the Davix::Copy public API:
std::string getTransferSourceHost() const;
std::string getTransferDestinationHost() const;
After the transfer is done, they will return either the hostname or IP value, or empty, if no information is available.
The active party is resolved by Davix, just before initiating the COPY
request. The passive party is extracted by Davix from the HTTP-TPC Performance Markers, via the RemoteConnections
field.
Here is an example of how FTS/Gfal2 will leverage this information:
$ gfal-copy -v https://eospublic.cern.ch:443/eos/opstest/dteam/file.test https://eospps.cern.ch:443/eos/opstest/dteam/file.test
event: [1670855013815] BOTH http_plugin TRANSFER:EXIT https://eospublic.cern.ch:443/eos/opstest/dteam/file.test (p05151113281066.cern.ch) => https://eospps.cern.ch:443/eos/opstest/dteam/file.test ([2001:1458:301:17::100:e])
All Davix changes have been implemented in commit 969b9a5.
Cheers,
Mihai
from davix.
Related Issues (20)
- Unit tests fail when run by more than one user on the same machine HOT 3
- CVE-2022-32221 in bundled curl library HOT 2
- Wrong format for downloaded files for release 0.8.4 HOT 5
- CVE-2023-38545 in bundled curl library HOT 1
- 1 test fails, and several tests hang
- Recursively deleting directories with davix-rm not working HOT 1
- Return value of `DavFile::get` is -1 but no error is thrown HOT 1
- [0.7.6.200.3c48eab] davix-get failing auth with DAVIX_USE_LIBCURL HOT 2
- how to use in wndows? HOT 4
- Cannot build against system libcurl HOT 1
- CTest tests removed HOT 6
- Uploading large files (streaming) to Google S3 fail (issue with multi-part)
- davix version 0.8.0 cannot authenticate using username and password HOT 14
- The role of the different branches is unclear HOT 1
- Update CMake FindPackage Python to allow Python3 on macOS HOT 7
- Missing libneon.a in 0.8.1 HOT 4
- Please add instructions how to run tests to the README
- Copyrighted song lyrics in codebase HOT 2
- Infinite loop in davix slow unit tests when using external libcurl HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from davix.