Comments (13)
The FtpProgress
class contains many of the necessary properties etc. to handle the "Overall Progress" for mutil file operations.
Taking DownloadDirectory as an example, what needs to be done to provide progress?
from fluentftp.
Let's say the directory has 5 files, all of them nice and big.
You now implement a progress callback.
If you show just the "progress" it wil go from 0->n five times.
But the "FileIndex" property will increase 1, 2, 3, 4, 5.
So you if you would know the individual file sizes, you could calculate that "overall progress" in percent.
from fluentftp.
The
FtpProgress
class contains many of the necessary properties etc. to handle the "Overall Progress" for mutil file operations.Taking DownloadDirectory as an example, what needs to be done to provide progress?
When we provide the progress to function with FullProgress
it should in loop fire the IProgress every single time when something happended to a file, for example we checked the file, and we like it, we skip it, and after skipping we add += 1, to file count. When we do so, update the progress imidiately. Which should give us effect of file checking.
from fluentftp.
Whatever is done, it should all be stored in FtpProgress object. So the user can access the information and do with it what he likes.
from fluentftp.
Whatever is done, it should all be stored in FtpProgress object. So the user can access the information and do with it what he likes.
Yes, would be just perfect either provide specific class that will collect all events no matter what, or implement functions that will fire all of the time when something happened, not only when file was downloaded.
from fluentftp.
So not just bytes, files processed, but also "Downloading", "Checking", etc. etc. Need an enum of different actions.
from fluentftp.
So not just bytes, files processed, but also "Downloading", "Checking", etc. etc. Need an enum of different actions.
Here is the poor video that I made, it requires more logic but this was just the test.
This video shows on how exactly should it work.
I modified very little the main library to make it work, and it works by firing every single time when it checks or skips the file, and totally satisfies me. Only thing to change in here is the showing the amount of bytes to check, the Percentage Progress that checks the file and something more, sorry for not being clear enough.
from fluentftp.
I watched the video and it looks a lot cleaner now. What have you changed?
from fluentftp.
Author
In that video it fully showed the progress of downloading, it started from zero and always counted files it checked/downloaded before was ONLY firing the Progress when it was downloading in this video:
from fluentftp.
I watched the video and it looks a lot cleaner now. What have you changed?
Well, it will require to apply this in all possible cases of downloading but here is the little snippet:
I copied this from the DownloadFileInternal.cs
because that's the only way I saw to fire the progress.
if (progress != null) {
ReportProgress(progress, knownFileSize - restartPos, 0, 0, TimeSpan.Zero, localPath, remotePath, metaProgress);
}
After inside of the DownloadFile.cs
I paste the snippet on top to function DownloadFileToFileAsync
because it was getting called from the AsyncFtpClass.DownloadDirectory
Full code of DownloadFile.cs:
using System;
using System.IO;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using FluentFTP.Streams;
using FluentFTP.Helpers;
using FluentFTP.Exceptions;
using FluentFTP.Client.Modules;
using System.Threading;
using System.Threading.Tasks;
namespace FluentFTP {
public partial class AsyncFtpClient {
/// <summary>
/// Downloads the specified file onto the local file system asynchronously.
/// High-level API that takes care of various edge cases internally.
/// Supports very large files since it downloads data in chunks.
/// </summary>
/// <param name="localPath">The full or relative path to the file on the local file system</param>
/// <param name="remotePath">The full or relative path to the file on the server</param>
/// <param name="existsMode">Overwrite if you want the local file to be overwritten if it already exists. Append will also create a new file if it doesn't exists</param>
/// <param name="verifyOptions">Sets if checksum verification is required for a successful download and what to do if it fails verification (See Remarks)</param>
/// <param name="progress">Provide an implementation of IProgress to track download progress.</param>
/// <param name="token">The token that can be used to cancel the entire process</param>
/// <returns>FtpStatus flag indicating if the file was downloaded, skipped or failed to transfer.</returns>
/// <remarks>
/// If verification is enabled (All options other than <see cref="FtpVerify.None"/>) the hash will be checked against the server. If the server does not support
/// any hash algorithm, then verification is ignored. If only <see cref="FtpVerify.OnlyChecksum"/> is set then the return of this method depends on both a successful
/// upload & verification. Additionally, if any verify option is set and a retry is attempted then overwrite will automatically be set to true for subsequent attempts.
/// </remarks>
public async Task<FtpStatus> DownloadFile(string localPath, string remotePath, FtpLocalExists existsMode = FtpLocalExists.Resume, FtpVerify verifyOptions = FtpVerify.None, IProgress<FtpProgress> progress = null, CancellationToken token = default(CancellationToken)) {
// verify args
if (localPath.IsBlank()) {
throw new ArgumentException("Required parameter is null or blank.", nameof(localPath));
}
if (remotePath.IsBlank()) {
throw new ArgumentException("Required parameter is null or blank.", nameof(remotePath));
}
return await DownloadFileToFileAsync(localPath, remotePath, existsMode, verifyOptions, progress, token, new FtpProgress(1, 0));
}
/// <summary>
/// Download a remote file to a local file
/// </summary>
protected async Task<FtpStatus> DownloadFileToFileAsync(string localPath, string remotePath, FtpLocalExists existsMode, FtpVerify verifyOptions, IProgress<FtpProgress> progress, CancellationToken token, FtpProgress metaProgress) {
// verify args
if (localPath.IsBlank()) {
throw new ArgumentException("Required parameter is null or blank.", nameof(localPath));
}
if (remotePath.IsBlank()) {
throw new ArgumentException("Required parameter is null or blank.", nameof(remotePath));
}
// skip downloading if the localPath is a folder
if (LocalPaths.IsLocalFolderPath(localPath)) {
throw new ArgumentException("Local path must specify a file path and not a folder path.", nameof(localPath));
}
remotePath = remotePath.GetFtpPath();
LogFunction(nameof(DownloadFile), new object[] { localPath, remotePath, existsMode, verifyOptions });
bool isAppend = false;
// skip downloading if the local file exists
long knownFileSize = 0;
long restartPos = 0;
#if NETSTANDARD || NET5_0_OR_GREATER
if (existsMode == FtpLocalExists.Resume && await Task.Run(() => File.Exists(localPath), token)) {
knownFileSize = (await GetFileSize(remotePath, -1, token));
restartPos = await FtpFileStream.GetFileSizeAsync(localPath, false, token);
if (knownFileSize.Equals(restartPos)) {
#else
if (existsMode == FtpLocalExists.Resume && File.Exists(localPath)) {
knownFileSize = (await GetFileSize(remotePath, -1, token));
restartPos = FtpFileStream.GetFileSize(localPath, false);
if (knownFileSize.Equals(restartPos)) {
#endif
LogWithPrefix(FtpTraceLevel.Info, "Skipping file because Resume is enabled and file is fully downloaded (Remote: " + remotePath + ", Local: " + localPath + ")");
if (progress != null) { // here is the poor implementation, we just reporting progress every single time when we skip, which is what I need.
ReportProgress(progress, knownFileSize - restartPos, 0, 0, TimeSpan.Zero, localPath, remotePath, metaProgress);
}
return FtpStatus.Skipped;
}
else {
isAppend = true;
}
}
#if NETSTANDARD || NET5_0_OR_GREATER
else if (existsMode == FtpLocalExists.Skip && await Task.Run(() => File.Exists(localPath), token)) {
#else
else if (existsMode == FtpLocalExists.Skip && File.Exists(localPath)) {
#endif
LogWithPrefix(FtpTraceLevel.Info, "Skipping file because Skip is enabled and file already exists locally (Remote: " + remotePath + ", Local: " + localPath + ")");
return FtpStatus.Skipped;
}
try {
// create the folders
var dirPath = Path.GetDirectoryName(localPath);
#if NETSTANDARD || NET5_0_OR_GREATER
if (!string.IsNullOrWhiteSpace(dirPath) && !await Task.Run(() => Directory.Exists(dirPath), token)) {
#else
if (!string.IsNullOrWhiteSpace(dirPath) && !Directory.Exists(dirPath)) {
#endif
Directory.CreateDirectory(dirPath);
}
}
catch (Exception ex1) {
// catch errors creating directory
throw new FtpException("Error while creating directories. See InnerException for more info.", ex1);
}
// if not appending then fetch remote file size since mode is determined by that
/*if (knownFileSize == 0 && !isAppend) {
knownFileSize = GetFileSize(remotePath);
}*/
bool downloadSuccess;
var verified = true;
var attemptsLeft = verifyOptions.HasFlag(FtpVerify.Retry) ? Config.RetryAttempts : 1;
do {
// download the file from the server to a file stream or memory stream
downloadSuccess = await DownloadFileInternalAsync(localPath, remotePath, null, restartPos, progress, token, metaProgress, knownFileSize, isAppend, 0);
attemptsLeft--;
if (!downloadSuccess) {
LogWithPrefix(FtpTraceLevel.Info, "Failed to download file.");
if (attemptsLeft > 0)
LogWithPrefix(FtpTraceLevel.Info, "Retrying to download file.");
}
// if verification is needed
if (downloadSuccess && verifyOptions != FtpVerify.None) {
verified = await VerifyTransferAsync(localPath, remotePath, token);
LogWithPrefix(FtpTraceLevel.Info, "File Verification: " + (verified ? "PASS" : "FAIL"));
if (!verified && attemptsLeft > 0) {
LogWithPrefix(FtpTraceLevel.Verbose, "Retrying due to failed verification." + (existsMode == FtpLocalExists.Resume ? " Overwrite will occur." : "") + " " + attemptsLeft + " attempts remaining");
// Force overwrite if a retry is required
existsMode = FtpLocalExists.Overwrite;
}
}
} while ((!downloadSuccess || !verified) && attemptsLeft > 0);
if (downloadSuccess && !verified && verifyOptions.HasFlag(FtpVerify.Delete)) {
File.Delete(localPath);
}
if (downloadSuccess && !verified && verifyOptions.HasFlag(FtpVerify.Throw)) {
throw new FtpException("Downloaded file checksum value does not match remote file");
}
return downloadSuccess && verified ? FtpStatus.Success : FtpStatus.Failed;
}
}
}
from fluentftp.
So in summary, I want to just see every single time the Progress even if it was skipped.
from fluentftp.
Well, your PR #1453 at least fixes the missing updates of the multi-file progress (filecount etc.) in cases where files are skipped.
It is a start towards more detailed reporting that would give you more than just numbers, but also tell you more about "what" is currently happening.
In my own apps using FluentFTP, I have pursued this also - for me, a solution was to use progress and to monitor the detailed log messages which were telling me what the transfer was doing. A risky business in case log message texts would change (already happened to me a number of times).
from fluentftp.
I'm totally satisfied with this now, I think that's it 😄
See PR's of fixes in: #1453, #1454
from fluentftp.
Related Issues (20)
- FtpCommandException: Code: 553 Message: Could not create file.
- Enhancement: Extend the built-in verification of all major file transfer methods with size comparison. HOT 5
- Support handling retries on Authentication Exceptions HOT 1
- Connection process hangs when attempting AutoConnect HOT 14
- Error in DownloadDirectory with progress parameter HOT 3
- Download with Progress throws exception HOT 9
- Download doesn't proceed when deployed to Azure Container HOT 16
- Process hang with GnuTLS on Linux/ARM64 HOT 9
- Upload within Parallel.ForEach() HOT 11
- Expection FluentFTP.Exceptions.FtpCommandException: Code: 550 Message: Command not recognized or allowed when use GetListing method HOT 2
- AuthenticateAsClientAsync doesn't respect ReadTimeout in a certain scenario HOT 4
- FtpException thrown downloading file with 49.0.2 works in 49.0.1 HOT 4
- Problems uploading a file into a ftp server with TLS HOT 7
- AsyncFtpClient - The operation has timed out HOT 29
- PRT and EPRT don't respect the configured proxy HOT 6
- GetObjectInfo fails for file names with spaces HOT 6
- Error on FTP UPDATFILE after sucessfull connect HOT 1
- Override Read(Span), ReadAsync(Memory), Write(ROS), WriteAsync(ROM) in streams HOT 3
- Issue with AS400 change - AfterConnect actions cannot be undone HOT 21
- Should FTP work be done with temp files? HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fluentftp.