Giter VIP home page Giter VIP logo

Comments (13)

FanDjango avatar FanDjango commented on June 15, 2024

The FtpProgress class contains many of the necessary properties etc. to handle the "Overall Progress" for mutil file operations.

Taking DownloadDirectory as an example, what needs to be done to provide progress?

from fluentftp.

FanDjango avatar FanDjango commented on June 15, 2024

Let's say the directory has 5 files, all of them nice and big.

You now implement a progress callback.

If you show just the "progress" it wil go from 0->n five times.
But the "FileIndex" property will increase 1, 2, 3, 4, 5.
So you if you would know the individual file sizes, you could calculate that "overall progress" in percent.

from fluentftp.

J0nathan550 avatar J0nathan550 commented on June 15, 2024

The FtpProgress class contains many of the necessary properties etc. to handle the "Overall Progress" for mutil file operations.

Taking DownloadDirectory as an example, what needs to be done to provide progress?

When we provide the progress to function with FullProgress it should in loop fire the IProgress every single time when something happended to a file, for example we checked the file, and we like it, we skip it, and after skipping we add += 1, to file count. When we do so, update the progress imidiately. Which should give us effect of file checking.

from fluentftp.

FanDjango avatar FanDjango commented on June 15, 2024

Whatever is done, it should all be stored in FtpProgress object. So the user can access the information and do with it what he likes.

from fluentftp.

J0nathan550 avatar J0nathan550 commented on June 15, 2024

Whatever is done, it should all be stored in FtpProgress object. So the user can access the information and do with it what he likes.

Yes, would be just perfect either provide specific class that will collect all events no matter what, or implement functions that will fire all of the time when something happened, not only when file was downloaded.

from fluentftp.

FanDjango avatar FanDjango commented on June 15, 2024

So not just bytes, files processed, but also "Downloading", "Checking", etc. etc. Need an enum of different actions.

from fluentftp.

J0nathan550 avatar J0nathan550 commented on June 15, 2024

So not just bytes, files processed, but also "Downloading", "Checking", etc. etc. Need an enum of different actions.

Here is the poor video that I made, it requires more logic but this was just the test.
This video shows on how exactly should it work.
I modified very little the main library to make it work, and it works by firing every single time when it checks or skips the file, and totally satisfies me. Only thing to change in here is the showing the amount of bytes to check, the Percentage Progress that checks the file and something more, sorry for not being clear enough.

https://youtu.be/edFIjycNvXg

from fluentftp.

FanDjango avatar FanDjango commented on June 15, 2024

I watched the video and it looks a lot cleaner now. What have you changed?

from fluentftp.

J0nathan550 avatar J0nathan550 commented on June 15, 2024

Author

In that video it fully showed the progress of downloading, it started from zero and always counted files it checked/downloaded before was ONLY firing the Progress when it was downloading in this video:

https://youtu.be/1JpgldcjjEA

from fluentftp.

J0nathan550 avatar J0nathan550 commented on June 15, 2024

I watched the video and it looks a lot cleaner now. What have you changed?

Well, it will require to apply this in all possible cases of downloading but here is the little snippet:

I copied this from the DownloadFileInternal.cs because that's the only way I saw to fire the progress.

	if (progress != null) {
		ReportProgress(progress, knownFileSize - restartPos, 0, 0, TimeSpan.Zero, localPath, remotePath, metaProgress);
	}

After inside of the DownloadFile.cs I paste the snippet on top to function DownloadFileToFileAsync because it was getting called from the AsyncFtpClass.DownloadDirectory

Full code of DownloadFile.cs:

using System;
using System.IO;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using FluentFTP.Streams;
using FluentFTP.Helpers;
using FluentFTP.Exceptions;
using FluentFTP.Client.Modules;
using System.Threading;
using System.Threading.Tasks;

namespace FluentFTP {
	public partial class AsyncFtpClient {

		/// <summary>
		/// Downloads the specified file onto the local file system asynchronously.
		/// High-level API that takes care of various edge cases internally.
		/// Supports very large files since it downloads data in chunks.
		/// </summary>
		/// <param name="localPath">The full or relative path to the file on the local file system</param>
		/// <param name="remotePath">The full or relative path to the file on the server</param>
		/// <param name="existsMode">Overwrite if you want the local file to be overwritten if it already exists. Append will also create a new file if it doesn't exists</param>
		/// <param name="verifyOptions">Sets if checksum verification is required for a successful download and what to do if it fails verification (See Remarks)</param>
		/// <param name="progress">Provide an implementation of IProgress to track download progress.</param>
		/// <param name="token">The token that can be used to cancel the entire process</param>
		/// <returns>FtpStatus flag indicating if the file was downloaded, skipped or failed to transfer.</returns>
		/// <remarks>
		/// If verification is enabled (All options other than <see cref="FtpVerify.None"/>) the hash will be checked against the server.  If the server does not support
		/// any hash algorithm, then verification is ignored.  If only <see cref="FtpVerify.OnlyChecksum"/> is set then the return of this method depends on both a successful 
		/// upload &amp; verification.  Additionally, if any verify option is set and a retry is attempted then overwrite will automatically be set to true for subsequent attempts.
		/// </remarks>
		public async Task<FtpStatus> DownloadFile(string localPath, string remotePath, FtpLocalExists existsMode = FtpLocalExists.Resume, FtpVerify verifyOptions = FtpVerify.None, IProgress<FtpProgress> progress = null, CancellationToken token = default(CancellationToken)) {
			// verify args
			if (localPath.IsBlank()) {
				throw new ArgumentException("Required parameter is null or blank.", nameof(localPath));
			}

			if (remotePath.IsBlank()) {
				throw new ArgumentException("Required parameter is null or blank.", nameof(remotePath));
			}

			return await DownloadFileToFileAsync(localPath, remotePath, existsMode, verifyOptions, progress, token, new FtpProgress(1, 0));
		}

		/// <summary>
		/// Download a remote file to a local file
		/// </summary>
		protected async Task<FtpStatus> DownloadFileToFileAsync(string localPath, string remotePath, FtpLocalExists existsMode, FtpVerify verifyOptions, IProgress<FtpProgress> progress, CancellationToken token, FtpProgress metaProgress) {

			// verify args
			if (localPath.IsBlank()) {
				throw new ArgumentException("Required parameter is null or blank.", nameof(localPath));
			}

			if (remotePath.IsBlank()) {
				throw new ArgumentException("Required parameter is null or blank.", nameof(remotePath));
			}

			// skip downloading if the localPath is a folder
			if (LocalPaths.IsLocalFolderPath(localPath)) {
				throw new ArgumentException("Local path must specify a file path and not a folder path.", nameof(localPath));
			}

			remotePath = remotePath.GetFtpPath();

			LogFunction(nameof(DownloadFile), new object[] { localPath, remotePath, existsMode, verifyOptions });


			bool isAppend = false;

			// skip downloading if the local file exists
			long knownFileSize = 0;
			long restartPos = 0;
#if NETSTANDARD || NET5_0_OR_GREATER
			if (existsMode == FtpLocalExists.Resume && await Task.Run(() => File.Exists(localPath), token)) {
				knownFileSize = (await GetFileSize(remotePath, -1, token));
				restartPos = await FtpFileStream.GetFileSizeAsync(localPath, false, token);
				if (knownFileSize.Equals(restartPos)) {
#else
			if (existsMode == FtpLocalExists.Resume && File.Exists(localPath)) {
				knownFileSize = (await GetFileSize(remotePath, -1, token));
				restartPos = FtpFileStream.GetFileSize(localPath, false);
				if (knownFileSize.Equals(restartPos)) {
#endif
					LogWithPrefix(FtpTraceLevel.Info, "Skipping file because Resume is enabled and file is fully downloaded (Remote: " + remotePath + ", Local: " + localPath + ")");
					if (progress != null) { // here is the poor implementation, we just reporting progress every single time when we skip, which is what I need. 
						ReportProgress(progress, knownFileSize - restartPos, 0, 0, TimeSpan.Zero, localPath, remotePath, metaProgress);
					}

					return FtpStatus.Skipped;
				}
				else {
					isAppend = true;
				}
			}
#if NETSTANDARD || NET5_0_OR_GREATER
			else if (existsMode == FtpLocalExists.Skip && await Task.Run(() => File.Exists(localPath), token)) {
#else
			else if (existsMode == FtpLocalExists.Skip && File.Exists(localPath)) {
#endif
				LogWithPrefix(FtpTraceLevel.Info, "Skipping file because Skip is enabled and file already exists locally (Remote: " + remotePath + ", Local: " + localPath + ")");
				return FtpStatus.Skipped;
			}

			try {
				// create the folders
				var dirPath = Path.GetDirectoryName(localPath);
#if NETSTANDARD || NET5_0_OR_GREATER
				if (!string.IsNullOrWhiteSpace(dirPath) && !await Task.Run(() => Directory.Exists(dirPath), token)) {
#else
				if (!string.IsNullOrWhiteSpace(dirPath) && !Directory.Exists(dirPath)) {
#endif
					Directory.CreateDirectory(dirPath);
				}
			}
			catch (Exception ex1) {
				// catch errors creating directory
				throw new FtpException("Error while creating directories. See InnerException for more info.", ex1);
			}

			// if not appending then fetch remote file size since mode is determined by that
			/*if (knownFileSize == 0 && !isAppend) {
				knownFileSize = GetFileSize(remotePath);
			}*/

			bool downloadSuccess;
			var verified = true;
			var attemptsLeft = verifyOptions.HasFlag(FtpVerify.Retry) ? Config.RetryAttempts : 1;
			do {

				// download the file from the server to a file stream or memory stream
				downloadSuccess = await DownloadFileInternalAsync(localPath, remotePath, null, restartPos, progress, token, metaProgress, knownFileSize, isAppend, 0);
				attemptsLeft--;

				if (!downloadSuccess) {
					LogWithPrefix(FtpTraceLevel.Info, "Failed to download file.");

					if (attemptsLeft > 0)
						LogWithPrefix(FtpTraceLevel.Info, "Retrying to download file.");
				}

				// if verification is needed
				if (downloadSuccess && verifyOptions != FtpVerify.None) {
					verified = await VerifyTransferAsync(localPath, remotePath, token);
					LogWithPrefix(FtpTraceLevel.Info, "File Verification: " + (verified ? "PASS" : "FAIL"));
					if (!verified && attemptsLeft > 0) {
						LogWithPrefix(FtpTraceLevel.Verbose, "Retrying due to failed verification." + (existsMode == FtpLocalExists.Resume ? "  Overwrite will occur." : "") + "  " + attemptsLeft + " attempts remaining");
						// Force overwrite if a retry is required
						existsMode = FtpLocalExists.Overwrite;
					}
				}
			} while ((!downloadSuccess || !verified) && attemptsLeft > 0);

			if (downloadSuccess && !verified && verifyOptions.HasFlag(FtpVerify.Delete)) {
				File.Delete(localPath);
			}

			if (downloadSuccess && !verified && verifyOptions.HasFlag(FtpVerify.Throw)) {
				throw new FtpException("Downloaded file checksum value does not match remote file");
			}

			return downloadSuccess && verified ? FtpStatus.Success : FtpStatus.Failed;
		}

	}
}

from fluentftp.

J0nathan550 avatar J0nathan550 commented on June 15, 2024

So in summary, I want to just see every single time the Progress even if it was skipped.

from fluentftp.

FanDjango avatar FanDjango commented on June 15, 2024

Well, your PR #1453 at least fixes the missing updates of the multi-file progress (filecount etc.) in cases where files are skipped.

It is a start towards more detailed reporting that would give you more than just numbers, but also tell you more about "what" is currently happening.

In my own apps using FluentFTP, I have pursued this also - for me, a solution was to use progress and to monitor the detailed log messages which were telling me what the transfer was doing. A risky business in case log message texts would change (already happened to me a number of times).

from fluentftp.

J0nathan550 avatar J0nathan550 commented on June 15, 2024

I'm totally satisfied with this now, I think that's it 😄
See PR's of fixes in: #1453, #1454

from fluentftp.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.