• add debug option to print ALL errors from server
• add debug option to retry on 400
• describe workaround in docs

The backend configuration should run a browser for user if the configured password is an empty string. This will prepare the backend for future SMS authentication announced by mailru.

Currently chunking backend requires rclone user to understand both C-like printing format and regular expressions. Should rather use a simple format like *.rclone-chunk.###

Rename mailru configuration flags:
username -> user, password -> pass -- for consistency with other backends (Mega, WebDAV, SFTP)
fastput_* -> speedup_* -- since official Disk-O client names the "put by hash" method as "speed up"

See commit d9a61fe

Name "debug" does not correctly convey meaning of the option, "quirks" is the best.
Additionally, parsing of the param string is scattered around backend code. It should be gathered in a single place.

2019/09/07 00:56:22 ERROR : Angels & Demons (2009)/Angels.and.Demons.2009.Theatrical.Cut.2160p.UHD.BluRay.REMUX.HDR.HEVC.Atmos-EPSiLON.mkv: Failed to copy: multipart upload failed to read source: file already closed

When copying from google drive to box (1GB Split) once it gets ready to do it's split it gets this error.

Chunker internally implements Update via temporary Put, thus two files with different but UTF-normally equal names will still be treated as different files. This results in the SyncUTFNorm test failing (as described here https://github.com/ivandeex/rclone/blob/master/fstest/test_all/config.yaml#L39).

However, we may try a mechanism similar to the behaviour triggered by the optimizeFirstChunk flag (see code here https://github.com/ivandeex/rclone/blob/master/backend/chunker/chunker.go#L448) to overcome this limitation.

When dowloading a large file, the backend periodically requests new download servers from mailru dispatcher, e.g. when a chunk download fails. The current implementation in the serverList.take procedure (https://github.com/ivandeex/rclone/blob/master/backend/mailru/mailru.go#L2037) may result in sticking with an overloaded server. This issue is to improve this situation and make the the backend leave a failing server earlier. In particular, we should request a new download server whenever

• need new server (the current chunk is done)
• the old server lock is is expired (even if the server is till in the active list)

See source https://github.com/ivandeex/rclone/tree/mailru
See help https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#writing-documentation

Write documentation for:

• mailru filesystem backend https://github.com/ivandeex/rclone/tree/mailru
• chunker filesystem backend https://github.com/ivandeex/rclone/tree/chunker
• describe mailru hashing algorithm https://github.com/ivandeex/rclone/blob/mailru/backend/mailru/mrhash/mrhash.go
• describe use case of the --vfs-case-insentive option d9a61fe

Mailru calculates hashes only for complete files

If rclone could signal to mailru backend that a file is temporary, the backend could skip creation of an extra copy. Cannot change this.

The "minimum size option" is an overkill. Remove it and have a hardcoded minimum of 20 to 512 bytes (a typical internet packet size).

Defaults for other parameters: speedup_max_mem (32M) and speedup_max_disk (3G) -- somewhat arbitrary defaults.

Maybe, "speedup_enable" option can be removed too. As a signal to disable speedup, users could set speedup disk limit to zero ro set speedup file pattern to empty string.

1. The most appropriate default chunk size is 2GB
2. The "wdmrc" name of metadata type is vague. Should use something like "simplejson".
3. The default minimum chunk number should be "0"
4. We should not check for maximum chunk number by default (i.e. default max = -1), as it may cause unexpected behaviour for an average user

See https://github.com/ivandeex/rclone/blob/mailru/backend/mailru/mrhash/mrhash.go

If an operation returns error 403 (access denied) and a non-empty password is configured, then refresh token using configued password. If this does not help, return "access denied" to the upprt later.

I'm trying to fix 403 error with mail.ru on mac, but there are no binaries for mac in your project, only Linux and win. Can you add mac binary for rclone mail.ru?

See https://github.com/ivandeex/rclone/tree/chunker

Empty file patterns should disable speedup completely,
"*" in file patterns should enable speedup unconditionally.

Such programs as borgbackup and restic use mkdir as a way to atomically lock repository
They expect that if a directory already exists, mkdir should fail.
Note that mkdir -p linux command is not required to fail in this case.

What is the problem you are having with rclone?

Very cool new feature! Thanks for your help with getting it working.

I get the following error when transferring a large file using the default settings with the chunker backend, with a Box remote repository:
DEBUG : projects/archive/INDIVGraphTheory.tar.gz__tmp.1564856403.001: Cancelling multipart upload: multipart upload failed to read source: EOF

What is your rclone version (output from rclone version)

rclone v1.48.mr02

• os/arch: linux/amd64
• go version: go1.12.6

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Linux CentOS 7, 64 bit

Which cloud storage system are you using? (eg Google Drive)

Box

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync / Chunker_Box_ProjectBackups_RawDataAndDocsDirs: -vv --log-file=logs/chunker_box_projectsrawdataanddocsdirs.log --filter-from FILTERFROM_projectsrawdataanddocsdirs_archiveonly.txt --copy-links --retries 1 --progress

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

Attached. Note that I cancelled the process for this log since it would keep retrying but fail after the last attempt.

chunker_box_projectsrawdataanddocsdirs_2019-08-04T07:48_mod.log