--exclude about fpart HOT 18 CLOSED

OK, I see.

fpart exclusions, when used through fpsync, are relative to the src path (because fpsync changed its current working directory to the src dir), so you should use the following :

sudo "PATH=$PATH" /usr/local/bin/fpsync -n $_thread_count -v -o "-a --stats --numeric-ids" -O '-X ./cache' pwd/src/ pwd/trg/

Remember you can build fpart with the --enable-debug option to get feedback about included/excluded files and directories.

from fpart.

Hi David,

Yes, it should be :

$ mkdir /tmp/{src,dst}                                                                                    
$ touch /tmp/src/{foo,bar,baz}                                                                                 
$ fpsync -o '-lptgoD -v --numeric-ids --exclude=bar'  /tmp/src /tmp/dst                                        
$ ls /tmp/dst                                                                                                  
baz foo

from fpart.

Yeah, just tried that but it still syncs that folder. I think it is worth to mention that I'm trying to exclude a folder and not a file.

sudo "PATH=$PATH" /usr/local/bin/fpsync -o '-lptgoD -v --numeric-ids --exclude=cache' \
`pwd`/source/ `pwd`/destination/

It still copies over the cache folder which I'm trying to exclude.

from fpart.

Right, this is because fpsync calls fpart which, by default, generates a list of files to synchronize. Thus, excluding a directory at rsync level cannot work because it will always create a parent directory even if it is excluded :

$ mkdir /tmp/{src,dst}
$ mkdir /tmp/src/{foo,bar,baz}
$ touch /tmp/src/{foo,bar,baz}/file
$ fpsync -o '-lptgoD -v --numeric-ids --exclude=bar'  /tmp/src /tmp/dst
$ ls /tmp/dst/bar
file

Here, rsync created the 'bar' directory because it was asked to synchronize bar/file.

You have to exclude the directory earlier by using fpart options :

$ fpsync -O '-x bar' /tmp/src /tmp/dst
$ ls /tmp/dst
baz foo

That way, bar/file will not appear at all in the (internal) file list to synchronize.

from fpart.

@martymac thanks for your assistance on this. Will it accept path in the -x arg ? As it's now it's excluding everything from /tmp/src with a name bar
e.g.
fpsync -O '-x /bar' /tmp/src /tmp/dst

Thanks !

from fpart.

This is the intended behaviour. Unfortunately, there is currently no way to exclude a specific path from fpart as -x only acts on file or directory names (not paths, but that could be an improvement).

Coming back to my first answer, rsync accepts relative paths in its --exclude option, so you may (this is to be tested) be able to exclude all files from that directory with something like : --exclude=bar/*

Note that fpart will crawl that directory and add its files to the produced partitions anyway.

Best regards,
Ganael.

from fpart.

Hello David,

I've implemented ability to specify paths with options '-y', '-Y', '-x' and '-X', that should solve your problem. An exclusion will be considered to be a path when it contains at leat a '/' character. You can mix file names and paths and inclusion/exclusion rules remain the same as before.

Hope this helps,

Best regards,

Ganael.

from fpart.

Hello Ganael,

Thanks for spending time on this ! Much appreciated.

All the best !
David

from fpart.

Hey @martymac
Downloaded and installed the v1.1.0

I just tried:
sudo "PATH=$PATH" /usr/local/bin/fpsync -n $_thread_count -v -o "-a --stats --numeric-ids" -O '-X /cache' /tmp/src/ /tmp/dest/

and nothing really changed. It's still copying over the entire cache folder.

Thanks !

from fpart.

Hi David,

The fix is not in 1.1.0, but in git only (master branch) ; it will be integrated in the next release. You can rebuild fpart from source to get it now if necessary.

Best regards,

Ganael.

from fpart.

I tried to use the master and compiled it but it still copies over the directory.

Used:

sudo "PATH=$PATH" /usr/local/bin/fpsync -n $_thread_count -v -o "-a --stats --numeric-ids" -O '-X /cache' /tmp/src/ /tmp/dest/

$ fpsync
fpsync v1.2.0 - Sync directories in parallel using fpart
Copyright (c) 2014-2019 Ganael LAPLANCHE <[email protected]>

from fpart.

Hi David,

Excluded path must patch the root path of the crawled filesystem. In your case, you should exclude the following path : '/tmp/src/cache'. Can you try that ?

from fpart.

Hey Ganael,

Nope, tried it and it still copies it over.

sudo "PATH=$PATH" /usr/local/bin/fpsync -n $_thread_count -v -o "-a --stats --numeric-ids" -O '-X '`pwd`'/src/cache' `pwd`/src/ `pwd`/trg/

from fpart.

Hi,

are you sure fpsync uses the latest version of fpart you compiled and not a previous version of the binary you installed before ?

from fpart.

Hey,

Yes, looks like it.

$ sudo /usr/local/bin/fpsync -v
fpsync v1.2.0 - Sync directories in parallel using fpart
Copyright (c) 2014-2019 Ganael LAPLANCHE <[email protected]>
WWW: http://contribs.martymac.org
Usage: /usr/local/bin/fpsync [OPTIONS...] src_dir/ dst_url/
Usage: /usr/local/bin/fpsync [-r jobname] [OPTIONS...]

from fpart.

Hello,

I mean, check that the fpsync script itself uses the latest version of fpart. That latest version should be in your $PATH; it is determined that way by the script :

FPART_BIN="$(which fpart)"

Best regards,

Ganael.

from fpart.

Yes it is.

fpart v1.2.0
Copyright (c) 2011-2019 Ganael LAPLANCHE <[email protected]>
WWW: http://contribs.martymac.org
Build options: debug=no, fts=system

from fpart.

Hey Ganael,

This did the trick:

'-X ./cache'

Thanks so much for your assistance !

from fpart.