tohuwabohu / puppet-duplicity Goto Github PK

I was unable to use an s3 backend with your module because I got the error:
BackendException: Could not initialize backend: No module named boto

I was using the following hiera configuration:

classes:
  duplicity:
    duply_package_ensure: present
    duply_use_logrotate_module: false
  duplicity::profiles:
    profiles:
      system:
        source: "/"
        target: s3://s3-eu-west-1.amazonaws.com/********/system
        full_if_older_than: 7D
        max_full_backups: 2
        cron_enabled: true
        cron_hour: 4
        cron_minute: 0
        gpg_encryption: false
        include_filelist:
        - "/etc/**"
        duplicity_extra_params:
        - "--s3-use-new-style"
        duply_environment:
        - x

This was on Ubuntu 16.04.2 running in AWS. Installing python-boto solved the issue:
apt-get install python-boto

In order to disable encryption, GPG_KEY needs to be set to 'disabled'. The template sets GPG_KEY_SIGN='disabled'. This does not disable the encryption and makes the GPG test fail instead. Or am I overlooking something?

Hello,

We are using puppet server 5.x and puppet clients >= 4.8.2.

Doing a module installation into a working puppet 5.x server fails with:

$ sudo puppet module install tohuwabohu-duplicity --version 4.9.0 --modulepath modules
Notice: Preparing to install into /etc/puppetlabs/code/environments/development/modules ...
Notice: Downloading from https://forgeapi.puppet.com ...
Error: Could not install module 'tohuwabohu-duplicity' (???)
No version of 'tohuwabohu-duplicity' can satisfy all dependencies
Use puppet module install --ignore-dependencies to install only this module

Installing into a fresh directory works

$ sudo puppet module install tohuwabohu-duplicity --version 4.9.0 --modulepath modules
/tmp/modules
└─┬ tohuwabohu-duplicity (v4.9.0)
├── camptocamp-archive (v0.9.0)
├─┬ puppetlabs-concat (v2.2.1)
│ └── puppetlabs-stdlib (v4.19.0)
└── yo61-logrotate (v1.4.0)

Basically, logrotate and concat are causing the failures.

yo61/logrotate has been deprecated since May 25th 2017.

It has been replaced/adopted by puppet/logrotate ( v2.0.0 )

Stale logrotate requirement is:

{
  "name": "yo61/logrotate",
  "version_requirement": "1.x"
},

Should be?

{
"name": "puppet/logrotate",
"version_requirement": ">= 1.2.0 < 3.0.0"
},

Additionally, puppetlabs-concat current version is v4.0.1

Stale concat requirement is:

{
  "name": "puppetlabs/concat",
  "version_requirement": ">= 1.1.0 < 3.0.0"
},

Should be?

{
  "name": "puppetlabs/concat",
  "version_requirement": ">= 1.1.0 < 5.0.0"
},

Thanks!

Attempting to apply the manifest gives this error:

Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, Logrotate::Rule[duply]: rotate must be an integer at /etc/puppet/modules/logrotate/manifests/rule.pp:306:7  at /etc/puppet/modules/duplicity/manifests/setup.pp:59

The relevant part of duplicity::setup is:

  logrotate::rule { 'duply':
    ensure       => present,
    path         => "${duplicity::duply_log_dir}/*.log",
    rotate       => '5',

It doesn't seem to like the quoted '5', but removing the quotes doesn't fix the issue. This may be a problem with the logrotate module, but I'm not sure.

Hi,

I was wondering if it would be possible to get multiple targets for a single profile? for example:

duplicity::profile {'conf_file':
  full_if_older_than => '2W',
  max_full_backups => 3,
  volsize => 500,
  cron_enabled => true,
  cron_hour => '03',
  cron_minute => '00',
  gpg_encryption => false,
  targets => ["sftp://user1:[email protected]//srv/backups/$hostname", "sftp://user2:[email protected]//srv/backups/$hostname"],
}

which would back objects declared with duplicity::file to the two different remote servers.

Seeing as this is unsupported I tried doing it with two profiles (that have the two different targets) and duplicating the duplicity::file objects, but that breaks ( Duplicate declaration: Exec[restore /etc/] is already declared at [...]; cannot redeclare, the Exec gets declared with the path, not the name of the duplicity::file object). It would also be a lot easier not to have to duplicate every single duplicity::file object for all my servers...

Thanks

I want to restore a directory (recursive) from backup, if it is not existing. Therefore I have defined this in my manifest:

duplicity::file{'some/directory':
    ensure => present,
}

But how am I dealing with the related

file{ 'some/directory':
  ensure => 'directory',
}

entry? Is it not needed at all? What will happen, if it is the first run and there is no backup so far? (Or it is none available).

How is the thinking in this situation? Most likely it is more a documentation lack, than a software issue.

Please push 3.0.2 to the puppet forge to get the couple of minor fixes.

Thanks!

If i set "backup_target_url => "file:///backup/${fqdn}"," the duply conf file also includes TARGET_USER & TARGET_PASS even if "backup_target_username" & "backup_target_password" is not set or empty.

When backup runs a new "@" directory in the current directory is created where the backup is written to. Seems duply trys to use "file://user@/backup/" shema if TARGET_USER & TARGET_PASS exists.

If i delete TARGET_USER & TARGET_PASS from conf file it works as expected.

Warning: Scope(Duplicity::File[/var/lib/jenkins]): Could not look up qualified variable 'duplicity::exec_path'; class duplicity has not been evaluated
Notice: Compiled catalog for vault-77.home.gateway in environment production in 2.02 seconds
Error: Validation of Exec[restore /var/lib/jenkins] failed: 'duply system fetch "var/lib/jenkins" "/var/lib/jenkins"' is not qualified and no path was specified. Please qualify the command or specify a path. at /opt/puppenkiste/environments/production/modules/duplicity/manifests/file.pp:89

Looks like this is an issue with resource ordering: the duplicity::file requires duplicity::params but later it uses duplicity::exec_path which may not be resolved by the time.

When using the S3 backend, we got the following error:

--- Start running command BKP at 05:37:34.976 ---

Duplicity 0.6 series is being deprecated:
See http://www.nongnu.org/duplicity/

Traceback (most recent call last):
  File "/bin/duplicity", line 1509, in <module>
    with_tempdir(main)
  File "/bin/duplicity", line 1503, in with_tempdir
    fn()
  File "/bin/duplicity", line 1336, in main
    action = commandline.ProcessCommandLine(sys.argv[1:])
  File "/usr/lib64/python2.7/site-packages/duplicity/commandline.py", line 1062, in ProcessCommandLine
    backup, local_pathname = set_backend(args[0], args[1])
  File "/usr/lib64/python2.7/site-packages/duplicity/commandline.py", line 955, in set_backend
    globals.backend = backend.get_backend(bend)
  File "/usr/lib64/python2.7/site-packages/duplicity/backend.py", line 163, in get_backend
    return _backends[pu.scheme](pu)
  File "/usr/lib64/python2.7/site-packages/duplicity/backends/_boto_single.py", line 163, in __init__
    self.resetConnection()
  File "/usr/lib64/python2.7/site-packages/duplicity/backends/_boto_single.py", line 185, in resetConnection
    self.conn = get_connection(self.scheme, self.parsed_url, self.storage_uri)
  File "/usr/lib64/python2.7/site-packages/duplicity/backends/_boto_single.py", line 101, in get_connection
    is_secure=(not globals.s3_unencrypted_connection))
  File "/usr/lib/python2.7/site-packages/boto/storage_uri.py", line 117, in connect
    **connection_args)
  File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 191, in __init__
    validate_certs=validate_certs, profile_name=profile_name)
  File "/usr/lib/python2.7/site-packages/boto/connection.py", line 569, in __init__
    host, config, self.provider, self._required_auth_capability())
  File "/usr/lib/python2.7/site-packages/boto/auth.py", line 989, in get_auth_handler
    'Check your credentials' % (len(names), str(names)))
NoAuthHandlerFound: No handler was ready to authenticate. 1 handlers were checked. ['HmacAuthV1Handler'] Check your credentials

It turns out to be related to a duply upgrade, somewhere between 1.9.1 and 1.11.1 per this bug: https://sourceforge.net/p/ftplicity/bugs/90/

My current workaround was to set:

duplicity::duply_package_ensure: 1.9.1-1.el7

in my hiera, which forced puppet to downgrade the package, but I have to assume there is a proper way to fix this? (and that it has already been fixed for everyone else 🍼 .

It appears that we need to set some new "environment variables" (AWS_....)

NOTE: we had an issue which was blocking our epel package upgrades getting into pulp, so we were a bit behind on receiving this upgrade.

~tommy

duplicity::profile attempts to check for setting of the environment variables AWS_ACCCESS_KEY_ID andAWS_SECRET_ACCESS_KEY when using an s3 backend. I have noticed 2 issues with that:

1. I got en error on the check itself, presumably because I haven't set $duply_version but install the standard version from packages.

Error while evaluating a Function Call, 'versioncmp' parameter 'a' expects a String value, got Undef at /etc/puppetlabs/code/environments/development/modules/duplicity/manifests/profile.pp:180:59

2. The check doesn't account for using key-less S3. AWS has a feature where you can assign a role to an EC2 instance and the S3 libraries then take care of all permissions for you. They connect to S3 to retrieve temporary keys that rotate every hour and use them for authentication. No environment variables necessary.

The current code is easy enough to work around, just supply some other environment variable and the check is skipped, but it is not ideal. I am unsure on how to fix it without removing the check completely, which I presume is not something you would want. Maybe an extra $check_aws_keys boolean argument?

When I download the latest module from the forge, it appears that the tarfile contains the exclude.erb with a 600 permission on the file. This causes my puppet runs to fail because the modules are installed as root (via r10k), but puppet runs not as root.

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to parse template duplicity/etc/duply/exclude.erb:
  Filepath: /opt/puppet/lib/ruby/site_ruby/1.9.1/puppet/parser/templatewrapper.rb
  Line: 100
  Detail: Permission denied - /etc/puppetlabs/puppet/environments/staging/modules/duplicity/templates/etc/duply/exclude.erb

ls -la /etc/puppetlabs/puppet/environments/staging/modules/duplicity/templates/etc/duply/exclude.erb
-r--------. 1 root root 402 May 23  2014 /etc/puppetlabs/puppet/environments/staging/modules/duplicity/templates/etc/duply/exclude.erb

tar -ztvf ~/Downloads/tohuwabohu-duplicity-3.1.0.tar.gz tohuwabohu-duplicity-3.1.0/templates/etc/duply/exclude.erb
-rw-------  0 martin martin    402 May 23  2014 tohuwabohu-duplicity-3.1.0/templates/etc/duply/exclude.erb

I've not seen modules do this before. This feels like the module tool should fix these sorts of things, but that's a separate issue.

Hey, how can I have multiple backup destinations? I tried making a new class (class duplicity_whatever inherits duplicity {}) but I don't know how to set the variables, and I can't declare the duplicity class multiple times...

Solution:

templates/conf.erb:

Set TARGET_USERand TARGET_PASS only if the corresponding puppet vars are not undef

In manifests/params.pp:

$backup_target_username = undef
$backup_target_password = undef

could you please release a new version with vorlons changes on puppetforge?

thanks, and merry xmas

My initial attempt to do a backup failed with lots of timed out errors because the target directory was missing. Is it really required to precreate the target directory when using rsync:// protocol? Not sure if this is a limitation in duplicty or duply or if this module should just make sure the target directory exists.

I worked around this by adding pre script hook to use sftp to create the target directory.

As described in https://bugs.launchpad.net/ubuntu/+source/duplicity/+bug/1169463 the python-paramiko package dependency is missing in Ubuntu 12.04 - 16.04 and backups using sftp are throwing an error:

Import of duplicity.backends.sshbackend Failed: No module named paramiko
Import of duplicity.backends.giobackend Failed: No module named gio

I suggest to set the package as dependency for the duply package as long as the bug is not fixed.

We have noticed our /root/.cache directory is excessively large (3+GB) ....

We found this "workaround" out in the Internets, and it seems to clean things up nicely:

duply <BACKUPNAME> cleanup --extra-clean --force

We think it should probably be added to the puppet module? What say you?

The example profile in the README:

duplicity::profile { 'system':
  full_if_older_than => '7D',
  max_full_backups   => 2,
  cron_hour          => '4',
  cron_minute        => '0',
}

gives this error with Puppet 4.3:

Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, Failed to parse template duplicity/etc/duply/conf.erb:
  Filepath: /etc/puppet/modules/duplicity/templates/etc/duply/conf.erb
  Line: 113
  Detail: undefined method `empty?' for 2:Fixnum

The template is testing max_full_backups.empty?, but the value is a Fixnum, which doesn't support .empty?. Quoting the value fixes this problem.

Dear,

First of all I would like to thank you for this nice module.
I'm able to use it without encryption but no sure how to do with encryption.
I don't know how to specify private key in my hiera. Same think for the public key in a the template :-(

Thank you for your assistance.
Best regards.

The command batch used for cron backup is:

cleanup_backup_purgeFull

This batch is expanded to:

cleanup_pre_bkp_post_purgeFull

Each of this is a duply command, and the _ is a "neutral" separator.

Duply defines other separators, such as "+" which means executing the next step only if the previous succeeded.

In our particular use case it is desirable that the backup is aborted if the pre script fails. We currently use pre to create a snapshot of a LVM volme, which is then mounted before backup. If this step fails we currently end up with the backup of the empty mount-point.

Currently this duply command batch is hard-coded.

It would be desirable that could be, optionally, customized when defining the profile.

Hi!

I just started testing this module, and I am getting the following error:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Invalid relationship: Exec[restore /var/log] { require => File[/usr/bin/duply] }, because File[/usr/bin/duply] doesn't seem to be in the catalog

my guess is that this resource is only declared in the case in which:

$duplicity::duply_package_provider == archive

but not in the default case.

Regards.

Puppet error:

Error: Parameter hour failed on Cron[backup-system]: 1,13 is not a valid hour at /etc/puppet/modules/duplicity/manifests/profile.pp:275

The "cron_hour" should have validation that allows for backup runs more than once per day by adjusting cron_hour.

I'm running this on a CentOS 7 and it seems to be working so far. Any particular testing I should be doing before claiming v7 support?

Hi,

I tried using the role with an empty configuration and took this from the readme:

  class { 'duplicity':
    ensure => present,
  }

This resulted in the following error:

Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Class[Duplicity]: has no parameter named 'ensure' (file: /etc/puppetlabs/code/environments/infrastructure/manifests/01_main.pp, line: 17, column: 3) on node ipam.tpac.org.au
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

I see that text appearing in the intial commit - perhaps its never worked?

If thats the case I'm happy to provide a PR to remove that example.

I got the following error...

Error:

Error: empty(): Requires either array, hash or string to work with at /etc/puppet/modules/duplicity/manifests/profile.pp:112 on node app.dev

Puppet Code:

# Duplicity Profile to implement all the types
class site::profile::duplicity (
  $profiles = {},
  $public_keys = {},
  $private_keys = {},
  $files = {},

) {
  validate_hash($profiles)
  validate_hash($public_keys)
  validate_hash($private_keys)
  validate_hash($files)

  # Base Class
  # -- note set any settings in hiera
  include ::duplicity

  create_resources( '::duplicity::profile', $profiles)
  create_resources( '::duplicity::public_key', $public_keys)
  create_resources( '::duplicity::private_key', $private_keys)
  create_resources( '::duplicity::file', $files)

}

HieraData:

site::profile::duplicity::profiles:
  system:
    full_if_older_than: 7D
    max_full_backups: 4
    cron_hour: 1
    cron_minute: 0
    volsize: 100
    include_filelist:
      - /web
      - /etc
      - /opt
    exclude_filelist:
      - /var/log/lastlog
      - '**/tmp'
      - '**/cache'

Puppet Open Source v 3.8.1
├── puppetlabs-stdlib (v4.9.0)

So... It would appear that empty() doesn't like integers?

For what its worth, it would be nice if the duplicity init did this so that I wouldn't have to implement this wrapper at all ;)

~tommy

https://github.com/tohuwabohu/puppet-duplicity/blob/master/manifests/profile.pp#L253

Shouldn't this be "purge-full" instead of "purgeFull". The latter one will lead to duply complaining that it does not know this command.