Trying to setup monitor-only role

$id=fqdn_rand(7)
$mon_key = "AQCXR1tUQK5DHhAAEp/Nfj/tGBWbYoXvHB586w=="   

    ceph::mon { id:
        monitor_secret => $mon_key,
        #mon_addr       => '192.168.0.10', # The host's «public» IP address
    }

generates such ceph.conf

[mon]
mon data = /var/lib/ceph/mon/mon.$id

not

[mon]
mon data = /var/lib/ceph/mon/mon.7

Today we have been testing drive failing/replacement and noticed a couple short-comings in the device.pp manifest:

1. When a disk is replaced, the ceph.conf will change and this results in a service restart of all the osd's in a server. (because of the subscribe => Concat /etc/ceph/ceph.conf in each osd service). These restarts result in a noticeable disruption. Ideally we want only to start the affected service, not all of them!
2. Using the /dev/sdX names for disks isn't ideal, since when a replacement drive is inserted it will get a new name (e.g. today we pulled sdq, then reinserted it and it got sdab). We then need to do one of
   (a) change our host manifests to add osd::device (sdab), but this isn't good since the device will return to sdq after a reboot, or
   (b) reboot the server, to get the device called sdq once again.

Do people have experience already with better practices to prevent these two problems?? Help is much appreciated!

Cheers, Dan
CERN IT

Hello, i am install you plugin, and use your example file,but get error

Hi,

I suggest to get ride of the ceph-authool command and thus to simplify the key creation.

$ ceph -n mon. -k /var/lib/ceph/mon/mon.0/keyring auth get-or-create client.admin mon 'allow *' osd 'allow *' mds allow | tee -a /etc/ceph/keyring

Cheers!

I'm unable to download the image. There is some issue with the SSL Cert. I getting this error messages:

vagrant up mon0
There were warnings and/or errors while loading your Vagrantfile.
Your Vagrantfile was written for an earlier version of Vagrant,
and while Vagrant does the best it can to remain backwards
compatible, there are some cases where things have changed
significantly enough to warrant a message. These messages are
shown below.

Warnings:
* `config.vm.customize` calls are VirtualBox-specific. If you're
using any other provider, you'll have to use config.vm.provider in a
v2 configuration block.

Bringing machine 'mon0' up with 'virtualbox' provider...
[mon0] Box 'wheezy' was not found. Fetching box from specified URL for
the provider 'virtualbox'. Note that if the URL does not have
a box for this provider, you should interrupt Vagrant now and add
the box yourself. Otherwise Vagrant will attempt to download the
full box prior to discovering this error.
Downloading or copying the box...
An error occurred while downloading the remote file. The error
message, if any, is reproduced below. Please fix this error and try
again.

SSL certificate problem: unable to get local issuer certificate
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option

For the moment is the workaround that did work to add the box by hand.

vagrant box add --insecure wheezy http://labs.enovance.com/pub/wheezy.box
Downloading or copying the box...
Extracting box...te: 2590k/s, Estimated time remaining: --:--:--)
Successfully added box 'wheezy' with provider 'virtualbox'!

But the SSL Error has to go, need to clone and want to run this. Great when http is pointing to the https site, but only when the Cert is valid, any idea how to solve?

Greetings 4k3nd0

Installing puppet-ceph as of commit 77c8db9 breaks facter.

francois@puppet:~$ sudo facter -p --debug --trace
value for macaddress is still nil
value for macaddress is still nil
Not an EC2 host
/var/lib/puppet/lib/facter/ceph_osd_bootstrap_key.rb:35: undefined method `each' for nil:NilClass (NoMethodError)
        from /usr/lib/ruby/vendor_ruby/facter/util/loader.rb:81:in `load'
        from /usr/lib/ruby/vendor_ruby/facter/util/loader.rb:81:in `load_file'
        from /usr/lib/ruby/vendor_ruby/facter/util/loader.rb:43:in `load_all'
        from /usr/lib/ruby/vendor_ruby/facter/util/loader.rb:38:in `each'
        from /usr/lib/ruby/vendor_ruby/facter/util/loader.rb:38:in `load_all'
        from /usr/lib/ruby/vendor_ruby/facter/util/loader.rb:35:in `each'
        from /usr/lib/ruby/vendor_ruby/facter/util/loader.rb:35:in `load_all'
        from /usr/lib/ruby/vendor_ruby/facter/util/collection.rb:93:in `load_all'
        from /usr/lib/ruby/vendor_ruby/facter.rb:100:in `to_hash'
        from /usr/lib/ruby/vendor_ruby/facter/application.rb:26:in `run'
        from /usr/bin/facter:71
francois@puppet:~$ $ facter --version
1.6.18
francois@puppet:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 6.0.7 (squeeze)
Release:        6.0.7
Codename:       squeeze
francois@puppet:~$ 

The ceph service isn't enabled so after a reboot ceph will not work anymore.

On adding the following manifest file, it fails for Key['admin'].

node 'ubuntu-vanilla' {

        class { 'apt': update_timeout => '5000' }
        class { 'ceph::apt::ceph' : release => 'dumpling' }

          if !empty($::ceph_admin_key) {
            @@ceph::key { 'admin':
              secret       => $::ceph_admin_key,
              keyring_path => '/etc/ceph/keyring',
            }
          }
        class { 'role_ceph_mon': id => 'ubuntu-vanilla' }
  ceph::osd::device { '/dev/sdb':  journalsize => '200' }
  ceph::osd::device { '/dev/sdc': journalsize => '200'}
  ceph::osd::device { '/dev/sdd': journalsize => '200'}
}

I got following message
message: "Could not retrieve catalog from remote server: Error 400 on SERVER: Another local or imported resource exists with the type and title Ceph::Key[admin] on node ubuntu-vanilla"

After this I made following changes, at-least it allowed adding OSDs.

diff -r ./osd/device.pp /home/dgautam/puppet-ceph/manifests/osd/device.pp
55d54
<   #if $blkid != 'undefined'  and defined( Ceph::Key['admin'] ){
59c58
<       #require => Ceph::Key['admin'],

---
>       require => Ceph::Key['admin'],
diff -r ./osd.pp /home/dgautam/puppet-ceph/manifests/osd.pp
30c30
<   #Package['ceph'] -> Ceph::Key <<| title == 'admin' |>>

---
>   Package['ceph'] -> Ceph::Key <<| title == 'admin' |>>
[root@puppet manifests]$

Being new to puppet/ceph, I am not sure on the correct solution.

The comments in ceph::client say:

# [*keys*]
#   (required) List of pairs of key names & secrets.
#   At least a key named 'admin' should be passed.
#   Example: {
#              'admin' => {
#                secret       => 'secretadmin'
#                keyring_path => '/etc/ceph/ceph.client.admin.keyring'
#              },
#              'client1' => {
#                secret       => 'secretclient1',
#                keyring_path => '/etc/ceph/ceph.client.client1.keyring'
#              }
#            }
#   Note: if path is ommited, the current default from enovance/puppet-ceph
#   will be used: '/var/lib/ceph/tmp/${name}.keyring' which might be unsafe

Yet ceph::key has:

define ceph::key (
  $secret,
  $keyring_path = "/var/lib/ceph/tmp/${name}.keyring",
  $keyname_int,
) {

so keyname_int is required.

Though you could fix the comments, I personally think that keyname_int could default to the $name or $title of the resource. I feel like I'm specifying that value multiple times. Here's my example:

$ceph_keys = {'xyzkey' => {
                                keyring_path => '/etc/ceph/ceph.client.xyzkey.keyring',
                                secret  => hiera('xyzkey::ceph::secret'),
                                keyname_int =>  'xyzkey'
                                }
                             }

Note how 'xyzkey' is repeated several times. I think at least keyname_int could default to the name of the resource.

Looking at: https://github.com/redhat-cip/puppet-ceph/blob/master/manifests/yum/ceph.pp

el6 is hard coded in the manifest, I'll a submit a patch this week if I have enough time!

Regards,

ceph::conf::mon cannot get imported without using puppetdb. This is mentioned nowhere.

Are you planning to release the enovance-ceph module on forge.puppetlabs.com ?
Would be great to have this module there available so it can be installed with the "puppet module" command.

Is there some reason you didn't add an "unless" to the ceph-osd-crush-* and ceph-osd-register-* Exec's in device.pp?? As it is now the auth add and crush set is executed at every puppet instance for each osd device... which in our case is once per 10 minutes at the moment.

If you see nothing wrong with it, I'd like to add something like:

ceph-osd-register...:
unless => 'ceph auth list | grep osd.${osd_id}'

ceph-osd-crush...:
unless => '... to be determined ;) '

Cheers, Dan

The first install of an OSD works. Subsequent installs work, but the OSD fails to see the disk as usable. Possibly some remnant data? I'm still researching the issue. It may be a bug in ceph itself, or perhaps we need to dev/zero the drive?

hello,to unknown reasons to me does not create a keyring for osd here is an example site.pp

node 'ceph-mds1.test' {
    class { 'ceph_mds': id => 1 }

 node 'ceph-mon0.test' {
  if !empty($::ceph_admin_key) {
    @@ceph::key { 'admin':
      secret       => $::ceph_admin_key,
      keyring_path => '/etc/ceph/keyring',
    }
  }
  class { 'role_ceph_mon': id => 0 }
}

node 'ceph-mon1.test' {
  class { 'role_ceph_mon': id => 1 }
}   
node 'tnode01' {

  class { 'role_ceph':
    fsid      => $::fsid,
    auth_type => 'cephx',
  }

  class { 'ceph::osd' :
    public_address  => $ipaddress_eth0,
    cluster_address => $ipaddress_eth0,
  }

  ceph::osd::device { '/dev/sdb': }

}

'ceph-authtool /var/lib/ceph/tmp/keyring.mon.6 --create-keyring --name=mon. --add-key='AQCXR1tUQK5DHhAAEp/Nfj/tGBWbYoXvHB586w==' --cap mon 'allow *'' is not qualified and no path was specified. Please qualify the command or specify a path

With future parser and puppet 4.0 strict variable types are enforced. The integers in the ceph::conf are compared to single quoted values. Due to future parser and puppet 4.0 changes the variable types are strict and must match during expressions.

Variables that need quoting:

• $pool_default_size
• $pool_default_pg_num
• $pool_default_pgp_num

The $conf_group variable in ceph::conf needs to be quoted due to a deprication in puppet and file, user, and group modes.

The error is

err: Failed to apply catalog: Could not find dependency Ceph::Key[admin] for Exec[ceph_osd_create_sdb] at /etc/puppet/environments/eno-ceph/modules/ceph/manifests/osd/device.pp:56

Hello guys,

I do run in some issue with including a devices as a OSD. I want to that the block devices to be encrypted. I unlock this via cryptsetup luksOpen /dev/sdb OSD-1. But the when ceph::osd::devices is creating a gpt on it, it assumes that the new partition can be access via ${name}1, but the block devices is offering it via /dev/mapper/OSD-1p1.

Here the piece of code that have to change:

  exec { "mkfs_${devname}":
    command => "mkfs.xfs -f -d agcount=${::processorcount} -l \
size=1024m -n size=64k ${name}1",
    unless  => "xfs_admin -l ${name}",
    require => [Package['xfsprogs'], Exec["mkpart_${devname}"]],
  }

I guess it would be better to read the printable of parted and then try to find the partition. For the moment I disable the creation of the partition table, there is no need for now.

In general you don't need to create partition tables. You only have to be aware not to run a grub-install on the disk. This indeed would overwrite the Superblock. But are you planning to separate the OSD Journal to extra partition?

Further this devices isn't able to get a UUID via facter. It only display this:

Notice: /Stage[main]/cloud::Osd/Ceph::Osd::Device[/dev/mapper/OSD-0]/Notify[BLKID FACT OSD-0: blkid_uuid_OSD-01]/message: defined 'message' as 'BLKID FACT OSD-0: blkid_uuid_OSD-01'
Notice: BLKID OSD-0: undefined
Notice: /Stage[main]/cloud::Osd/Ceph::Osd::Device[/dev/mapper/OSD-0]/Notify[BLKID OSD-0: undefined]/message: defined 'message' as 'BLKID OSD-0: undefined'

I could a workaround using /sbin/blkid to getting the UUID of the dmcrypt devices. But maybe this issue is related to my VM? I do use Ubuntu 12.04 with ruby 1.8.7 and puppet 3.2.2.

I'll may work on a fork and try to fix this, maybe add in some parameter to handles this?
Like to enable gpt or not and some fine work. For example defining the inode size. For performance reason you should use them with 2048.

so far
4k3nd0

Could you please tag release 1.3.1 on github?