By default, configure OSDs on all devices except the system disk about rook HOT 9 CLOSED

Design for the formatting of devices is as follows. More options will come in the future as discussed in #102.

Rookd flags
The rookd command line arguments relevant to working with devices include:

• --data-dir
  Default: /var/lib/rook
  The directory where all rook config is stored
• --data-devices
  Default: None. Filestore will be initialized in the data dir
  A comma-separated list of device names such as "sda,sdb,sdc". Note that the device names could change when the machine is rebooted and devices are removed.
  If set to "all", all of the devices detected will be used by rook, as long as they meet formatting criteria. See below.
• --force-format
  Default: false
  When true, all rules for whether to format a volume are ignored. The only exception is that the system volume cannot be formatted.

Formatting criteria
In order for Rook to utilize devices, they must be formatted such that bluestore can use them. The default behavior is that devices will only be formatted if rook detects that they are clean. If there is any indication there could be other data on the device, rook will skip working with the device.

Criteria for determining if a device should be formatted by rook includes:

• No file system
• No partitions
• No disk UUID or label

Exceptions to the criteria:

• If the force-format flag is passed, the devices will be formatted
• OSD partitions from an old cluster will be re-formatted automatically. This is detected where the configuration is not found in the data dir (/var/lib/rook), but the partitions exist with known osd labels. Thus, loss of the data dir assumes loss of the data.

Bluestore Partitions
For context, there are three bluestore partitions:

• RocksDB metadata db
• RocksDB write-ahead-log
• Data: The raw data storage

Ideally the metadata and write-ahead-log would be stored on SSD or NVME, while the data partition is typically on an HDD. The scheme for selecting devices on which to place the partitions for each OSD is TBD. Currently all three partitions are placed on the same device.

from rook.

closing in favor of #102

from rook.

@travisn Can you please tell where to find the rookd utility ?

from rook.

@travisn Can you please tell where to find the rookd utility ?

@JayjeetAtGithub Wow this is an old issue, rookd hasn't existed for a long time. That was for a non-K8s solution, but Rook is only focused on K8s now.

from rook.

Thanks for response @travisn . I am actually looking for ways to tell rook to format any previously used blockdevice that is currently dirty and use it again as OSD. In short, I am looking how to get the functionality of --force-format in the current version of rook. Any pointers would be really helpful. Thanks.

from rook.

@JayjeetAtGithub You need to clean an OSD from a previous cluster? See the cleanup guide

from rook.

I see. So, currently rook doesn't support detecting disks previously used as OSD and cleaning it by itself right?

from rook.

In the v1.4 release soon, rook will have an option during uninstall to clean the disks automatically.

from rook.

Wow. That would be really helpful. Thanks a lot.

from rook.