democratic-csi
implements the csi
(container storage interface) spec
providing storage for various container orchestration systems (ie: Kubernetes).
The current focus is providing storage via iscsi/nfs from zfs-based storage
systems, predominantly FreeNAS / TrueNAS
and ZoL
on Ubuntu
.
The current drivers implement the depth and breadth of the csi
spec, so you
have access to resizing, snapshots, clones, etc functionality.
democratic-csi
is 2 things:
- several implementations of
csi
driversfreenas-nfs
(manages zfs datasets to share over nfs)freenas-iscsi
(manages zfs zvols to share over iscsi)freenas-smb
(manages zfs datasets to share over smb)zfs-generic-nfs
(works with any ZoL installation...ie: Ubuntu)zfs-generic-iscsi
(works with any ZoL installation...ie: Ubuntu)zfs-local-ephemeral-inline
(provisions node-local zfs datasets)nfs-client
(crudely provisions storage using a shared nfs share/directory for all volumes)
- framework for developing
csi
drivers
If you have any interest in providing a csi
driver, simply open an issue to
discuss. The project provides an extensive framework to build from making it
relatively easy to implement new drivers.
Predominantly 3 things are needed:
- node prep
- server prep
- deploy the driver into the cluster (
helm
chart provided with samplevalues.yaml
)
You should install/configure the requirements for both nfs and iscsi.
Follow the instructions here: https://netapp-trident.readthedocs.io/en/stable-v20.04/kubernetes/operations/tasks/worker.html
If you are running Kubernetes with rancher/rke please see the following:
If using with Windows based machines you may need to enable guest access (even if you are connecting with credentiasl)
Set-ItemProperty HKLM:\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters AllowInsecureGuestAuth -Value 1
Restart-Service LanmanWorkstation -Force
This driver
provisions node-local ephemeral storage on a per-pod basis. Each
node should have an identically named zfs pool created and avaialble to the
driver
. Note, this is NOT the same thing as using the docker zfs storage
driver (although the same pool could be used). No other requirements are
necessary.
- https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/20190122-csi-inline-volumes.md
- https://kubernetes-csi.github.io/docs/ephemeral-local-volumes.html
Server preparation depends slightly on which driver
you are using.
Ensure the following services are configurged and running:
- ssh (if you use a password for authentication make sure it is allowed)
- ensure
zsh
,bash
, orsh
is set as the root shell,csh
gives false errors due to quoting - nfs
- iscsi
- smb
Ensure ssh and zfs is installed on the server and that you have installed
targetcli
.
yum install targetcli -y
apt-get -y install targetcli-fb
helm repo add democratic-csi https://democratic-csi.github.io/charts/
helm repo update
# helm v2
helm search democratic-csi/
# helm v3
helm search repo democratic-csi/
# copy proper values file from https://github.com/democratic-csi/charts/tree/master/stable/democratic-csi/examples
# edit as appropriate
# examples are from helm v2, alter as appropriate for v3
# add --create-namespace for helm v3
helm upgrade \
--install \
--values freenas-iscsi.yaml \
--namespace democratic-csi \
zfs-iscsi democratic-csi/democratic-csi
helm upgrade \
--install \
--values freenas-nfs.yaml \
--namespace democratic-csi \
zfs-nfs democratic-csi/democratic-csi
Some distrobutions, such as minikube
and microk8s
uses a non-standard kubelet path.
In such cases it is necessary to provide a new kubelet host path, microk8s example below:
microk8s helm upgrade \
--install \
--values freenas-nfs.yaml \
--set node.kubeletHostPath="/var/snap/microk8s/common/var/lib/kubelet" \
--namespace democratic-csi \
zfs-nfs democratic-csi/democratic-csi
You may install multiple deployments of each/any driver. It requires the following:
- Use a new helm release name for each deployment
- Make sure you have a unique
csiDriver.name
in the values file - Use unqiue names for your storage classes (per cluster)
- Use a unique parent dataset (ie: don't try to use the same parent across deployments or clusters)
Install beta (v1.17+) CRDs (once per cluster):
kubectl apply -f snapshot.storage.k8s.io_volumesnapshotclasses.yaml
kubectl apply -f snapshot.storage.k8s.io_volumesnapshotcontents.yaml
kubectl apply -f snapshot.storage.k8s.io_volumesnapshots.yaml
Install snapshot controller (once per cluster):
# replace namespace references to your liking
kubectl apply -f rbac-snapshot-controller.yaml
kubectl apply -f setup-snapshot-controller.yaml
Install democratic-csi
as usual with volumeSnapshotClasses
defined as appropriate.