Giter VIP home page Giter VIP logo

velero-plugin's Introduction

Getting Started

This README explains how to install and configure the DigitalOcean Block Storage provider plugin for Velero. The plugin is designed to create filesystem snapshots of Block Storage backed PersistentVolumes that are used in a Kubernetes cluster running on DigitalOcean.

Prerequisites

Credentials setup

  1. To use this plugin with Velero to create persistent volume snapshots, you will need a DigitalOcean API token. Create one before proceeding with the rest of these steps.

  2. For the object storage Velero component, generate a Spaces access key and secret key

Velero installation

  1. Complete the Prerequisites and Credentials setup steps mentioned above.

  2. Clone this repository. cd into the examples directory and edit the cloud-credentials file. The file will look like this:

    [default]
    aws_access_key_id=<AWS_ACCESS_KEY_ID>
    aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
    

Edit the <AWS_ACCESS_KEY_ID> and <AWS_SECRET_ACCESS_KEY> placeholders to use your DigitalOcean Spaces keys. Be sure to remove the < and > characters.

  1. Still in the examples directory, edit the 01-velero-secret.patch.yaml file. It should look like this:

    ---
    apiVersion: v1
    kind: Secret
    stringData:
    digitalocean_token: <DIGITALOCEAN_API_TOKEN>
    type: Opaque
    
    • Change the entire <DIGITALOCEAN_API_TOKEN> portion to use your DigitalOcean personal API token. The line should look something like digitalocean_token: 18a0d730c0e0....
  2. Now you're ready to install velero, configure the snapshot storage location, and work with backups. Ensure that you edit each of the following settings to match your Spaces configuration befor running the velero install command:

    • --bucket velero-backups - Ensure you change the velero-backups value to match the name of your Space.
    • --backup-location-config s3Url=https://nyc3.digitaloceanspaces.com,region=nyc3 - Change the URL and region to match your Space's settings. Specifically, edit the nyc3 portion in both to match the region where your Space is hosted. Use one of nyc3, sfo2, sgp1, or fra1 depending on your region.
  3. Now run the install command:

    velero install \
        --provider velero.io/aws \
        --bucket velero-backups \
        --plugins velero/velero-plugin-for-aws:v1.3.0,digitalocean/velero-plugin:v1.1.0 \
        --backup-location-config s3Url=https://nyc3.digitaloceanspaces.com,region=nyc3 \
        --use-volume-snapshots=false \
        --secret-file=./cloud-credentials
    

Snapshot configuration

  1. Enable the digitalocean/velero-plugin:v1.1.0 snapshot provider. This command will configure Velero to use the plugin for persistent volume snapshots.

    velero snapshot-location create default --provider digitalocean.com/velero
    
  2. Patch the cloud-credentials Kubernetes Secret object that the velero install command installed in the cluster. This command will add your DigitalOcean API token to the cloud-credentials object so that this plugin can use the DigitalOcean API:

    kubectl patch secret cloud-credentials -p "$(cat 01-velero-secret.patch.yaml)" --namespace velero
    
  3. Patch the velero Kubernetes Deployment to expose your API token to the Velero pod(s). Velero needs this change in order to authenticate to the DigitalOcean API when manipulating snapshots:

    kubectl patch deployment velero -p "$(cat 02-velero-deployment.patch.yaml)" --namespace velero
    

Backup and restore example

  1. Install the Nginx examples/nginx-example.yaml Deployment into your cluster. The example uses a persistent volume for Nginx logs. It also creates a LoadBalancer with a public IP address:

    kubectl apply -f examples/nginx-example.yaml
    
  2. Ensure that your Nginx Deployment is running and there is a Service with an EXTERNAL-IP (kubectl get service --namespace nginx-example). Browse the IP a few times to write some log entries to the persistent volume. Then create a backup with Velero:

    velero backup create nginx-backup --selector app=nginx --snapshot-volumes=true
    velero backup describe nginx-backup --details
    
  3. The various backup files will be in your Spaces bucket. A snapshot of the persistent volume will be listed in the DigitalOcean control panel under the Images link. Now you can simulate a disaster by deleting the nginx-example namespace.

    kubectl delete namespace nginx-example
    
  4. Once the delete finishes, restore the nginx-backup backup:

    velero restore create --from-backup nginx-backup
    
  5. Check the restored PersistentVolume, Deployment, and Service are back using kubectl:

    kubectl get persistentvolume --namespace nginx-example
    kubectl get service --namespace nginx-example
    kubectl get deployment --namespace nginx-example
    

Build the plugin

make clean
make container IMAGE=digitalocean/velero-plugin:dev

velero-plugin's People

Contributors

edevenport avatar jamonation avatar therealkevinard avatar varshavaradarajan avatar zevisert avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

velero-plugin's Issues

Snapshots of expired backups are not deleted

For some reason it seems that velero doesn't cleanup the digitalocean PVC snapshots of expired backups. Is that expected or a bug with the digitalocean plugin?
I've only noticed because new backups started failing with "too many snapshots for requested volume" and realized that there were 2500+ snapshots.

Restore velero backup into different DigitalOcean account

I am trying to restore a k8s cluster from one DO account into another. At the first attempt it went "fine", except I had put the source DO Token in the velero config. Hence the volumes was restored into the old account, everything else in the target account. Not satisfied with this failure I updated the token and tried again. This time the velero restore just did not complete. After trying again, possibly inducing some duplication errors, i saw some errors on the velero pod itself, indication that it could not read the snapshot:

Storage.GetSnapshot returned error: GET https://api.digitalocean.com/v2/snapshots/654fe732-f99f-11ea-bc4a-0a58ac14807e: 404 

This makes sense, given that snapshots are used for the backup - the target site token would not be able to read them. How can I then backup/restore between accounts?

The second run actually ended, and among some certificate errors, I saw this:

error executing PVAction for persistentvolumes/pvc-0b3f9d26-9fcf-4b0f-940a-e925fb3163a4: rpc error: code = Aborted desc = plugin panicked: runtime error: invalid memory address or nil pointer dereference 

BTW: From my testing, velero on DO seems to suit our needs. Restores across accounts is just for testing, not a requirement. Is this assumption correct, or does DO recommend a better, or more professional, tool?

Velero partially fails to take snapshot 429 rate limit

I'm using Velero to backup my digital ocean Kubernetes cluster for 2 namespaces.
One namespace has only one PVC and I can successfully backup and restore it, the second namespace has 2 pvc and for it I face the following issue:

time="2021-05-03T17:03:30Z" level=error msg="Storage.CreateSnapshot returned error: POST https://api.digitalocean.com/v2/volumes/57cea4da-7eb1-11eb-9cd5-0a58ac14d0a4/snapshots: 429 (request \"af8ed563-d8dd-40fb-a5df-4aa294608722\") failed to create snapshot: this operation is rate-limited - one request per 10m0s" backup=velero/b2waste-cms-prod3 cmd=/plugins/velero-digitalocean logSource="/go/src/github.com/digitalocean/velero-plugin/velero-digitalocean/volume_snapshotter.go:139" pluginName=velero-digitalocean
time="2021-05-03T17:03:30Z" level=info msg="1 errors encountered backup up item" backup=velero/b2waste-cms-prod3 logSource="pkg/backup/backup.go:427" name=b2waste-wordpress-cms-mariadb-0
time="2021-05-03T17:03:30Z" level=error msg="Error backing up item" backup=velero/b2waste-cms-prod3 error="error taking snapshot of volume: rpc error: code = Aborted desc = plugin panicked: runtime error: invalid memory address or nil pointer dereference" logSource="pkg/backup/backup.go:431" name=b2waste-wordpress-cms-mariadb-0

I read online about the API limit for PVC snapshot https://docs.digitalocean.com/products/images/snapshots/ but I'm wondering why I'm hitting this limit at all since these are 2 different PVC / volumes

PVC yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    meta.helm.sh/release-name: b2waste-wordpress-cms
    meta.helm.sh/release-namespace: default
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: dobs.csi.digitalocean.com
  creationTimestamp: "2021-03-06T19:22:48Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app.kubernetes.io/instance: b2waste-wordpress-cms
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: wordpress
    helm.sh/chart: wordpress-10.6.10
  name: b2waste-wordpress-cms
  namespace: default
  resourceVersion: "2985"
  uid: 7a101758-e9e6-4e29-89a0-b6c5b254f463
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: do-block-storage
  volumeMode: Filesystem
  volumeName: pvc-7a101758-e9e6-4e29-89a0-b6c5b254f463
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 10Gi
  phase: Bound
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: dobs.csi.digitalocean.com
  creationTimestamp: "2021-03-06T19:22:48Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app.kubernetes.io/component: primary
    app.kubernetes.io/instance: b2waste-wordpress-cms
    app.kubernetes.io/name: mariadb
  name: data-b2waste-wordpress-cms-mariadb-0
  namespace: default
  resourceVersion: "2990"
  uid: 25e42613-21a3-4878-b178-2613ebe4b165
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 8Gi
  storageClassName: do-block-storage
  volumeMode: Filesystem
  volumeName: pvc-25e42613-21a3-4878-b178-2613ebe4b165
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 8Gi
  phase: Bound

Error installing Velero on Digitalocean - "unable to locate ObjectStore plugin named velero.io/digitalocean"

After adding the credentials, and following the install instructions on the README I was not able to take proper backup. I checked the Velero pod logs and saw this error:

level=error msg="Error getting backup store for this location" backupLocation=default controller=backup-sync error="unable to locate ObjectStore plugin named velero.io/digitalocean" logSource="pkg/controller/backup_sync_controller.go:182"

Any ideas or solutions for this particular error?

Thanks for any help on this!

digital ocean k8s backup alternatives

Is there any other way through we can access Digital ocean spaces wihout creating space access key and secret key and also without digital ocean api token

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.