Giter VIP home page Giter VIP logo

Comments (4)

aagrxyz avatar aagrxyz commented on June 15, 2024 2

Ah, I think I found the issue:

The AWS_ENDPOINTS is s3.us-east-005.backblazeb2.com and the target is s3://<bucket_name>@us-east-005/

That fixed the issue.

Can we make the error message in the UI surface the 404 issue?

from longhorn.

mantissahz avatar mantissahz commented on June 15, 2024

Hi @aagrxyz,
Any abnormal logs in the longhorn-manager-xxxx pods about the backup target?
The access permission of the bucket may not be right and you could check this #8299 (comment).
Could you send the support bundle to the mail [email protected]?

from longhorn.

derekbit avatar derekbit commented on June 15, 2024

Ref: https://longhorn.io/docs/1.6.1/snapshots-and-backups/backup-and-restore/set-backup-target/#set-up-aws-s3-backupstore

from longhorn.

aagrxyz avatar aagrxyz commented on June 15, 2024

I am using bacblaze and their S3 compatible API - https://www.backblaze.com/apidocs/introduction-to-the-s3-compatible-api

Here are some logs from the longhorn-manager-xxxx pods:

time="2024-04-29T01:32:56Z" level=info msg="Updated setting backup-target-credential-secret to longhorn-b2" func="manager.(*VolumeManager).CreateOrUpdateSetting" file="setting.go:55"
10.42.0.1 - - [29/Apr/2024:01:32:56 +0000] "PUT /v1/settings/backup-target-credential-secret HTTP/1.1" 200 457 "https://longhorn.in.aagr.xyz/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
time="2024-04-29T01:38:27Z" level=error msg="Failed to list backup volumes from backup target" func="controller.(*BackupTargetController).syncBackupVolume" file="backup_target_controller.go:436" controller=longhorn-backup-target cred=longhorn-b2 error="error listing backup volume names: failed to execute: /var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.6.0/longhorn [/var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.6.0/longhorn backup ls --volume-only s3://aagr-longhorn@us-east-005/], output failed to list objects with param: {\n  Bucket: \"aagr-longhorn\",\n  Delimiter: \"/\",\n  Prefix: \"/\"\n} error: AWS Error:  NotFound Not Found <nil>\n404 \n\n, stderr time=\"2024-04-29T01:38:27Z\" level=error msg=\"Failed to list s3\" func=\"s3.(*BackupStoreDriver).List\" file=\"s3.go:128\" error=\"failed to list objects with param: {\\n  Bucket: \\\"aagr-longhorn\\\",\\n  Delimiter: \\\"/\\\",\\n  Prefix: \\\"/\\\"\\n} error: AWS Error:  NotFound Not Found <nil>\\n404 \\n\" pkg=s3\ntime=\"2024-04-29T01:38:27Z\" level=error msg=\"failed to list objects with param: {\\n  Bucket: \\\"aagr-longhorn\\\",\\n  Delimiter: \\\"/\\\",\\n  Prefix: \\\"/\\\"\\n} error: AWS Error:  NotFound Not Found <nil>\\n404 \\n\" func=main.ResponseLogAndError file=\"main.go:47\"\n: exit status 1" interval=5m0s node=n100 url="s3://aagr-longhorn@us-east-005/"
time="2024-04-29T01:38:28Z" level=error msg="Failed to list backup backing images from backup target" func="controller.(*BackupTargetController).syncBackupBackingImage" file="backup_target_controller.go:507" controller=longhorn-backup-target cred=longhorn-b2 error="failed to list backup backing image names: failed to execute: /var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.6.0/longhorn [/var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.6.0/longhorn backup ls-backing-image s3://aagr-longhorn@us-east-005/], output failed to list objects with param: {\n  Bucket: \"aagr-longhorn\",\n  Delimiter: \"/\",\n  Prefix: \"/\"\n} error: AWS Error:  NotFound Not Found <nil>\n404 \n\n, stderr time=\"2024-04-29T01:38:28Z\" level=error msg=\"Failed to list s3\" func=\"s3.(*BackupStoreDriver).List\" file=\"s3.go:128\" error=\"failed to list objects with param: {\\n  Bucket: \\\"aagr-longhorn\\\",\\n  Delimiter: \\\"/\\\",\\n  Prefix: \\\"/\\\"\\n} error: AWS Error:  NotFound Not Found <nil>\\n404 \\n\" pkg=s3\ntime=\"2024-04-29T01:38:28Z\" level=error msg=\"failed to list objects with param: {\\n  Bucket: \\\"aagr-longhorn\\\",\\n  Delimiter: \\\"/\\\",\\n  Prefix: \\\"/\\\"\\n} error: AWS Error:  NotFound Not Found <nil>\\n404 \\n\" func=main.ResponseLogAndError file=\"main.go:47\"\n: exit status 1" interval=5m0s node=n100 url="s3://aagr-longhorn@us-east-005/"
E0429 01:38:28.616156       1 backup_target_controller.go:201] longhorn-backup-target: failed to sync longhorn-system/default: failed to list system backups in s3://aagr-longhorn@us-east-005/: error listing system backup in s3://aagr-longhorn@us-east-005/: failed to execute: /var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.6.0/longhorn [/var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.6.0/longhorn system-backup list s3://aagr-longhorn@us-east-005/], output , stderr time="2024-04-29T01:38:28Z" level=error msg="Failed to list s3" func="s3.(*BackupStoreDriver).List" file="s3.go:128" error="failed to list objects with param: {\n  Bucket: \"aagr-longhorn\",\n  Delimiter: \"/\",\n  Prefix: \"/\"\n} error: AWS Error:  NotFound Not Found <nil>\n404 \n" pkg=s3
time="2024-04-29T01:38:28Z" level=fatal msg="Failed to run list system backup command" func=cmd.SystemBackupCmd.SystemBackupListCmd.func4 file="system_backup.go:72" error="failed to list objects with param: {\n  Bucket: \"aagr-longhorn\",\n  Delimiter: \"/\",\n  Prefix: \"/\"\n} error: AWS Error:  NotFound Not Found <nil>\n404 \n"
: exit status 1

To set up permissions, I have created a application key in backblaze

from longhorn.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.