Comments (4)
Ah, I think I found the issue:
The AWS_ENDPOINTS is s3.us-east-005.backblazeb2.com
and the target is s3://<bucket_name>@us-east-005/
That fixed the issue.
Can we make the error message in the UI surface the 404 issue?
from longhorn.
Hi @aagrxyz,
Any abnormal logs in the longhorn-manager-xxxx
pods about the backup target?
The access permission of the bucket may not be right and you could check this #8299 (comment).
Could you send the support bundle to the mail [email protected]?
from longhorn.
from longhorn.
I am using bacblaze and their S3 compatible API - https://www.backblaze.com/apidocs/introduction-to-the-s3-compatible-api
Here are some logs from the longhorn-manager-xxxx pods:
time="2024-04-29T01:32:56Z" level=info msg="Updated setting backup-target-credential-secret to longhorn-b2" func="manager.(*VolumeManager).CreateOrUpdateSetting" file="setting.go:55"
10.42.0.1 - - [29/Apr/2024:01:32:56 +0000] "PUT /v1/settings/backup-target-credential-secret HTTP/1.1" 200 457 "https://longhorn.in.aagr.xyz/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
time="2024-04-29T01:38:27Z" level=error msg="Failed to list backup volumes from backup target" func="controller.(*BackupTargetController).syncBackupVolume" file="backup_target_controller.go:436" controller=longhorn-backup-target cred=longhorn-b2 error="error listing backup volume names: failed to execute: /var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.6.0/longhorn [/var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.6.0/longhorn backup ls --volume-only s3://aagr-longhorn@us-east-005/], output failed to list objects with param: {\n Bucket: \"aagr-longhorn\",\n Delimiter: \"/\",\n Prefix: \"/\"\n} error: AWS Error: NotFound Not Found <nil>\n404 \n\n, stderr time=\"2024-04-29T01:38:27Z\" level=error msg=\"Failed to list s3\" func=\"s3.(*BackupStoreDriver).List\" file=\"s3.go:128\" error=\"failed to list objects with param: {\\n Bucket: \\\"aagr-longhorn\\\",\\n Delimiter: \\\"/\\\",\\n Prefix: \\\"/\\\"\\n} error: AWS Error: NotFound Not Found <nil>\\n404 \\n\" pkg=s3\ntime=\"2024-04-29T01:38:27Z\" level=error msg=\"failed to list objects with param: {\\n Bucket: \\\"aagr-longhorn\\\",\\n Delimiter: \\\"/\\\",\\n Prefix: \\\"/\\\"\\n} error: AWS Error: NotFound Not Found <nil>\\n404 \\n\" func=main.ResponseLogAndError file=\"main.go:47\"\n: exit status 1" interval=5m0s node=n100 url="s3://aagr-longhorn@us-east-005/"
time="2024-04-29T01:38:28Z" level=error msg="Failed to list backup backing images from backup target" func="controller.(*BackupTargetController).syncBackupBackingImage" file="backup_target_controller.go:507" controller=longhorn-backup-target cred=longhorn-b2 error="failed to list backup backing image names: failed to execute: /var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.6.0/longhorn [/var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.6.0/longhorn backup ls-backing-image s3://aagr-longhorn@us-east-005/], output failed to list objects with param: {\n Bucket: \"aagr-longhorn\",\n Delimiter: \"/\",\n Prefix: \"/\"\n} error: AWS Error: NotFound Not Found <nil>\n404 \n\n, stderr time=\"2024-04-29T01:38:28Z\" level=error msg=\"Failed to list s3\" func=\"s3.(*BackupStoreDriver).List\" file=\"s3.go:128\" error=\"failed to list objects with param: {\\n Bucket: \\\"aagr-longhorn\\\",\\n Delimiter: \\\"/\\\",\\n Prefix: \\\"/\\\"\\n} error: AWS Error: NotFound Not Found <nil>\\n404 \\n\" pkg=s3\ntime=\"2024-04-29T01:38:28Z\" level=error msg=\"failed to list objects with param: {\\n Bucket: \\\"aagr-longhorn\\\",\\n Delimiter: \\\"/\\\",\\n Prefix: \\\"/\\\"\\n} error: AWS Error: NotFound Not Found <nil>\\n404 \\n\" func=main.ResponseLogAndError file=\"main.go:47\"\n: exit status 1" interval=5m0s node=n100 url="s3://aagr-longhorn@us-east-005/"
E0429 01:38:28.616156 1 backup_target_controller.go:201] longhorn-backup-target: failed to sync longhorn-system/default: failed to list system backups in s3://aagr-longhorn@us-east-005/: error listing system backup in s3://aagr-longhorn@us-east-005/: failed to execute: /var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.6.0/longhorn [/var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.6.0/longhorn system-backup list s3://aagr-longhorn@us-east-005/], output , stderr time="2024-04-29T01:38:28Z" level=error msg="Failed to list s3" func="s3.(*BackupStoreDriver).List" file="s3.go:128" error="failed to list objects with param: {\n Bucket: \"aagr-longhorn\",\n Delimiter: \"/\",\n Prefix: \"/\"\n} error: AWS Error: NotFound Not Found <nil>\n404 \n" pkg=s3
time="2024-04-29T01:38:28Z" level=fatal msg="Failed to run list system backup command" func=cmd.SystemBackupCmd.SystemBackupListCmd.func4 file="system_backup.go:72" error="failed to list objects with param: {\n Bucket: \"aagr-longhorn\",\n Delimiter: \"/\",\n Prefix: \"/\"\n} error: AWS Error: NotFound Not Found <nil>\n404 \n"
: exit status 1
To set up permissions, I have created a application key in backblaze
from longhorn.
Related Issues (20)
- [BACKPORT][v1.5.6][BUG] Lost connection to unix:///csi/csi.sock HOT 2
- [BACKPORT][v1.6.2][BUG] Lost connection to unix:///csi/csi.sock HOT 2
- [BUG] Instance manager pod consumes high CPU usage HOT 3
- [BUG] Unable to join new node - Longhorn v1.7.0-dev HOT 7
- [FEATURE] Add New Disks via Config of Longhorn / HELM HOT 3
- [TEST] support Rancher prime version in Jenkins pipeline
- [BACKPORT][v1.5.6][BUG] Backup marked as "completed" cannot be restored, gzip: invalid header
- [UI][IMPROVEMENT] Allow users to request backup volume update HOT 3
- [BUG][V1.5.5] fs trim job fails on RWX volume HOT 9
- [BACKPORT][v1.5.6][CI] Move CI builds to Github Action
- [BACKPORT][v1.6.3][CI] Move CI builds to Github Action HOT 3
- [Bug] DaemonSet longhorn-manager has too much RBAC permission which may leads the whole cluster being hijacked HOT 7
- [BUG] Test case test_support_bundle_should_not_timeout timed out
- [IMPROVEMENT] Read-only volume monitoring check HOT 13
- [TEST] Fix flaky test case test_csi_block_volume_online_expansion HOT 1
- [BACKPORT][v1.5.6][IMPROVEMENT] Investigate performance bottleneck in v1 data path HOT 5
- [BACKPORT][v1.6.2][IMPROVEMENT] Investigate performance bottleneck in v1 data path HOT 4
- [BACKPORT][v1.6.2][BUG] Secret for backup not found
- [FEATURE] Enhance Volume Expansion by Relocating and Rebuilding Replicas on Nodes with Sufficient Space HOT 1
- [BUG] backing image isn't restored after restored a system backup HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from longhorn.