garethgeorge / backrest Goto Github PK
View Code? Open in Web Editor NEWBackrest is a web UI and orchestrator for restic backup.
License: GNU General Public License v3.0
Backrest is a web UI and orchestrator for restic backup.
License: GNU General Public License v3.0
Hi! Coming from the forums.
I believe the best way to implement this would be to have a "password command" field next to the password field where one could input the command. Similar to restic's own RESTIC_PASSWORD_COMMAND env. Backrest will then run that command everytime it needs the password.
For the docker container, an env variable could be used.
This way multiple repositories could have different commands/env variables.
Hi, I just noticed the plans retention policy is not showing properly in the UI.
I've set the plan to keep the last 7 backups, but the UI is showing none:
Selecting retention by count defaults to 30:
The config.json file shows the correct configuration:
I also tested setting the retention by time, and the same behavior happens, but the config file is correctly saved.
The cron selector UI is a bit unintuitive for the uninitiated. Improve the schedule selection UI / UX. One improvement might be to include human readable text explaining the currently configured policy e.g. similar to https://crontab.guru/
Can a hook test button be implemented?
When we create new hooks, it would be nice to force a test hook trigger to test the configuration.
Scheduling wrt last execution time will prove important for very infrequent backups e.g. monthly or yearly which will not run unless resticui is online at exactly the time the backup is scheduled for.
Hi Gareth,
I've got a warning in a job about at least one source file was not able to be added to the backup. This is the log message I have:
2024-02-11T11:00:00.193Z INFO running task {"task": "backup for plan \"Backup_Job\""}
2024-02-11T11:26:11.911Z DEBUG hook output: ---- payload ----{"content":"Backrest Notification
Task: \"backup for plan \"Backup_Job\"\" at 2024-02-11T11:26:10Z
Error: failed to backup: command \"/data/restic-0.16.4 backup --json --exclude-caches -o sftp.args=-oBatchMode=yes /source/To_Backup --tag plan:Backup_Job --parent 17fa10ea92cb8342ff6bc620ff5da678c01ac778b003a68f801b8c649b3d062d\" failed: incomplete backup
Process STDOUT:
{\"message_type\":\"status\",\"seconds_elapsed\":4,\"percent_done\":0}
{\"message_type\":\"status\",\"seconds_elapsed\":4,\"percent_done\":0}
{\"message_type\":\"status\",\"seconds_elapsed\":4,\"percent_done\":0}
{\"message_type\":\"status\",\"seconds_elapsed\":4,\"percent_done\":0}
{\"message_type\":\"status\",\"seconds_elapsed\":4,\"percent_done\":0}
{\"message_type\":\"status\",\"seconds_elapsed\":4,\"percent_done\":0,\"total_files\":1,\"total_bytes\":288}
{\"message_type\":\"status\",\"seconds_elapsed\":4,\"percent_done\":0,\"total_files\":1,\"total_byt...[17259073 bytes dropped]..._bytes\":46112872238,\"bytes_done\":46112853380,\"error_count\":2}
{\"message_type\":\"summary\",\"files_new\":402,\"files_changed\":818,\"files_unmodified\":134051,\"dirs_new\":6,\"dirs_changed\":807,\"dirs_unmodified\":219415,\"data_blobs\":11842,\"tree_blobs\":579,\"data_added\":6488076064,\"total_files_processed\":135271,\"total_bytes_processed\":46112853380,\"total_duration\":1563.640250311,\"snapshot_id\":\"6fa17b6516676fb8ca86697af0cd2c3de767c66b0fed418068b097dc1b15ba6a\"}
Warning: at least one source file could not be read
"}
2024-02-11T11:26:15.474Z INFO task finished {"task": "collect garbage", "duration": "208.288397ms"}
2024-02-11T11:26:11.911Z INFO Backup complete {"plan": "Backup_Job", "duration": "26m11.121661696s", "summary": {"message_type":"summary","files_new":402,"files_changed":818,"files_unmodified":134051,"dirs_new":6,"dirs_changed":807,"dirs_unmodified":219415,"data_blobs":11842,"tree_blobs":579,"data_added":6488076064,"total_files_processed":135271,"total_bytes_processed":46112853380,"total_duration":1563.640250311,"snapshot_id":"6fa17b6516676fb8ca86697af0cd2c3de767c66b0fed418068b097dc1b15ba6a","percent_done":0,"total_files":0,"files_done":0,"total_bytes":0,"bytes_done":0,"current_files":null}}
How can I know what is the failed file?
Is there any other log available than the one from the docker container?
Improve UI performance with an on-disk caching layer for commonly fetched data -- particularly snapshots
Refresh policy
Caching should be done on a per-repository basis.
Hi Gareth,
for some reason, one of my backups (I have 2), is now scheduled to run twice at the same time:
My backup configuration in the json file is the following:
{
"id": "Backup",
"repo": "Backup_Repo",
"paths": [
"/raid/2Backup"
],
"excludes": [],
"cron": "0 11 * * *",
"retention": {
"maxUnusedLimit": "",
"keepLastN": 7,
"keepHourly": 0,
"keepDaily": 0,
"keepWeekly": 0,
"keepMonthly": 0,
"keepYearly": 0,
"keepWithinDuration": ""
},
"hooks": []
},
Any idea why it happened?
On my laptop, I prefer to trigger backups on-demand rather than using a schedule. It would be great if it were possible to create a plan without a schedule - or offer on-demand backups directly from the repo. The only issue I see with the latter is that I do still want to prune after a backup, and retention polices are currently configured on the plan.
By the way, how do retention polices work with multiple plans? Does the restic forget
command target the plan tag so it only applies to snapshots created by a given plan? Is there a way to see the actual command that was run (or params passed to a restic library) in the operation list?
Cron monitoring services like healthchecks.io can be pinged using GET or POST requests. They can also store log information upto 100KB.
Reference : https://healthchecks.io/docs/http_api/
Considering adding support for get and post request using hooks.
The difference between this and existing hooks is cron monitoring services can detect if backrest is not running. They need to ping regularly. If they are not pinged then they determine backrest is down.
Current webhooks will not notify if backrest itself is not running.
Adding GET and POST requests will allow many other services to be used.
The API protos (e.g. v1.*) should not be leaked in the interfaces exposed by the restic
wrapper package.
hello!
FYI - I am trying out backrest on linux, deployed via docker...
When i choose Add Repo
to "Add Restic Repository", the first field Repo Name
will not accept a dot in the name. As my normal repository nomenclature is myname.repo
so Backrest complained: invalid symbol
.
Restic allows me to put dots in the name, why not Backrest?
p.s. for now I continued with myname_repo
so it is a minor item for me.
For me, thr big problem with Restic is to manage, I mean, for each server it's installed, will need a different management location.
It's difficult to manage a big set of servers, so I would like to know if is there a plan for Backrest to manage all the Restic backups from a single WebUI.
It could be like Cockpit do with Linux Server, but it will be better if there is a dashbord where we can follow the latest backups results from all servers.
Add support to check the repo.
The following checks are available :
--read-data
--read-data-subset=n/t
--read-data-subset=x%
--read-data-subset=nS
The existing plans can be classified into backup plan and check plan. Check plan can be scheduled similar to backup plans.
This could be considered a high priority issue since repo can be corrupted due to updates or existing undiscovered bugs (eg : recent compression bug). If repo corruption is known soon, it can rectified.
I added a hook to one of my backups with the following configuration:
but it seems this hook is generating an error when executed:
This is the log I see in the frontend:
------- script -------
#! sh
echo 'Task: "backup for plan "Series"" at 2024-02-06T08:02:08Z
Event: snapshot end
Repo: Series
Plan: Series
Snapshot: 0e734343be9e288cafeaec9de515d473f709fb59babb6de0b30fdd6ead50c066
Overview:
- Data added: 1.001 GB
- Total files processed: 6610
- Total bytes processed: 1.001 TB
Backup Statistics:
- Files new: 4
- Files changed: 6
- Files unmodified: 6600
- Dirs new: 0
- Dirs changed: 9
- Dirs unmodified: 227
- Data blobs: 2026
- Tree blobs: 10
- Total duration: 115.67551034s
'
------- output -------
Error: exec: "sh": executable file not found in $PATH
I'm using the latest version 0.11.0, running on Docker.
I also can't open a shell connection to the docker container, so the shell binary is probably missing.
Hi,
I've added the hostname
to my docker compose file, but at the top of backrest interface, the hostname shows some ID.
I've looked at the container details, and the hostname is correctly showing there:
Then I realized the hostname was being pulled from the config.json file:
This was probably the ID from the 1st time I've created the backrest container...
Is this the expected behavior? For me I would expect the hostname to be dynamic, either by reading the container hostname when opening the user interface, or updating the config file on every time the containter is started.
The repo view should show the most recent operations on the repository.
Investigate good options for generating a documentation site to elaborate in more detailes on the capability of backrest.
Areas to expand in particular are:
To manage performance in the case of VERY large append only repositories --latest n
flag should be applied to restic snapshots
command.
A reasonable limit might be the most recent 2000
snapshots.
Right now, Backrest requires AZURE_ACCOUNT_KEY
when configuring a Azure Blob Repo. However, according to the documentation, both AZURE_ACCOUNT_KEY
as well as AZURE_ACCOUNT_SAS
are interchangeably valid.
First I'd like to thank you for this great project -- very nice idea, can't wait to see further releases! Makes restic so much easier to use and/or setup.
A thing I really miss is the support for pre / post scripts which can run before / after (successful or failed) backups. In my specific case I need to wake up my NAS (via WOL, can be done with an external script) before backrest can start the actual backup.
Would be really great to have such a feature on a per-repo basis. Thanks!
At start of install.sh i got the following error on a fresh Linux-Debian-Bookworm:
`++ uname -s
backrest.exe
2024-02-06T21:44:57.660+0100 INFO Installing restic C:\Program Files (x86)/restic/restic-0.16.3.exe to 0.16.3...
2024-02-06T21:44:59.574+0100 INFO Decompressing zip archive (size=9163895)...
2024-02-06T21:45:01.300+0100 INFO Installed restic 0.16.3
2024-02-06T21:45:01.301+0100 INFO Generating new auth secret
2024-02-06T21:45:01.301+0100 FATAL Error writing secret to file: open C:\Users\asd\AppData\Roaming/backrest/data/jwt-secret: The system cannot find the path specified.
Change
/backrest/data/jwt-secret
to \backrest\data\jwt-secret
rclone is ostensibly supported by repo URI's such as rclone:name:path
but users hit an error when backrest tries to fork the rclone executable (e.g. it can not be found on the path).
Let's try to figure out why the environment isn't properly being propagated such that the executable isn't found.
Hi there! Thanks for mentioning backrest on HN. I downloaded the version for macOS and first ran backrest via Right Click > Open. This ran the program in a foreground shell, so I realized I should run install.sh
. I opened a terminal, ran it, and got the following output:
❯ ./install.sh
Installing on Darwin
Installing backrest to /usr/local/bin
Password:
Creating launchd plist at /Library/LaunchAgents/com.backrest.plist
Enabling launchd plist com.backrest.plist
Unload failed: 5: Input/output error
Try running `launchctl bootout` as root for richer errors.
That being said, it apperas to be running successfully. Anything I should be aware of?
The state store will store quick-access information about operations performed by Restic UI. This is not for metadata that can be computed from the live repo (a caching abstraction should be built for that purpose).
Example data for state storage
Hi,
I would like to propose 3 UI improvements:
Collect and delete old entries in the operations log. An entry can be deleted if it is pushed out of the last 100 operations for any repo or plan.
Note: windows support is experimental
Right now windows fails to automatically download and install restic from a zip archive.
This can be worked around by manually installing restic to the path prompted at first run.
The UI is built against a specific version of restic. This ensures that changes to restic do not break the way the UI handles it's output (each UI release is validated with a single version of restic).
Use https://github.com/marketplace/actions/create-pull-request and simple sed
based automation to determine the latest version of restic and create a pull request if the hard coded version is out of date. Each time a PR is generated a new minor release should additionally be published that releases Restic UI with the latest minor version of restic.
Hi,
Sorry for the dumb question, but completely new to restic / backrest. HI have an S3 bucket repo setup, and a plan. But how do you configure a client to use backrest to push backups? Sorry if this is a completely stupid question.
Backrest downloads and installs restic automatically. An https connection is used to be sure that we are connecting to the real github.com but for added security backrest should verify the pgp signature of the downloaded artifact.
Implement protections matching the process used by restic's self-update script: https://github.com/restic/restic/tree/master/internal/selfupdate
Right now, we can modify BACKREST_PORT
, BACKREST_DATA
, BACKREST_RESTIC_COMMAND
, etc only through environment variables. Allow modifying them through commandline parameters or config file for easier install/usage.
Currently the only way is to use a script or terminal to modify these settings, atleast in windows.
Basic idea
Open questions: how should garbage collection in the case of an offline prune or forget operation be handled?
Recoil for state is unnecessary, React's builtin providers are sufficient for this project.
Hello,
I have these 2 hooks:
If I delete Hook 0, Hook 1 will become Hook 0 and will loose it's configuration, becoming this:
It is then not possible to save the configuration due to missing configuration:
Removing all Hooks and adding a new one does not work, unless we save the configuration after removing the existing hooks.
It happens in the plans and in the repositories configuration.
It should be possible to eliminate the global config state in favor of passing around a specific instance.
Hi,
I tried to build a new docker image from the git code. Everything runs well, the image is created but when I create a container from that image it does not run, instead I get the error:
exec /backrest: no such file or directory
I'm using the following commands to build the image and container:
docker build . -t testbackrest:v1 --no-cache
docker container run -it --name=atest -p 9899:9898 -e BACKREST_DATA=/data -e BACKREST_PORT=9898 -e XDG_CACHE_HOME=/cache -v $(pwd)/.backrest/config:/.config/backrest -v $(pwd)/.backrest/data:/data -v $(pwd)/.backrest/cache:/cache testbackrest:v1
If I run the built backrest executable, it runs well outside the container.
I then modified the Dockerfile to add a shell, and modified the entrypoint to launch the sheet instead of backrest. I looks as follows:
FROM busybox:1.35.0-uclibc as busybox
FROM golang:alpine as gobuild
RUN mkdir /tmp-orig
COPY backrest /backrest
FROM scratch
COPY --from=gobuild /tmp-orig /tmp
COPY --from=gobuild /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
# Now copy the static shell into base image.
COPY --from=busybox /bin/sh /bin/sh
COPY --from=busybox /bin/mkdir /bin/mkdir
COPY --from=busybox /bin/cat /bin/cat
COPY --from=busybox /bin/ls /bin/ls
ENTRYPOINT ["/bin/sh"]
#ENTRYPOINT ["/backrest"]
COPY backrest /backrest
Building the container from the new image now launches the shell, from where I can see the backrest executable, but launching it still throws the same error.
/ # ls -lsa
total 19124
4 drwxr-xr-x 1 0 0 4096 Feb 7 18:30 .
4 drwxr-xr-x 1 0 0 4096 Feb 7 18:30 ..
4 -rw------- 1 0 0 8 Feb 7 18:30 .ash_history
4 drwxr-xr-x 3 0 0 4096 Feb 7 18:30 .config
0 -rwxr-xr-x 1 0 0 0 Feb 7 18:30 .dockerenv
19088 -rwxr-xr-x 1 0 0 19542680 Feb 7 16:50 backrest
4 drwxr-xr-x 1 0 0 4096 Feb 7 17:18 bin
4 drwxr-xr-x 2 0 0 4096 Feb 7 16:53 cache
4 drwxr-xr-x 2 0 0 4096 Feb 7 16:53 data
0 drwxr-xr-x 5 0 0 340 Feb 7 18:30 dev
4 drwxr-xr-x 1 0 0 4096 Feb 7 18:30 etc
0 dr-xr-xr-x 224 0 0 0 Feb 7 18:30 proc
0 dr-xr-xr-x 12 0 0 0 Feb 7 18:30 sys
4 drwxr-xr-x 2 0 0 4096 Feb 7 17:18 tmp
/ # ./backrest
/bin/sh: ./backrest: not found
Am I doing something wrong? Why is backrest not working inside the container, but works outside?
The only problem I had when creating the development environment was running this command:
go install github.com/GeertJohan/go.rice@latest
package github.com/GeertJohan/go.rice is not a main package
Tried to remove the @latest
part from the command, but it does not seem to do anything, but there's no error as well.
I get the error package github.com/GeertJohan/go.rice is not a main package
I then installed rice from the package manager:
sudo apt install -y golang-rice
At least I learned how to add the shell command and external cli tools.
install.sh is sensitive to the current working directory.
Add
cd $(dirname "$0")
to the install and uninstall scripts to correct the working directory when invoked from different folders.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.