Giter VIP home page Giter VIP logo

arius's Introduction

Arius: a Lightweight Tiered Archival Solution for Azure Blob Storage

Arius.Core - Test Suite codecov CodeQL: Arius.Core

Arius Release Docker Arius.Core Version ClickOnce

Arius is a lightweight archival solution, specifically built to leverage the Azure Blob Archive tier.

The name derives from the Greek for 'immortal'.

Why?

  1. Supporting 3-2-1 Backup Strategy: Arius offers a secure and cost-effective offsite backup solution, complementing offline disk backups. The use of Azure Blob Archive tier allows for low-cost archiving, approximately 1 EUR per TB per month.

  2. Single Pane of Glass: Arius ensures seamless visibility to offline backups without requiring a separate application. By creating small "pointers" locally, they remain present in the local filesystem and easily discoverable in Windows Explorer.

  3. Encryption: For those with privacy concerns, Arius uses AES256/openssl-compatible encryption, providing robust client-side encryption.

  4. Deduplication: Arius does not store duplicate files or file parts. It offers file-level deduplication by default and allows optional variable block size deduplication within files.

How

Arius is a command-line tool (CLI) that can be run manually or scheduled. It can also be run as a Docker container.

Arius Explorer is a Windows application that offers a graphical user interface into an Arius repository.

Arius Explorer

Install it via ClickOnce by clicking on the shield at the top of the page.

The status icon is a combination of the state in the local file system (left half moon) and the state in the repository (right half moon). For example:

Icon Local file systen Azure Repository
Not present Pointer and Binary
Only the Pointer Pointer and Binary
Pointer and Binary Pointer and Binary

Arius CLI

Archive to Azure

CLI:

arius archive <path>
   --accountname <accountname>
   --accountkey  <accountkey>
   --passphrase  <passphrase>
   --container   <containername>
  [--remove-local]
  [--tier=<hot/cool/archive>]
  [--dedup]
  [--fasthash]

CLI in Docker:

docker run
  -v <absolute_path_to_archive>:/archive
 [-v <absolute_path_to_logs>:/logs]
  woutervanranst/arius

  archive
   --accountname <accountname>
   --accountkey  <accountkey>
   --passphrase  <passphrase>
   --container   <containername>
  [--remove-local]
  [--tier=<hot/cool/archive>]
  [--dedup]
  [--fasthash]

Restore from Azure

CLI:

arius restore <path>
   --accountname <accountname>
   --accountkey  <accountkey>
   --passphrase  <passphrase>
   --container   <containername>
  [--synchronize]
  [--download]
  [--keep-pointers]

CLI in Docker:

docker run
  -v <absolute_path_to_archive>:/archive
 [-v <absolute_path_to_logs>:/logs]
  woutervanranst/arius

  restore
   --accountname <accountname>
   --accountkey  <accountkey>
   --passphrase  <passphrase>
   --container   <containername>
  [--synchronize]
  [--download]
  [--keep-pointers]

Arguments

Argument Description Notes
path The path on the local file system For archive:
The root directory to archive

For restore:
  • If path is a directory: restore all pointer files in the (sub)directories.
  • If path is a file: restore this file.
logpath Path to the folder to store the logs OPTIONAL. NOTE: Only for Docker.
‑‑accountname, ‑n Storage Account Name
‑‑accountkey, ‑k Storage Account Key Can be set through:
  • Argument
  • Environment variable ARIUS_ACCOUNT_KEY
  • Docker environment variable ARIUS_ACCOUNT_KEY
‑‑passphrase, ‑p Passphrase with which the blobs are encrypted
‑‑container, ‑c Blob container to use OPTIONAL. Default: 'arius'.
‑‑remove-local Remove local file after a successful upload archive-only
OPTIONAL. Default: Local files are not deleted after archiving.
‑‑tier Blob storage tier (hot/cool/archive) archive-only
OPTIONAL. Default: 'archive'.
‑‑dedup Deduplicate on block level archive-only
OPTIONAL. Default: deduplicate on file level.
‑‑fasthash When a pointer file is present, use that hash instead of re-hashing the full file again archive-only
OPTIONAL. Default: false.
NOTE: Do NOT use this if the contents of the files are modified. Arius will not pick up the changes.
‑‑synchronize Bring the structure of the local file system (pointer files) in line with the latest state of the remote repository restore-only
OPTIONAL. Default: do not synchronize.
This command only touches the pointers (ie. .pointer.arius files). Other files are left untouched:
  • Pointers that exist in the archive but not locally are created.
  • Pointers that exist locally but not in the archive are deleted
‑‑download Download and restore the actual file (contents) restore-only
OPTIONAL. Default: do not download.
NOTE: If the file is in the archive blob tier, hydration to an online tier is started. Run the restore command again after ~15 hours to download the file.
‑‑keep-pointers Keep pointer files after downloading content files restore-only
OPTIONAL. Default: keep the pointers.

Restoring manually

For detailed instructions on restoring files manually, without using any of the Arius tools, please consult the Restore Manually section.

Advanced

Deduplication

Arius employs deduplication to optimize storage efficiency. For detailed explanations, diagrams, and benchmarks on deduplication, refer to the Deduplication section in the documentation.

Technical Documentation

Dependencies

Arius.Core Architecture

Arius.Core Domain Model

Attributions

Arius Icon by Freepik.

Many thanks to ReSharper and PostSharp for their open source generosity.

arius's People

Contributors

woutervanranst avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar

Forkers

sandeep-majumdar

arius's Issues

Re-work the Options mechanism

  • The dependency of IBlobCopier.Options into AzureRepositoryOptions is dirty -- to get the details of a repo we should not need the IBlobCopier.Options but it is needed for the ChunkRepository which needs an IBlobCopier.
  • Furthermore, the options for IBlobCopier are a subset of the AzureRepositoryOptions (ie Passphrase is not needed)

What we tried:

Arius Archive does not update Tier

when running arius archive to cool tier
and then again arius archive with archive tier

the files that are in cool tier remain in cool tier
expected (?) that they are moved to archive tier

Suggestion: insert a new block at A115 / before reconcileChunksWithManifestsBlock -> "setTierBlock"

Cross Platform support for same repository

Linux uses forward slashes "/"
Windows uses backslashes ""

This generates unnecessary updates:
10:01:22 info: Arius.Services.ManifestService[0] Added a/b.pointer.arius
10:01:22 info: Arius.Services.ManifestService[0] Marked a\b.pointer.arius as deleted
10:01:22 info: Arius.Services.ManifestService[0] Manifest '3faab5383260b514c8e8a4f3d94ce02c17c8f0f28445a63cbe45f376d126d60d' has modified entries, marked for writeback

Bug: concurrent access to pointerfile

With large files
with a lot of multithreading in the indexblock
one thread encounters the pointerfile and decides to upgrade (= delete and recreate)
the other thread encounters the binary and does the same

The process cannot access the file 'e:\SERIES_TODO_mkv.pointer.arius' because it is being used by another process. System.IO.IOException: The process cannot access the file 'e:\SERIES_TODO_.pointer.arius' because it is being used by another process. at System.IO.FileSystem.DeleteFile(String fullPath) at System.IO.FileInfo.Delete() at Arius.Core.Services.PointerService.OpenPointerFile(DirectoryInfo root, FileInfo pointerFileInfo) in C:\Users\Wouter\Documents\GitHub\Arius\Arius\Arius.Core\Services\PointerService.cs:line 177 at Arius.Core.Services.PointerService.GetPointerFile(DirectoryInfo root, FileInfo fi) in C:\Users\Wouter\Documents\GitHub\Arius\Arius\Arius.Core\Services\PointerService.cs:line 126 at Arius.Core.Commands.Archive.IndexBlock.<>c__DisplayClass10_0.<b__0>d.MoveNext() in C:\Users\Wouter\Documents\GitHub\Arius\Arius\Arius.Core\Commands\Archive\ArchiveCommandBlocks.cs:line 70 --- End of stack trace from previous location --- at System.Threading.Tasks.Parallel.<>c__501.<<ForEachAsync>b__50_0>d.MoveNext() --- End of stack trace from previous location --- at Arius.Core.Commands.Archive.IndexBlock.TaskBodyImplAsync(DirectoryInfo root) in C:\Users\Wouter\Documents\GitHub\Arius\Arius\Arius.Core\Commands\Archive\ArchiveCommandBlocks.cs:line 64 at Arius.Core.Commands.TaskBlockBase1.b__3_0() in C:\Users\Wouter\Documents\GitHub\Arius\Arius\Arius.Core\Commands\BlockBase.CS:line 119

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.