elastio / ssstar Goto Github PK

View Code? Open in Web Editor NEW

15.0 15.0 2.0 553 KB

tar-like utility for constructing archives of objects stored on S3 or S3-compatible services

License: Apache License 2.0

Rust 98.21% Shell 1.48% Dockerfile 0.31%

ssstar's People

Contributors

Stargazers

Watchers

Forkers

ethever

ssstar's Issues

Support partial restore of only certain objects without reading the entire stream

When restoring a tar archive from stdin, it's not possible to seek, and therefore the entire archive must be read even if the user wants to restore only a small subset of objects. However, for file and S3-based archives this isn't the case, and a seekable interface could be presented, so that partial restores can be more efficient.

The challenge to doing this for archives in object storage is that some intelligent buffering is still needed to avoid an excessive amount of GET operations being performed as the tar crate reads bits of metadata here and there.

Add support of Google Cloud Storage

Motivations

I would like to backup Google BigQuery. It supports export to the Google Cloud Storage which is done to multiple files. I would like to use ssstar to stream an archive of all files stored in Google Cloud Storage.

Alternatives

Google CLI support streaming multiples files from the Google Cloud Storage, but this approach doesn't suite for elastio backup.

Extract performance from a local file or stream is poor

Bug description

@arybitskyi tested extract performance both from elastio stream restore as well as from a local tar file, and reported the findings in a Google Sheet (NOTE: link is internal Elastio only)

To Reproduce

Steps to reproduce the behavior:

Extract a tar archive either from a local file with --file, or from the output of elastio stream restore with the --stdin argument
Note that extract performance is much slower than archive create performance

Expected behavior

Extract performance should be limited only by the speed of the network connection to S3. In this case meaning roughly the same as create performance.

Screenshots

N/A

Environment

ssstar 0.2.0
EC2 instance type m5.2xlarge
Amazon Linux 2

Additional context

N/A

Set user agent string to a valid APN ID per STG-010

The same task as described in elastio/elastio#6283 but specific to ssstar. The user agent string should be made fully configurable here so code in the elastio CLI that will already be generating a proper user agent string can just push that to ssstar as well.

Implement Storage API to support future incremental S3 backup

There's an experimental storage API already in the latest release, but it's not finished and not at all implemented. This needs to be finished and implemented so that Elastio can implement incremental S3 backup using ssstar as the engine.

Capture all object metadata in tar stream

Currently we capture only the object's key, and its data. But to be useful for a general-purpose backup tool, we need to preserve object metadata as well. Including, but not limited to:

tags
access control lists (ACLs)
user-defined metadata
original creation date
version ID

When restoring, the default behavior should be to restore all possible metadata (unfortunately it's not possible to preserve the original create date or the version ID), with an option to restore only specific metadata components instead.

This should be doable by storing our custom metadata in the tar archive in a separate "file" which appears in the archive before the actual object. We can make this file hidden and append a suffix like .$$metadata or something to ensure it's not confused for a real object. The extract stage would need to be modified to handle this, but that complexity would be hidden completely from the public API.

Add a Builder pattern to `Config`

In a PR elastio/elastio#6680 it became clear that because Config is marked non_exhaustive, the Config struct can't be initialized the normal way. Instead an ugly mut was used, so the Default() struct could be permuted with the appropriate config options.

This works fine but it's not particularly idiomatic or pleasant. A builder should be added for Config so that this type can be constructed without mutable state, while retaining the non_exhaustive attribute that is important for future flexibility.

Ssstar create sometimes crashes with panic

Bug description

When running multiple commands with ssstar, got this error on creating a tar archive from s3:

[ec2-user@ip-172-31-29-209 ~]$ ssstar create s3://ssstar-test/ --file backup.tar
⠁           D/L parts (all): file122486 (part 4)                                     [##############>-----] 7.03 GiB/9.54 GiB (210.26 MiB/s)
⠈       D/L parts (ordered): file122486 (part 5)                                     [##############>-----] 7.01 GiB/9.54 GiB (182.18 MiB/s)
⠈        Write parts to tar: file122486 (part 5)                                     [##############>-----] 7.01 GiB/9.54 GiB (187.83 MiB/s)
⠚           D/L parts (all): file122486 (part 7)                                     [##############>-----] 7.05 GiB/9.54 GiB (426.82 KiB/s)
⠚       D/L parts (ordered): file122486 (part 12)                                    [##############>-----] 7.05 GiB/9.54 GiB (427.22 KiB/s)
⠁        Write parts to tar: file122486 (part 8)                                     [##############>-----] 7.03 GiB/9.54 GiB (427.00 KiB/s)
⠖         Tar bytes written: Writing                                                 [##############>-----] 7.02 GiB/9.54 GiB (440.22 MiB/s)
⠁  Tar bytes uploaded to S3:                                                         [--------------------] 0B/9.54 GiB (0B/s)                                                                            
The application panicked (crashed).
Message:  called `Option::unwrap()` on a `None` value
Location: /home/ec2-user/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.24.1/src/fs/file.rs:642

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
Error:
   0: Error reading byte stream for object 'file122486' in S3 bucket 'ssstar-test'
   1: error reading a body from connection: Connection reset by peer (os error 104)
   2: error reading a body from connection: Connection reset by peer (os error 104)
   3: Connection reset by peer (os error 104)

Location:
   /home/ec2-user/.cargo/registry/src/github.com-1ecc6299db9ec823/ssstar-cli-0.2.0/src/main.rs:338

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.

To Reproduce

Steps to reproduce the behavior:

Run ssstar create s3://ssstar-test/ --file backup.tar multiple times

Expected behavior

App shouldn't panic and if the job fails, the error message should be more clear

Environment

OS: Linux 2.0
ssstar version: 0.3.0
aws account: 558095615212
s3 bucket: ssstar-test
Elastio CLI: 0.22.28 (22c95e0 2022-12-08 19:46:20)
Account-level stack: 2022-11-21
Region-level stack: 0.22.28

Additionally:

s3://ssstar-test consists of 100 files (97.5 MiB each), that were made by running count=1 ; while true; do sudo -E dd if=/dev/urandom of=file$count bs=1024 count=100000 ; count=$((count + 1)) ; done

Add support for selecting objects by tag

Root specific tests fails

Bug description

I tried to run all tests but some of them fails:
test objstore::minio::extract::large_object_no_filters::extract_from_archive_reader_to_bucket_root ... FAILED
test objstore::minio::extract::large_object_no_filters::extract_from_archive_file_to_bucket_root ... FAILED
test objstore::minio::extract::large_object_no_filters::extract_from_archive_on_bucket_to_bucket_root ... FAILED

full logs:


failures:

---- objstore::minio::extract::large_object_no_filters::extract_from_archive_file_to_bucket_root stdout ----
[ssstar-testing/src/test_data.rs:352] object.checksum_sha256 = Some(
    "JOrFYMK1e5u3xBgR1gGVsDrgd9EtDta4cE4Op2WMXXM=",
)
thread 'objstore::minio::extract::large_object_no_filters::extract_from_archive_file_to_bucket_root' panicked at 'assertion failed: `(left == right)`
  left: `[36, 234, 197, 96, 194, 181, 123, 155, 183, 196, 24, 17, 214, 1, 149, 176, 58, 224, 119, 209, 45, 14, 214, 184, 112, 78, 14, 167, 101, 140, 93, 115]`,
 right: `[203, 148, 32, 60, 63, 136, 43, 214, 1, 232, 138, 58, 62, 219, 15, 38, 58, 203, 171, 99, 168, 106, 35, 225, 53, 226, 155, 159, 139, 169, 163, 62]`: S3 object 'test' (key 'test') hash doesn't match expected value', /media/core/kingston-luks/ssstar/ssstar-testing/src/test_data.rs:481:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Log events from this test: 
2023-01-05T09:21:12.561162Z DEBUG ThreadId(541) ssstar_testing::minio: get() found minio server endpoint=127.0.0.1:36313
2023-01-05T09:21:12.568467Z DEBUG ThreadId(541) ssstar_testing::minio: Creating bucket bucket="large-object-no-filters-src"
2023-01-05T09:21:12.568567Z DEBUG ThreadId(541) ssstar_testing::minio: Transformed bucket name into valid and unique bucket ID bucket=d5f0da28-large-object-no-filters-src
2023-01-05T09:21:12.590903Z DEBUG ThreadId(541) ssstar_testing::minio: Bucket created bucket=d5f0da28-large-object-no-filters-src
2023-01-05T09:21:12.591014Z DEBUG ThreadId(541) ssstar_testing::minio: Creating bucket bucket="large-object-no-filters-tgt"
2023-01-05T09:21:12.591062Z DEBUG ThreadId(541) ssstar_testing::minio: Transformed bucket name into valid and unique bucket ID bucket=36e67a5e-large-object-no-filters-tgt
2023-01-05T09:21:12.605707Z DEBUG ThreadId(541) ssstar_testing::minio: Bucket created bucket=36e67a5e-large-object-no-filters-tgt
2023-01-05T09:21:17.207973Z DEBUG ThreadId(541) ssstar::create: Adding archive input url=s3://d5f0da28-large-object-no-filters-src/
2023-01-05T09:21:17.210596Z DEBUG ThreadId(541) ssstar::objstore::s3: Validating access to bucket bucket="d5f0da28-large-object-no-filters-src"
2023-01-05T09:21:17.215496Z DEBUG ThreadId(541) ssstar::objstore::s3: Access to bucket is confirmed bucket="d5f0da28-large-object-no-filters-src" region=None
2023-01-05T09:21:17.218121Z DEBUG ThreadId(541) ssstar::objstore::s3: Checked if bucket versioning is enabled bucket="d5f0da28-large-object-no-filters-src" versioning_enabled=true
2023-01-05T09:21:17.218177Z DEBUG ThreadId(541) ssstar::create: Confirmed bucket access for input url=s3://d5f0da28-large-object-no-filters-src/ bucket=S3Bucket { name: "d5f0da28-large-object-no-filters-src", versioning_enabled: true, region: None, client: "<...>" }
2023-01-05T09:21:17.218204Z DEBUG ThreadId(541) ssstar::create: Adding archive input to job input=CreateArchiveInput { bucket: S3Bucket { name: "d5f0da28-large-object-no-filters-src", versioning_enabled: true, region: None, client: "<...>" }, selector: Bucket }
2023-01-05T09:21:17.218237Z DEBUG ThreadId(541) ssstar::create: Listing objects for all inputs input_count=1
2023-01-05T09:21:17.218307Z DEBUG ThreadId(541) into_input_objects: ssstar::create: Listing all object store objects that match this archive input self=Bucket d5f0da28-large-object-no-filters-src
2023-01-05T09:21:17.218342Z DEBUG ThreadId(541) into_input_objects: ssstar::objstore::s3: Archive input matches everything in an S3 bucket bucket=d5f0da28-large-object-no-filters-src
2023-01-05T09:21:17.226316Z DEBUG ThreadId(541) into_input_objects: ssstar::objstore::s3: Matched all S3 objects in an S3 bucket input_objects=1 bucket=d5f0da28-large-object-no-filters-src
2023-01-05T09:21:17.226388Z DEBUG ThreadId(541) into_input_objects: ssstar::create: Listing matching objects completed self=Bucket d5f0da28-large-object-no-filters-src count=1
2023-01-05T09:21:17.226468Z DEBUG ThreadId(541) ssstar::create: Listed all objects in all inputs object_count=1
2023-01-05T09:21:17.234607Z DEBUG ThreadId(552) read_object_part{key="test" version_id=Some("de2d1e05-ac52-4fbe-961b-ee98d5b0d72a") byte_range=0..8388608}: ssstar::objstore::s3: Reading partial object
2023-01-05T09:21:17.236942Z DEBUG ThreadId(552) read_object_part{key="test" version_id=Some("de2d1e05-ac52-4fbe-961b-ee98d5b0d72a") byte_range=8388608..16777216}: ssstar::objstore::s3: Reading partial object
2023-01-05T09:21:17.238037Z DEBUG ThreadId(552) read_object_part{key="test" version_id=Some("de2d1e05-ac52-4fbe-961b-ee98d5b0d72a") byte_range=16777216..20971520}: ssstar::objstore::s3: Reading partial object
2023-01-05T09:21:17.260554Z DEBUG ThreadId(541) ssstar::create: Reading object and writing to tar archive key=test size=20971520
2023-01-05T09:21:17.381106Z DEBUG ThreadId(541) ssstar::create: Streamed object data into tar archive key=test size=20971520 final_part_number=2
2023-01-05T09:21:17.381205Z DEBUG ThreadId(541) ssstar::create: Completed processing of all input objects
2023-01-05T09:21:17.386282Z DEBUG ThreadId(541) ssstar::objstore::s3: Validating access to bucket bucket="36e67a5e-large-object-no-filters-tgt"
2023-01-05T09:21:17.388162Z DEBUG ThreadId(541) ssstar::objstore::s3: Access to bucket is confirmed bucket="36e67a5e-large-object-no-filters-tgt" region=None
2023-01-05T09:21:17.389633Z DEBUG ThreadId(541) ssstar::objstore::s3: Checked if bucket versioning is enabled bucket="36e67a5e-large-object-no-filters-tgt" versioning_enabled=true
2023-01-05T09:21:17.389899Z  INFO ThreadId(541) run{source_archive=File { path: "/tmp/extract-test.xRM9SSeWU5Tc/test.tar", metadata: Metadata { file_type: FileType(FileType { mode: 33188 }), is_dir: false, is_file: true, permissions: Permissions(FilePermissions { mode: 33188 }), modified: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 379078325 }), accessed: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 231080322 }), created: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 231080322 }), .. } } target_bucket="36e67a5e-large-object-no-filters-tgt" target_prefix=}: ssstar::extract: Starting extract archive job self.filters=[] self.archive_size=Some(20973056)
2023-01-05T09:21:17.390104Z DEBUG ThreadId(546) ssstar::extract: Starting tar entry processor task
2023-01-05T09:21:17.390112Z DEBUG ThreadId(674) run{source_archive=File { path: "/tmp/extract-test.xRM9SSeWU5Tc/test.tar", metadata: Metadata { file_type: FileType(FileType { mode: 33188 }), is_dir: false, is_file: true, permissions: Permissions(FilePermissions { mode: 33188 }), modified: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 379078325 }), accessed: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 231080322 }), created: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 231080322 }), .. } } target_bucket="36e67a5e-large-object-no-filters-tgt" target_prefix=}:read_tar_entries_blocking{source_archive=File { path: "/tmp/extract-test.xRM9SSeWU5Tc/test.tar", metadata: Metadata { file_type: FileType(FileType { mode: 33188 }), is_dir: false, is_file: true, permissions: Permissions(FilePermissions { mode: 33188 }), modified: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 379078325 }), accessed: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 231080322 }), created: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 231080322 }), .. } }}: ssstar::extract: Starting blocking tar read task
2023-01-05T09:21:17.390359Z DEBUG ThreadId(674) run{source_archive=File { path: "/tmp/extract-test.xRM9SSeWU5Tc/test.tar", metadata: Metadata { file_type: FileType(FileType { mode: 33188 }), is_dir: false, is_file: true, permissions: Permissions(FilePermissions { mode: 33188 }), modified: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 379078325 }), accessed: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 231080322 }), created: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 231080322 }), .. } } target_bucket="36e67a5e-large-object-no-filters-tgt" target_prefix=}:read_tar_entries_blocking{source_archive=File { path: "/tmp/extract-test.xRM9SSeWU5Tc/test.tar", metadata: Metadata { file_type: FileType(FileType { mode: 33188 }), is_dir: false, is_file: true, permissions: Permissions(FilePermissions { mode: 33188 }), modified: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 379078325 }), accessed: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 231080322 }), created: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 231080322 }), .. } }}:Processing tar entry{path=test len=20971520}: ssstar::extract: No filters are applied so this entry is included
2023-01-05T09:21:17.390439Z DEBUG ThreadId(674) run{source_archive=File { path: "/tmp/extract-test.xRM9SSeWU5Tc/test.tar", metadata: Metadata { file_type: FileType(FileType { mode: 33188 }), is_dir: false, is_file: true, permissions: Permissions(FilePermissions { mode: 33188 }), modified: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 379078325 }), accessed: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 231080322 }), created: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 231080322 }), .. } } target_bucket="36e67a5e-large-object-no-filters-tgt" target_prefix=}:read_tar_entries_blocking{source_archive=File { path: "/tmp/extract-test.xRM9SSeWU5Tc/test.tar", metadata: Metadata { file_type: FileType(FileType { mode: 33188 }), is_dir: false, is_file: true, permissions: Permissions(FilePermissions { mode: 33188 }), modified: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 379078325 }), accessed: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 231080322 }), created: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 231080322 }), .. } }}:Processing tar entry{path=test len=20971520}: ssstar::extract: Processing entry as multi-part
2023-01-05T09:21:17.398629Z DEBUG ThreadId(674) run{source_archive=File { path: "/tmp/extract-test.xRM9SSeWU5Tc/test.tar", metadata: Metadata { file_type: FileType(FileType { mode: 33188 }), is_dir: false, is_file: true, permissions: Permissions(FilePermissions { mode: 33188 }), modified: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 379078325 }), accessed: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 231080322 }), created: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 231080322 }), .. } } target_bucket="36e67a5e-large-object-no-filters-tgt" target_prefix=}:read_tar_entries_blocking{source_archive=File { path: "/tmp/extract-test.xRM9SSeWU5Tc/test.tar", metadata: Metadata { file_type: FileType(FileType { mode: 33188 }), is_dir: false, is_file: true, permissions: Permissions(FilePermissions { mode: 33188 }), modified: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 379078325 }), accessed: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 231080322 }), created: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 231080322 }), .. } }}: ssstar::extract: Completed processing all tar entries
2023-01-05T09:21:17.398724Z DEBUG ThreadId(674) run{source_archive=File { path: "/tmp/extract-test.xRM9SSeWU5Tc/test.tar", metadata: Metadata { file_type: FileType(FileType { mode: 33188 }), is_dir: false, is_file: true, permissions: Permissions(FilePermissions { mode: 33188 }), modified: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 379078325 }), accessed: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 231080322 }), created: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 231080322 }), .. } } target_bucket="36e67a5e-large-object-no-filters-tgt" target_prefix=}:read_tar_entries_blocking{source_archive=File { path: "/tmp/extract-test.xRM9SSeWU5Tc/test.tar", metadata: Metadata { file_type: FileType(FileType { mode: 33188 }), is_dir: false, is_file: true, permissions: Permissions(FilePermissions { mode: 33188 }), modified: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 379078325 }), accessed: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 231080322 }), created: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 231080322 }), .. } }}: ssstar::extract: Blocking tar read task completing successfully
2023-01-05T09:21:17.398878Z DEBUG ThreadId(551) process_tar_entries{target_prefix="" target_bucket="36e67a5e-large-object-no-filters-tgt"}: ssstar::extract: Starting multipart file upload path=test len=20971520 key=test
2023-01-05T09:21:17.401414Z DEBUG ThreadId(551) process_tar_entries{target_prefix="" target_bucket="36e67a5e-large-object-no-filters-tgt"}: ssstar::extract: Uploading file part path=test len=8388608
2023-01-05T09:21:17.402889Z DEBUG ThreadId(551) process_tar_entries{target_prefix="" target_bucket="36e67a5e-large-object-no-filters-tgt"}: ssstar::extract: Uploading file part path=test len=8388608
2023-01-05T09:21:17.403001Z DEBUG ThreadId(551) multipart_object_writer{key="test" upload_id="YjU1YWQ4ZDMtMWY3NS00ZDgyLWJkNGYtOWQxZDFiM2VlNDhmLjkwNTQ1ZjJkLTRlNjUtNGY2Yi04MzI4LTE1ZWJjNmYxYTM4OQ" bucket=36e67a5e-large-object-no-filters-tgt}: ssstar::objstore::s3: Commencing multi-part upload
2023-01-05T09:21:17.404314Z DEBUG ThreadId(551) multipart_object_writer{key="test" upload_id="YjU1YWQ4ZDMtMWY3NS00ZDgyLWJkNGYtOWQxZDFiM2VlNDhmLjkwNTQ1ZjJkLTRlNjUtNGY2Yi04MzI4LTE1ZWJjNmYxYTM4OQ" bucket=36e67a5e-large-object-no-filters-tgt}:upload chunk{part_number=1 chunk_size=8388608}: ssstar::objstore::s3: Uploading multi-part chunk
2023-01-05T09:21:17.412995Z DEBUG ThreadId(546) process_tar_entries{target_prefix="" target_bucket="36e67a5e-large-object-no-filters-tgt"}: ssstar::extract: Uploading final file part path=test len=4194304
2023-01-05T09:21:17.414311Z DEBUG ThreadId(546) ssstar::writers: encountered end of duplex stream; worker task exiting
2023-01-05T09:21:18.299316Z DEBUG ThreadId(546) multipart_object_writer{key="test" upload_id="YjU1YWQ4ZDMtMWY3NS00ZDgyLWJkNGYtOWQxZDFiM2VlNDhmLjkwNTQ1ZjJkLTRlNjUtNGY2Yi04MzI4LTE1ZWJjNmYxYTM4OQ" bucket=36e67a5e-large-object-no-filters-tgt}:upload chunk{part_number=2 chunk_size=8388608}: ssstar::objstore::s3: Uploading multi-part chunk
2023-01-05T09:21:18.836985Z DEBUG ThreadId(546) multipart_object_writer{key="test" upload_id="YjU1YWQ4ZDMtMWY3NS00ZDgyLWJkNGYtOWQxZDFiM2VlNDhmLjkwNTQ1ZjJkLTRlNjUtNGY2Yi04MzI4LTE1ZWJjNmYxYTM4OQ" bucket=36e67a5e-large-object-no-filters-tgt}:upload chunk{part_number=3 chunk_size=4194304}: ssstar::objstore::s3: Uploading multi-part chunk
2023-01-05T09:21:19.109543Z DEBUG ThreadId(546) multipart_object_writer{key="test" upload_id="YjU1YWQ4ZDMtMWY3NS00ZDgyLWJkNGYtOWQxZDFiM2VlNDhmLjkwNTQ1ZjJkLTRlNjUtNGY2Yi04MzI4LTE1ZWJjNmYxYTM4OQ" bucket=36e67a5e-large-object-no-filters-tgt}:upload chunk{part_number=1 chunk_size=8388608}: ssstar::objstore::s3: Uploaded multi-part chunk e_tag="7f5f484de6b97c780ef63d0e5763cf78" sha256="fNqqiz4HIy5jckO3taqlzkyU+wi+nd2fF5knAfUPiQI="
2023-01-05T09:21:19.157617Z DEBUG ThreadId(551) multipart_object_writer{key="test" upload_id="YjU1YWQ4ZDMtMWY3NS00ZDgyLWJkNGYtOWQxZDFiM2VlNDhmLjkwNTQ1ZjJkLTRlNjUtNGY2Yi04MzI4LTE1ZWJjNmYxYTM4OQ" bucket=36e67a5e-large-object-no-filters-tgt}:upload chunk{part_number=3 chunk_size=4194304}: ssstar::objstore::s3: Uploaded multi-part chunk e_tag="51d4cc6f9915c5ee9ee0aa200dd9a4fc" sha256="dFBIHlyoxPGJ4VlSMYqay9DghFdIWBE9ypf+1YP5zUg="
2023-01-05T09:21:19.169221Z DEBUG ThreadId(551) multipart_object_writer{key="test" upload_id="YjU1YWQ4ZDMtMWY3NS00ZDgyLWJkNGYtOWQxZDFiM2VlNDhmLjkwNTQ1ZjJkLTRlNjUtNGY2Yi04MzI4LTE1ZWJjNmYxYTM4OQ" bucket=36e67a5e-large-object-no-filters-tgt}:upload chunk{part_number=2 chunk_size=8388608}: ssstar::objstore::s3: Uploaded multi-part chunk e_tag="158fb90af4014d106e211e2b16e16f3b" sha256="XzRUhiUsYoETL5lGdqdUbs5PAqqcNymF//OPhNn+Nh8="
2023-01-05T09:21:19.169304Z DEBUG ThreadId(551) multipart_object_writer{key="test" upload_id="YjU1YWQ4ZDMtMWY3NS00ZDgyLWJkNGYtOWQxZDFiM2VlNDhmLjkwNTQ1ZjJkLTRlNjUtNGY2Yi04MzI4LTE1ZWJjNmYxYTM4OQ" bucket=36e67a5e-large-object-no-filters-tgt}: ssstar::objstore::s3: All parts uploaded; completing multi-part upload total_parts=3 total_bytes=20971520
2023-01-05T09:21:19.173977Z DEBUG ThreadId(546) process_tar_entries{target_prefix="" target_bucket="36e67a5e-large-object-no-filters-tgt"}: ssstar::extract: Finished multipart file upload path=test len=20971520 key=test
2023-01-05T09:21:19.174034Z DEBUG ThreadId(546) process_tar_entries{target_prefix="" target_bucket="36e67a5e-large-object-no-filters-tgt"}: ssstar::extract: Entry sender dropped; no more tar entries to process
2023-01-05T09:21:19.174151Z DEBUG ThreadId(546) ssstar::extract: Tar entry processor task completed successfully
2023-01-05T09:21:19.174252Z  INFO ThreadId(541) run{source_archive=File { path: "/tmp/extract-test.xRM9SSeWU5Tc/test.tar", metadata: Metadata { file_type: FileType(FileType { mode: 33188 }), is_dir: false, is_file: true, permissions: Permissions(FilePermissions { mode: 33188 }), modified: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 379078325 }), accessed: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 231080322 }), created: Ok(SystemTime { tv_sec: 1672910477, tv_nsec: 231080322 }), .. } } target_bucket="36e67a5e-large-object-no-filters-tgt" target_prefix=}: ssstar::extract: Finished extract job


---- objstore::minio::extract::large_object_no_filters::extract_from_archive_reader_to_bucket_root stdout ----
[ssstar-testing/src/test_data.rs:352] object.checksum_sha256 = Some(
    "Hv+/sEGd/KeOPSnJWNnAZNfRcDobv6Vax7E/ibYN+dg=",
)
thread 'objstore::minio::extract::large_object_no_filters::extract_from_archive_reader_to_bucket_root' panicked at 'assertion failed: `(left == right)`
  left: `[30, 255, 191, 176, 65, 157, 252, 167, 142, 61, 41, 201, 88, 217, 192, 100, 215, 209, 112, 58, 27, 191, 165, 90, 199, 177, 63, 137, 182, 13, 249, 216]`,
 right: `[197, 73, 189, 91, 244, 117, 46, 235, 161, 29, 17, 240, 75, 58, 190, 107, 150, 8, 170, 40, 102, 105, 21, 121, 136, 3, 231, 129, 31, 146, 49, 74]`: S3 object 'test' (key 'test') hash doesn't match expected value', /media/core/kingston-luks/ssstar/ssstar-testing/src/test_data.rs:481:9
Log events from this test: 
2023-01-05T09:21:12.707311Z DEBUG ThreadId(569) ssstar_testing::minio: get() found minio server endpoint=127.0.0.1:36313
2023-01-05T09:21:12.721617Z DEBUG ThreadId(569) ssstar_testing::minio: Creating bucket bucket="large-object-no-filters-src"
2023-01-05T09:21:12.721705Z DEBUG ThreadId(569) ssstar_testing::minio: Transformed bucket name into valid and unique bucket ID bucket=942b2a53-large-object-no-filters-src
2023-01-05T09:21:12.752741Z DEBUG ThreadId(569) ssstar_testing::minio: Bucket created bucket=942b2a53-large-object-no-filters-src
2023-01-05T09:21:12.752869Z DEBUG ThreadId(569) ssstar_testing::minio: Creating bucket bucket="large-object-no-filters-tgt"
2023-01-05T09:21:12.752930Z DEBUG ThreadId(569) ssstar_testing::minio: Transformed bucket name into valid and unique bucket ID bucket=1b40191a-large-object-no-filters-tgt
2023-01-05T09:21:12.774438Z DEBUG ThreadId(569) ssstar_testing::minio: Bucket created bucket=1b40191a-large-object-no-filters-tgt
2023-01-05T09:21:17.160341Z DEBUG ThreadId(569) ssstar::create: Adding archive input url=s3://942b2a53-large-object-no-filters-src/
2023-01-05T09:21:17.163146Z DEBUG ThreadId(569) ssstar::objstore::s3: Validating access to bucket bucket="942b2a53-large-object-no-filters-src"
2023-01-05T09:21:17.165441Z DEBUG ThreadId(569) ssstar::objstore::s3: Access to bucket is confirmed bucket="942b2a53-large-object-no-filters-src" region=None
2023-01-05T09:21:17.167459Z DEBUG ThreadId(569) ssstar::objstore::s3: Checked if bucket versioning is enabled bucket="942b2a53-large-object-no-filters-src" versioning_enabled=true
2023-01-05T09:21:17.167521Z DEBUG ThreadId(569) ssstar::create: Confirmed bucket access for input url=s3://942b2a53-large-object-no-filters-src/ bucket=S3Bucket { name: "942b2a53-large-object-no-filters-src", versioning_enabled: true, region: None, client: "<...>" }
2023-01-05T09:21:17.167562Z DEBUG ThreadId(569) ssstar::create: Adding archive input to job input=CreateArchiveInput { bucket: S3Bucket { name: "942b2a53-large-object-no-filters-src", versioning_enabled: true, region: None, client: "<...>" }, selector: Bucket }
2023-01-05T09:21:17.167608Z DEBUG ThreadId(569) ssstar::create: Listing objects for all inputs input_count=1
2023-01-05T09:21:17.167729Z DEBUG ThreadId(569) into_input_objects: ssstar::create: Listing all object store objects that match this archive input self=Bucket 942b2a53-large-object-no-filters-src
2023-01-05T09:21:17.167778Z DEBUG ThreadId(569) into_input_objects: ssstar::objstore::s3: Archive input matches everything in an S3 bucket bucket=942b2a53-large-object-no-filters-src
2023-01-05T09:21:17.180230Z DEBUG ThreadId(569) into_input_objects: ssstar::objstore::s3: Matched all S3 objects in an S3 bucket input_objects=1 bucket=942b2a53-large-object-no-filters-src
2023-01-05T09:21:17.180277Z DEBUG ThreadId(569) into_input_objects: ssstar::create: Listing matching objects completed self=Bucket 942b2a53-large-object-no-filters-src count=1
2023-01-05T09:21:17.180366Z DEBUG ThreadId(569) ssstar::create: Listed all objects in all inputs object_count=1
2023-01-05T09:21:17.180769Z DEBUG ThreadId(576) read_object_part{key="test" version_id=Some("acbc5ebb-52d3-4229-ade4-2c6e088786a1") byte_range=0..8388608}: ssstar::objstore::s3: Reading partial object
2023-01-05T09:21:17.181664Z DEBUG ThreadId(576) read_object_part{key="test" version_id=Some("acbc5ebb-52d3-4229-ade4-2c6e088786a1") byte_range=8388608..16777216}: ssstar::objstore::s3: Reading partial object
2023-01-05T09:21:17.182555Z DEBUG ThreadId(576) read_object_part{key="test" version_id=Some("acbc5ebb-52d3-4229-ade4-2c6e088786a1") byte_range=16777216..20971520}: ssstar::objstore::s3: Reading partial object
2023-01-05T09:21:17.239663Z DEBUG ThreadId(569) ssstar::create: Reading object and writing to tar archive key=test size=20971520
2023-01-05T09:21:17.372983Z DEBUG ThreadId(569) ssstar::create: Streamed object data into tar archive key=test size=20971520 final_part_number=2
2023-01-05T09:21:17.373153Z DEBUG ThreadId(569) ssstar::create: Completed processing of all input objects
2023-01-05T09:21:17.392166Z DEBUG ThreadId(569) ssstar::objstore::s3: Validating access to bucket bucket="1b40191a-large-object-no-filters-tgt"
2023-01-05T09:21:17.394264Z DEBUG ThreadId(569) ssstar::objstore::s3: Access to bucket is confirmed bucket="1b40191a-large-object-no-filters-tgt" region=None
2023-01-05T09:21:17.395722Z DEBUG ThreadId(569) ssstar::objstore::s3: Checked if bucket versioning is enabled bucket="1b40191a-large-object-no-filters-tgt" versioning_enabled=true
2023-01-05T09:21:17.395921Z  INFO ThreadId(569) run{source_archive=Reader("dyn Read") target_bucket="1b40191a-large-object-no-filters-tgt" target_prefix=}: ssstar::extract: Starting extract archive job self.filters=[] self.archive_size=None
2023-01-05T09:21:17.396068Z DEBUG ThreadId(575) ssstar::extract: Starting tar entry processor task
2023-01-05T09:21:17.396069Z DEBUG ThreadId(659) run{source_archive=Reader("dyn Read") target_bucket="1b40191a-large-object-no-filters-tgt" target_prefix=}:read_tar_entries_blocking{source_archive=Reader("dyn Read")}: ssstar::extract: Starting blocking tar read task
2023-01-05T09:21:17.396312Z DEBUG ThreadId(659) run{source_archive=Reader("dyn Read") target_bucket="1b40191a-large-object-no-filters-tgt" target_prefix=}:read_tar_entries_blocking{source_archive=Reader("dyn Read")}:Processing tar entry{path=test len=20971520}: ssstar::extract: No filters are applied so this entry is included
2023-01-05T09:21:17.396372Z DEBUG ThreadId(659) run{source_archive=Reader("dyn Read") target_bucket="1b40191a-large-object-no-filters-tgt" target_prefix=}:read_tar_entries_blocking{source_archive=Reader("dyn Read")}:Processing tar entry{path=test len=20971520}: ssstar::extract: Processing entry as multi-part
2023-01-05T09:21:17.396524Z DEBUG ThreadId(578) process_tar_entries{target_prefix="" target_bucket="1b40191a-large-object-no-filters-tgt"}: ssstar::extract: Starting multipart file upload path=test len=20971520 key=test
2023-01-05T09:21:17.399513Z DEBUG ThreadId(575) process_tar_entries{target_prefix="" target_bucket="1b40191a-large-object-no-filters-tgt"}: ssstar::extract: Uploading file part path=test len=8388608
2023-01-05T09:21:17.402012Z DEBUG ThreadId(582) multipart_object_writer{key="test" upload_id="YjU1YWQ4ZDMtMWY3NS00ZDgyLWJkNGYtOWQxZDFiM2VlNDhmLjEyZDkxNjk3LWJhMzgtNGFhMC04OGJjLTFkZGQ5NWIyMWY1Mg" bucket=1b40191a-large-object-no-filters-tgt}: ssstar::objstore::s3: Commencing multi-part upload
2023-01-05T09:21:17.403807Z DEBUG ThreadId(578) process_tar_entries{target_prefix="" target_bucket="1b40191a-large-object-no-filters-tgt"}: ssstar::extract: Uploading file part path=test len=8388608
2023-01-05T09:21:17.404705Z DEBUG ThreadId(659) run{source_archive=Reader("dyn Read") target_bucket="1b40191a-large-object-no-filters-tgt" target_prefix=}:read_tar_entries_blocking{source_archive=Reader("dyn Read")}: ssstar::extract: Completed processing all tar entries
2023-01-05T09:21:17.404770Z DEBUG ThreadId(659) run{source_archive=Reader("dyn Read") target_bucket="1b40191a-large-object-no-filters-tgt" target_prefix=}:read_tar_entries_blocking{source_archive=Reader("dyn Read")}: ssstar::extract: Blocking tar read task completing successfully
2023-01-05T09:21:17.404961Z DEBUG ThreadId(578) process_tar_entries{target_prefix="" target_bucket="1b40191a-large-object-no-filters-tgt"}: ssstar::extract: Uploading final file part path=test len=4194304
2023-01-05T09:21:17.409838Z DEBUG ThreadId(575) ssstar::writers: encountered end of duplex stream; worker task exiting
2023-01-05T09:21:17.410043Z DEBUG ThreadId(575) multipart_object_writer{key="test" upload_id="YjU1YWQ4ZDMtMWY3NS00ZDgyLWJkNGYtOWQxZDFiM2VlNDhmLjEyZDkxNjk3LWJhMzgtNGFhMC04OGJjLTFkZGQ5NWIyMWY1Mg" bucket=1b40191a-large-object-no-filters-tgt}:upload chunk{part_number=1 chunk_size=8388608}: ssstar::objstore::s3: Uploading multi-part chunk
2023-01-05T09:21:18.313150Z DEBUG ThreadId(575) multipart_object_writer{key="test" upload_id="YjU1YWQ4ZDMtMWY3NS00ZDgyLWJkNGYtOWQxZDFiM2VlNDhmLjEyZDkxNjk3LWJhMzgtNGFhMC04OGJjLTFkZGQ5NWIyMWY1Mg" bucket=1b40191a-large-object-no-filters-tgt}:upload chunk{part_number=2 chunk_size=8388608}: ssstar::objstore::s3: Uploading multi-part chunk
2023-01-05T09:21:18.906212Z DEBUG ThreadId(575) multipart_object_writer{key="test" upload_id="YjU1YWQ4ZDMtMWY3NS00ZDgyLWJkNGYtOWQxZDFiM2VlNDhmLjEyZDkxNjk3LWJhMzgtNGFhMC04OGJjLTFkZGQ5NWIyMWY1Mg" bucket=1b40191a-large-object-no-filters-tgt}:upload chunk{part_number=3 chunk_size=4194304}: ssstar::objstore::s3: Uploading multi-part chunk
2023-01-05T09:21:19.229606Z DEBUG ThreadId(578) multipart_object_writer{key="test" upload_id="YjU1YWQ4ZDMtMWY3NS00ZDgyLWJkNGYtOWQxZDFiM2VlNDhmLjEyZDkxNjk3LWJhMzgtNGFhMC04OGJjLTFkZGQ5NWIyMWY1Mg" bucket=1b40191a-large-object-no-filters-tgt}:upload chunk{part_number=3 chunk_size=4194304}: ssstar::objstore::s3: Uploaded multi-part chunk e_tag="4c34568466e02ee3121680c2614ede27" sha256="jAipFqjheVwBa9dyT4tDXt/F3KycAZE78VRh+LKsuMQ="
2023-01-05T09:21:19.275636Z DEBUG ThreadId(578) multipart_object_writer{key="test" upload_id="YjU1YWQ4ZDMtMWY3NS00ZDgyLWJkNGYtOWQxZDFiM2VlNDhmLjEyZDkxNjk3LWJhMzgtNGFhMC04OGJjLTFkZGQ5NWIyMWY1Mg" bucket=1b40191a-large-object-no-filters-tgt}:upload chunk{part_number=2 chunk_size=8388608}: ssstar::objstore::s3: Uploaded multi-part chunk e_tag="235135148df76539bcc75028fc7d296c" sha256="idCNISzNqgOBg3Yd/+eL086l4beX/DpJO7xOIc3nLdU="
2023-01-05T09:21:19.289588Z DEBUG ThreadId(578) multipart_object_writer{key="test" upload_id="YjU1YWQ4ZDMtMWY3NS00ZDgyLWJkNGYtOWQxZDFiM2VlNDhmLjEyZDkxNjk3LWJhMzgtNGFhMC04OGJjLTFkZGQ5NWIyMWY1Mg" bucket=1b40191a-large-object-no-filters-tgt}:upload chunk{part_number=1 chunk_size=8388608}: ssstar::objstore::s3: Uploaded multi-part chunk e_tag="2316297fbbc4ce2304ada31aebdeef0b" sha256="RGicJ57GwOyzxneEAMJTNRSvpwPjm9tQIL266L8rb0g="
2023-01-05T09:21:19.289679Z DEBUG ThreadId(578) multipart_object_writer{key="test" upload_id="YjU1YWQ4ZDMtMWY3NS00ZDgyLWJkNGYtOWQxZDFiM2VlNDhmLjEyZDkxNjk3LWJhMzgtNGFhMC04OGJjLTFkZGQ5NWIyMWY1Mg" bucket=1b40191a-large-object-no-filters-tgt}: ssstar::objstore::s3: All parts uploaded; completing multi-part upload total_parts=3 total_bytes=20971520
2023-01-05T09:21:19.294778Z DEBUG ThreadId(582) process_tar_entries{target_prefix="" target_bucket="1b40191a-large-object-no-filters-tgt"}: ssstar::extract: Finished multipart file upload path=test len=20971520 key=test
2023-01-05T09:21:19.294842Z DEBUG ThreadId(582) process_tar_entries{target_prefix="" target_bucket="1b40191a-large-object-no-filters-tgt"}: ssstar::extract: Entry sender dropped; no more tar entries to process
2023-01-05T09:21:19.294952Z DEBUG ThreadId(582) ssstar::extract: Tar entry processor task completed successfully
2023-01-05T09:21:19.295036Z  INFO ThreadId(569) run{source_archive=Reader("dyn Read") target_bucket="1b40191a-large-object-no-filters-tgt" target_prefix=}: ssstar::extract: Finished extract job


---- objstore::minio::extract::large_object_no_filters::extract_from_archive_on_bucket_to_bucket_root stdout ----
[ssstar-testing/src/test_data.rs:352] object.checksum_sha256 = Some(
    "4Hwz+RBQEn4A3k2kk5NMVo3bU0hqokM6SGWCF+smkh4=",
)
thread 'objstore::minio::extract::large_object_no_filters::extract_from_archive_on_bucket_to_bucket_root' panicked at 'assertion failed: `(left == right)`
  left: `[224, 124, 51, 249, 16, 80, 18, 126, 0, 222, 77, 164, 147, 147, 76, 86, 141, 219, 83, 72, 106, 162, 67, 58, 72, 101, 130, 23, 235, 38, 146, 30]`,
 right: `[159, 191, 191, 89, 94, 97, 27, 45, 149, 152, 86, 221, 215, 69, 206, 30, 192, 233, 54, 117, 21, 49, 72, 182, 220, 204, 178, 208, 162, 83, 94, 186]`: S3 object 'test' (key 'test') hash doesn't match expected value', /media/core/kingston-luks/ssstar/ssstar-testing/src/test_data.rs:481:9
Log events from this test: 
2023-01-05T09:21:12.615415Z DEBUG ThreadId(556) ssstar_testing::minio: get() found minio server endpoint=127.0.0.1:36313
2023-01-05T09:21:12.623747Z DEBUG ThreadId(556) ssstar_testing::minio: Creating bucket bucket="large-object-no-filters-src"
2023-01-05T09:21:12.623895Z DEBUG ThreadId(556) ssstar_testing::minio: Transformed bucket name into valid and unique bucket ID bucket=ded5cceb-large-object-no-filters-src
2023-01-05T09:21:12.656439Z DEBUG ThreadId(556) ssstar_testing::minio: Bucket created bucket=ded5cceb-large-object-no-filters-src
2023-01-05T09:21:12.656565Z DEBUG ThreadId(556) ssstar_testing::minio: Creating bucket bucket="large-object-no-filters-tgt"
2023-01-05T09:21:12.656620Z DEBUG ThreadId(556) ssstar_testing::minio: Transformed bucket name into valid and unique bucket ID bucket=417610ee-large-object-no-filters-tgt
2023-01-05T09:21:12.676288Z DEBUG ThreadId(556) ssstar_testing::minio: Bucket created bucket=417610ee-large-object-no-filters-tgt
2023-01-05T09:21:17.148228Z DEBUG ThreadId(556) ssstar::create: Adding archive input url=s3://ded5cceb-large-object-no-filters-src/
2023-01-05T09:21:17.150818Z DEBUG ThreadId(556) ssstar::objstore::s3: Validating access to bucket bucket="ded5cceb-large-object-no-filters-src"
2023-01-05T09:21:17.153044Z DEBUG ThreadId(556) ssstar::objstore::s3: Access to bucket is confirmed bucket="ded5cceb-large-object-no-filters-src" region=None
2023-01-05T09:21:17.157316Z DEBUG ThreadId(556) ssstar::objstore::s3: Checked if bucket versioning is enabled bucket="ded5cceb-large-object-no-filters-src" versioning_enabled=true
2023-01-05T09:21:17.157382Z DEBUG ThreadId(556) ssstar::create: Confirmed bucket access for input url=s3://ded5cceb-large-object-no-filters-src/ bucket=S3Bucket { name: "ded5cceb-large-object-no-filters-src", versioning_enabled: true, region: None, client: "<...>" }
2023-01-05T09:21:17.157427Z DEBUG ThreadId(556) ssstar::create: Adding archive input to job input=CreateArchiveInput { bucket: S3Bucket { name: "ded5cceb-large-object-no-filters-src", versioning_enabled: true, region: None, client: "<...>" }, selector: Bucket }
2023-01-05T09:21:17.157478Z DEBUG ThreadId(556) ssstar::create: Listing objects for all inputs input_count=1
2023-01-05T09:21:17.157571Z DEBUG ThreadId(556) into_input_objects: ssstar::create: Listing all object store objects that match this archive input self=Bucket ded5cceb-large-object-no-filters-src
2023-01-05T09:21:17.157619Z DEBUG ThreadId(556) into_input_objects: ssstar::objstore::s3: Archive input matches everything in an S3 bucket bucket=ded5cceb-large-object-no-filters-src
2023-01-05T09:21:17.163252Z DEBUG ThreadId(556) into_input_objects: ssstar::objstore::s3: Matched all S3 objects in an S3 bucket input_objects=1 bucket=ded5cceb-large-object-no-filters-src
2023-01-05T09:21:17.163325Z DEBUG ThreadId(556) into_input_objects: ssstar::create: Listing matching objects completed self=Bucket ded5cceb-large-object-no-filters-src count=1
2023-01-05T09:21:17.163441Z DEBUG ThreadId(556) ssstar::create: Listed all objects in all inputs object_count=1
2023-01-05T09:21:17.166492Z DEBUG ThreadId(556) ssstar::objstore::s3: Validating access to bucket bucket="ded5cceb-large-object-no-filters-src"
2023-01-05T09:21:17.168891Z DEBUG ThreadId(556) ssstar::objstore::s3: Access to bucket is confirmed bucket="ded5cceb-large-object-no-filters-src" region=None
2023-01-05T09:21:17.176675Z DEBUG ThreadId(556) ssstar::objstore::s3: Checked if bucket versioning is enabled bucket="ded5cceb-large-object-no-filters-src" versioning_enabled=true
2023-01-05T09:21:17.179524Z DEBUG ThreadId(566) multipart_object_writer{key="test.tar" upload_id="YjU1YWQ4ZDMtMWY3NS00ZDgyLWJkNGYtOWQxZDFiM2VlNDhmLjM2M2FlNGM5LTBjMGQtNGIyOS1hMzAwLTY2MjM4ZmI2MTU4Zg" bucket=ded5cceb-large-object-no-filters-src}: ssstar::objstore::s3: Commencing multi-part upload
2023-01-05T09:21:17.179735Z DEBUG ThreadId(566) read_object_part{key="test" version_id=Some("f228b958-18ed-4b32-bdd0-1b6e5170df81") byte_range=0..8388608}: ssstar::objstore::s3: Reading partial object
2023-01-05T09:21:17.180655Z DEBUG ThreadId(566) read_object_part{key="test" version_id=Some("f228b958-18ed-4b32-bdd0-1b6e5170df81") byte_range=8388608..16777216}: ssstar::objstore::s3: Reading partial object
2023-01-05T09:21:17.181787Z DEBUG ThreadId(566) read_object_part{key="test" version_id=Some("f228b958-18ed-4b32-bdd0-1b6e5170df81") byte_range=16777216..20971520}: ssstar::objstore::s3: Reading partial object
2023-01-05T09:21:17.226781Z DEBUG ThreadId(556) ssstar::create: Reading object and writing to tar archive key=test size=20971520
2023-01-05T09:21:17.283245Z DEBUG ThreadId(560) multipart_object_writer{key="test.tar" upload_id="YjU1YWQ4ZDMtMWY3NS00ZDgyLWJkNGYtOWQxZDFiM2VlNDhmLjM2M2FlNGM5LTBjMGQtNGIyOS1hMzAwLTY2MjM4ZmI2MTU4Zg" bucket=ded5cceb-large-object-no-filters-src}:upload chunk{part_number=1 chunk_size=8388608}: ssstar::objstore::s3: Uploading multi-part chunk
2023-01-05T09:21:17.285523Z DEBUG ThreadId(556) ssstar::create: Streamed object data into tar archive key=test size=20971520 final_part_number=2
2023-01-05T09:21:17.285651Z DEBUG ThreadId(556) ssstar::create: Completed processing of all input objects
2023-01-05T09:21:17.296724Z DEBUG ThreadId(562) ssstar::writers: encountered end of duplex stream; worker task exiting
2023-01-05T09:21:18.159232Z DEBUG ThreadId(560) multipart_object_writer{key="test.tar" upload_id="YjU1YWQ4ZDMtMWY3NS00ZDgyLWJkNGYtOWQxZDFiM2VlNDhmLjM2M2FlNGM5LTBjMGQtNGIyOS1hMzAwLTY2MjM4ZmI2MTU4Zg" bucket=ded5cceb-large-object-no-filters-src}:upload chunk{part_number=2 chunk_size=8388608}: ssstar::objstore::s3: Uploading multi-part chunk
2023-01-05T09:21:18.758224Z DEBUG ThreadId(566) multipart_object_writer{key="test.tar" upload_id="YjU1YWQ4ZDMtMWY3NS00ZDgyLWJkNGYtOWQxZDFiM2VlNDhmLjM2M2FlNGM5LTBjMGQtNGIyOS1hMzAwLTY2MjM4ZmI2MTU4Zg" bucket=ded5cceb-large-object-no-filters-src}:upload chunk{part_number=3 chunk_size=4195840}: ssstar::objstore::s3: Uploading multi-part chunk
2023-01-05T09:21:19.214740Z DEBUG ThreadId(566) multipart_object_writer{key="test.tar" upload_id="YjU1YWQ4ZDMtMWY3NS00ZDgyLWJkNGYtOWQxZDFiM2VlNDhmLjM2M2FlNGM5LTBjMGQtNGIyOS1hMzAwLTY2MjM4ZmI2MTU4Zg" bucket=ded5cceb-large-object-no-filters-src}:upload chunk{part_number=1 chunk_size=8388608}: ssstar::objstore::s3: Uploaded multi-part chunk e_tag="62596a832fdc3e1a9d29eac2b9019047" sha256="nhSj9N6KuiJsewqRFvcpj2sGeKqNFFfVu+/MQrIMp9o="
2023-01-05T09:21:19.263943Z DEBUG ThreadId(562) multipart_object_writer{key="test.tar" upload_id="YjU1YWQ4ZDMtMWY3NS00ZDgyLWJkNGYtOWQxZDFiM2VlNDhmLjM2M2FlNGM5LTBjMGQtNGIyOS1hMzAwLTY2MjM4ZmI2MTU4Zg" bucket=ded5cceb-large-object-no-filters-src}:upload chunk{part_number=3 chunk_size=4195840}: ssstar::objstore::s3: Uploaded multi-part chunk e_tag="2ad9ddd548ce51682b7003a13df3e286" sha256="TapScgUQhSanZi0PUIr2rRxorZZOr1LKRQb+hD6k4wA="
2023-01-05T09:21:19.306664Z DEBUG ThreadId(562) multipart_object_writer{key="test.tar" upload_id="YjU1YWQ4ZDMtMWY3NS00ZDgyLWJkNGYtOWQxZDFiM2VlNDhmLjM2M2FlNGM5LTBjMGQtNGIyOS1hMzAwLTY2MjM4ZmI2MTU4Zg" bucket=ded5cceb-large-object-no-filters-src}:upload chunk{part_number=2 chunk_size=8388608}: ssstar::objstore::s3: Uploaded multi-part chunk e_tag="0174f95a2087685991564d6220a77ce0" sha256="sddxXHtLFi0hzTTyH+QyGW5jWuFEQwkRFoy//bJBUDU="
2023-01-05T09:21:19.306798Z DEBUG ThreadId(562) multipart_object_writer{key="test.tar" upload_id="YjU1YWQ4ZDMtMWY3NS00ZDgyLWJkNGYtOWQxZDFiM2VlNDhmLjM2M2FlNGM5LTBjMGQtNGIyOS1hMzAwLTY2MjM4ZmI2MTU4Zg" bucket=ded5cceb-large-object-no-filters-src}: ssstar::objstore::s3: All parts uploaded; completing multi-part upload total_parts=3 total_bytes=20973056
2023-01-05T09:21:19.311583Z DEBUG ThreadId(556) ssstar::create: Upload of tar archive to object storage completed bytes_written=20973056
2023-01-05T09:21:19.314992Z DEBUG ThreadId(556) ssstar::objstore::s3: Validating access to bucket bucket="417610ee-large-object-no-filters-tgt"
2023-01-05T09:21:19.316626Z DEBUG ThreadId(556) ssstar::objstore::s3: Access to bucket is confirmed bucket="417610ee-large-object-no-filters-tgt" region=None
2023-01-05T09:21:19.317815Z DEBUG ThreadId(556) ssstar::objstore::s3: Checked if bucket versioning is enabled bucket="417610ee-large-object-no-filters-tgt" versioning_enabled=true
2023-01-05T09:21:19.319402Z DEBUG ThreadId(556) into_internal{self=ObjectStorage(Url { scheme: "s3", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("ded5cceb-large-object-no-filters-src")), port: None, path: "/test.tar", query: None, fragment: None })}: ssstar::objstore::s3: Validating access to bucket bucket="ded5cceb-large-object-no-filters-src"
2023-01-05T09:21:19.320787Z DEBUG ThreadId(556) into_internal{self=ObjectStorage(Url { scheme: "s3", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("ded5cceb-large-object-no-filters-src")), port: None, path: "/test.tar", query: None, fragment: None })}: ssstar::objstore::s3: Access to bucket is confirmed bucket="ded5cceb-large-object-no-filters-src" region=None
2023-01-05T09:21:19.321993Z DEBUG ThreadId(556) into_internal{self=ObjectStorage(Url { scheme: "s3", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("ded5cceb-large-object-no-filters-src")), port: None, path: "/test.tar", query: None, fragment: None })}: ssstar::objstore::s3: Checked if bucket versioning is enabled bucket="ded5cceb-large-object-no-filters-src" versioning_enabled=true
2023-01-05T09:21:19.323489Z  INFO ThreadId(556) run{source_archive=ObjectStorage { bucket: "ded5cceb-large-object-no-filters-src", key: "test.tar", version_id: None, len: 20973056 } target_bucket="417610ee-large-object-no-filters-tgt" target_prefix=}: ssstar::extract: Starting extract archive job self.filters=[] self.archive_size=Some(20973056)
2023-01-05T09:21:19.323585Z DEBUG ThreadId(556) run{source_archive=ObjectStorage { bucket: "ded5cceb-large-object-no-filters-src", key: "test.tar", version_id: None, len: 20973056 } target_bucket="417610ee-large-object-no-filters-tgt" target_prefix=}:read_object{key="test.tar" version_id=None byte_range=0..20973056}: ssstar::objstore::s3: Reading object
2023-01-05T09:21:19.323714Z DEBUG ThreadId(566) ssstar::extract: Starting tar entry processor task
2023-01-05T09:21:19.323744Z DEBUG ThreadId(664) run{source_archive=ObjectStorage { bucket: "ded5cceb-large-object-no-filters-src", key: "test.tar", version_id: None, len: 20973056 } target_bucket="417610ee-large-object-no-filters-tgt" target_prefix=}:read_tar_entries_blocking{source_archive=ObjectStorage { bucket: "ded5cceb-large-object-no-filters-src", key: "test.tar", version_id: None, len: 20973056 }}: ssstar::extract: Starting blocking tar read task
2023-01-05T09:21:19.323767Z DEBUG ThreadId(561) read_object_part{key="test.tar" version_id=None byte_range=0..8388608}: ssstar::objstore::s3: Reading partial object
2023-01-05T09:21:19.324520Z DEBUG ThreadId(561) read_object_part{key="test.tar" version_id=None byte_range=8388608..16777216}: ssstar::objstore::s3: Reading partial object
2023-01-05T09:21:19.325275Z DEBUG ThreadId(561) read_object_part{key="test.tar" version_id=None byte_range=16777216..20973056}: ssstar::objstore::s3: Reading partial object
2023-01-05T09:21:19.341362Z DEBUG ThreadId(664) run{source_archive=ObjectStorage { bucket: "ded5cceb-large-object-no-filters-src", key: "test.tar", version_id: None, len: 20973056 } target_bucket="417610ee-large-object-no-filters-tgt" target_prefix=}:read_tar_entries_blocking{source_archive=ObjectStorage { bucket: "ded5cceb-large-object-no-filters-src", key: "test.tar", version_id: None, len: 20973056 }}:Processing tar entry{path=test len=20971520}: ssstar::extract: No filters are applied so this entry is included
2023-01-05T09:21:19.341442Z DEBUG ThreadId(664) run{source_archive=ObjectStorage { bucket: "ded5cceb-large-object-no-filters-src", key: "test.tar", version_id: None, len: 20973056 } target_bucket="417610ee-large-object-no-filters-tgt" target_prefix=}:read_tar_entries_blocking{source_archive=ObjectStorage { bucket: "ded5cceb-large-object-no-filters-src", key: "test.tar", version_id: None, len: 20973056 }}:Processing tar entry{path=test len=20971520}: ssstar::extract: Processing entry as multi-part
2023-01-05T09:21:19.341592Z DEBUG ThreadId(562) process_tar_entries{target_prefix="" target_bucket="417610ee-large-object-no-filters-tgt"}: ssstar::extract: Starting multipart file upload path=test len=20971520 key=test
2023-01-05T09:21:19.344156Z DEBUG ThreadId(568) multipart_object_writer{key="test" upload_id="YjU1YWQ4ZDMtMWY3NS00ZDgyLWJkNGYtOWQxZDFiM2VlNDhmLjEyODkyMzY1LWYwMjEtNDFlZC04MDMyLWYyOGUwMDgyNDgzNg" bucket=417610ee-large-object-no-filters-tgt}: ssstar::objstore::s3: Commencing multi-part upload
2023-01-05T09:21:19.345256Z DEBUG ThreadId(561) ssstar::objstore::s3: Read object async task exiting key=test.tar
2023-01-05T09:21:19.345324Z DEBUG ThreadId(568) process_tar_entries{target_prefix="" target_bucket="417610ee-large-object-no-filters-tgt"}: ssstar::extract: Uploading file part path=test len=8388608
2023-01-05T09:21:19.348521Z DEBUG ThreadId(568) multipart_object_writer{key="test" upload_id="YjU1YWQ4ZDMtMWY3NS00ZDgyLWJkNGYtOWQxZDFiM2VlNDhmLjEyODkyMzY1LWYwMjEtNDFlZC04MDMyLWYyOGUwMDgyNDgzNg" bucket=417610ee-large-object-no-filters-tgt}:upload chunk{part_number=1 chunk_size=8388608}: ssstar::objstore::s3: Uploading multi-part chunk
2023-01-05T09:21:19.349002Z DEBUG ThreadId(561) process_tar_entries{target_prefix="" target_bucket="417610ee-large-object-no-filters-tgt"}: ssstar::extract: Uploading file part path=test len=8388608
2023-01-05T09:21:19.349793Z DEBUG ThreadId(664) run{source_archive=ObjectStorage { bucket: "ded5cceb-large-object-no-filters-src", key: "test.tar", version_id: None, len: 20973056 } target_bucket="417610ee-large-object-no-filters-tgt" target_prefix=}:read_tar_entries_blocking{source_archive=ObjectStorage { bucket: "ded5cceb-large-object-no-filters-src", key: "test.tar", version_id: None, len: 20973056 }}: ssstar::extract: Completed processing all tar entries
2023-01-05T09:21:19.349862Z DEBUG ThreadId(664) run{source_archive=ObjectStorage { bucket: "ded5cceb-large-object-no-filters-src", key: "test.tar", version_id: None, len: 20973056 } target_bucket="417610ee-large-object-no-filters-tgt" target_prefix=}:read_tar_entries_blocking{source_archive=ObjectStorage { bucket: "ded5cceb-large-object-no-filters-src", key: "test.tar", version_id: None, len: 20973056 }}: ssstar::extract: Blocking tar read task completing successfully
2023-01-05T09:21:19.349928Z DEBUG ThreadId(561) process_tar_entries{target_prefix="" target_bucket="417610ee-large-object-no-filters-tgt"}: ssstar::extract: Uploading final file part path=test len=4194304
2023-01-05T09:21:19.353761Z DEBUG ThreadId(561) ssstar::writers: encountered end of duplex stream; worker task exiting
2023-01-05T09:21:19.873379Z DEBUG ThreadId(561) multipart_object_writer{key="test" upload_id="YjU1YWQ4ZDMtMWY3NS00ZDgyLWJkNGYtOWQxZDFiM2VlNDhmLjEyODkyMzY1LWYwMjEtNDFlZC04MDMyLWYyOGUwMDgyNDgzNg" bucket=417610ee-large-object-no-filters-tgt}:upload chunk{part_number=2 chunk_size=8388608}: ssstar::objstore::s3: Uploading multi-part chunk
2023-01-05T09:21:20.391471Z DEBUG ThreadId(561) multipart_object_writer{key="test" upload_id="YjU1YWQ4ZDMtMWY3NS00ZDgyLWJkNGYtOWQxZDFiM2VlNDhmLjEyODkyMzY1LWYwMjEtNDFlZC04MDMyLWYyOGUwMDgyNDgzNg" bucket=417610ee-large-object-no-filters-tgt}:upload chunk{part_number=3 chunk_size=4194304}: ssstar::objstore::s3: Uploading multi-part chunk
2023-01-05T09:21:20.649335Z DEBUG ThreadId(561) multipart_object_writer{key="test" upload_id="YjU1YWQ4ZDMtMWY3NS00ZDgyLWJkNGYtOWQxZDFiM2VlNDhmLjEyODkyMzY1LWYwMjEtNDFlZC04MDMyLWYyOGUwMDgyNDgzNg" bucket=417610ee-large-object-no-filters-tgt}:upload chunk{part_number=1 chunk_size=8388608}: ssstar::objstore::s3: Uploaded multi-part chunk e_tag="cd9d737be43363631fd63930e4364265" sha256="HXrcPJax2dkdkfXigZCkE7FvLgnApUiz4Cojq17oHYc="
2023-01-05T09:21:20.681648Z DEBUG ThreadId(562) multipart_object_writer{key="test" upload_id="YjU1YWQ4ZDMtMWY3NS00ZDgyLWJkNGYtOWQxZDFiM2VlNDhmLjEyODkyMzY1LWYwMjEtNDFlZC04MDMyLWYyOGUwMDgyNDgzNg" bucket=417610ee-large-object-no-filters-tgt}:upload chunk{part_number=3 chunk_size=4194304}: ssstar::objstore::s3: Uploaded multi-part chunk e_tag="f8a486ac6fab5d26803408266cfcb779" sha256="x0uxBaO+Kqe3TEYGUlIsYkzNDcR1N3+taaSy/YtJmfo="
2023-01-05T09:21:20.753750Z DEBUG ThreadId(562) multipart_object_writer{key="test" upload_id="YjU1YWQ4ZDMtMWY3NS00ZDgyLWJkNGYtOWQxZDFiM2VlNDhmLjEyODkyMzY1LWYwMjEtNDFlZC04MDMyLWYyOGUwMDgyNDgzNg" bucket=417610ee-large-object-no-filters-tgt}:upload chunk{part_number=2 chunk_size=8388608}: ssstar::objstore::s3: Uploaded multi-part chunk e_tag="3bfb5b5af065a014d1e45206ac07b0b9" sha256="tS094WSkFca1apZJ5pvEFGRm6awW7YBHUR92WX+z7s8="
2023-01-05T09:21:20.753885Z DEBUG ThreadId(562) multipart_object_writer{key="test" upload_id="YjU1YWQ4ZDMtMWY3NS00ZDgyLWJkNGYtOWQxZDFiM2VlNDhmLjEyODkyMzY1LWYwMjEtNDFlZC04MDMyLWYyOGUwMDgyNDgzNg" bucket=417610ee-large-object-no-filters-tgt}: ssstar::objstore::s3: All parts uploaded; completing multi-part upload total_parts=3 total_bytes=20971520
2023-01-05T09:21:20.758557Z DEBUG ThreadId(560) process_tar_entries{target_prefix="" target_bucket="417610ee-large-object-no-filters-tgt"}: ssstar::extract: Finished multipart file upload path=test len=20971520 key=test
2023-01-05T09:21:20.758630Z DEBUG ThreadId(560) process_tar_entries{target_prefix="" target_bucket="417610ee-large-object-no-filters-tgt"}: ssstar::extract: Entry sender dropped; no more tar entries to process
2023-01-05T09:21:20.758751Z DEBUG ThreadId(560) ssstar::extract: Tar entry processor task completed successfully
2023-01-05T09:21:20.758826Z  INFO ThreadId(556) run{source_archive=ObjectStorage { bucket: "ded5cceb-large-object-no-filters-src", key: "test.tar", version_id: None, len: 20973056 } target_bucket="417610ee-large-object-no-filters-tgt" target_prefix=}: ssstar::extract: Finished extract job



failures:
    objstore::minio::extract::large_object_no_filters::extract_from_archive_file_to_bucket_root
    objstore::minio::extract::large_object_no_filters::extract_from_archive_on_bucket_to_bucket_root
    objstore::minio::extract::large_object_no_filters::extract_from_archive_reader_to_bucket_root

test result: FAILED. 39 passed; 3 failed; 3 ignored; 0 measured; 0 filtered out; finished in 16.52s

core@core-MSI:/media/core/kingston-luks/ssstar$

Would you like to work on a fix? [y/n]

To Reproduce

I just ran tests as is shown in CONTRIBUTING

Expected behavior

All tests have to be passed

Screenshots

Environment

I have installed minio by steps from https://unixcop.com/install-minio-ubuntu/

OS info:

core@core-MSI:/media/core/kingston-luks/ssstar$ cat /etc/upstream-release/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu Jammy Jellyfish"

It is on the latest master commit ff689cd

Additional context

This problem is only in tests, nothing related to production code

ssstar create performance is not optimal on big instances

Version: ssstar-cli 0.2.0 (target x86_64-unknown-linux-gnu)

Testing results:

ssstar create performance is not scaling with instance size scale up.

S3 creating archive panics with empty folder

Bug description

It panics if I pass empty folder to create an archive

The application panicked (crashed).
Message:  attempt to subtract with overflow
Location: /home/core/.cargo/registry/src/github.com-1ecc6299db9ec823/ssstar-0.3.0/src/objstore/s3.rs:940

Environment

core@core-MSI:/media/core/kingston-luks/ssstar$ cat /etc/upstream-release/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu Jammy Jellyfish"

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.