Comments (1)
mildbyte
commented on May 29, 2024
This looks like a disparity between object_store and the official Minio SDK. object_store uses these inputs to build a signature:
string_to_sign = "AWS4-HMAC-SHA256\n20220915T105705Z\n20220915//s3/aws4_request\nb1f63c698039ba6772a850c3f0df8dc7e4f544854197ff42ff46f8f0c45e9e0d"
scope = "20220915//s3/aws4_request"
hashed_canonical_request = "b1f63c698039ba6772a850c3f0df8dc7e4f544854197ff42ff46f8f0c45e9e0d"
canonical_request = "POST\n/seafowl/bc444215ffac9df867e0b9a26184e67df061dd50a1fd2de3d3f3e335f1f85de3.parquet\nuploads\nhost:localhost:9000\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20220915T105705Z\n\nhost;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
signed_headers = "host;x-amz-content-sha256;x-amz-date"
canonical_headers = "host:localhost:9000\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20220915T105705Z\n"
header_digest = http::header::value::HeaderValue {inner: bytes::bytes::Bytes {ptr: 0x7fffec05c4d0 "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\031\f\375\020eĻ\027q\306\f\000", len: 64, data: core::sync::atomic::AtomicPtr<()> {p: core::cell::UnsafeCell<*mut ()> {value: 0x7fffec05c4d1}}, vtable: 0x55555bfc2fe0 <bytes::bytes::PROMOTABLE_EVEN_VTABLE>}, is_sensitive: false}
digest = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
date_val = http::header::value::HeaderValue {inner: bytes::bytes::Bytes {ptr: 0x7fffec47d5b0 "20220915T105705Z:9000\177\000", len: 16, data: core::sync::atomic::AtomicPtr<()> {p: core::cell::UnsafeCell<*mut ()> {value: 0x7fffec47d5b1}}, vtable: 0x55555bfc2fe0 <bytes::bytes::PROMOTABLE_EVEN_VTABLE>}, is_sensitive: false}
date_str = "20220915T105705Z"
host_val = http::header::value::HeaderValue {inner: bytes::bytes::Bytes {ptr: 0x7fffec47d590 "localhost:9000\000", len: 14, data: core::sync::atomic::AtomicPtr<()> {p: core::cell::UnsafeCell<*mut ()> {value: 0x7fffec47d591}}, vtable: 0x55555bfc2fe0 <bytes::bytes::PROMOTABLE_EVEN_VTABLE>}, is_sensitive: false}
This is the main part:
canonical_request = "POST\n/seafowl/bc444215ffac9df867e0b9a26184e67df061dd50a1fd2de3d3f3e335f1f85de3.parquet\nuploads\nhost:localhost:9000\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20220915T105705Z\n\nhost;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
(note the query string is uploads)
When using the official Minio SDK to perform a multipart upload:
from io import BytesIO
import minio
from minio.helpers import MIN_PART_SIZE
client = minio.Minio("localhost:9000", "minioadmin", "minioadmin", secure=False)
data = BytesIO(b"test" * 10000000)
client.put_object(
bucket_name="seafowl",
object_name="test_multipart",
data=data,
part_size=MIN_PART_SIZE,
length=40000000,
)we get these inputs (note that this is a different file, I was just looking for discrepancies between what goes into the signature):
There are two differences here:
- the region isn't empty (us-east-1) in minio-py whereas it is in the object_store (because we make it empty)
- the URL to initiate a multipart upload has a
?uploads=query param in minio-py instead of?uploads(object_store)
Setting the region to us-east in object_store doesn't fix this, but changing object_store to use ?uploads= instead of ?uploads does. I haven't looked at the Minio signature verification code itself to see if we should keep the URL the same and change the object_store signing code instead.
from seafowl.
Related Issues (20)
- Fix SQLite race condition causing new DBs/schemas to not be immediately available after a HTTP 200 HOT 1
- Parsing fails if `-- comment` is included inside arguments of function call (but succeeds with `/* comment */`) HOT 1
- Seafowl 0.4 migration instructions
- Result JSON unsupported types
- Caching queries HOT 2
- CREATE EXTERNAL TABLE with empty Parquet file fails with `Invalid data type for Delta Lake: Null` HOT 7
- feat: iceberg support HOT 1
- `datafusion-remote-tables` crate HOT 4
- Remote tables: filter without sort HOT 2
- rust s3 HOT 1
- Implement a Seafowl CLI
- Implement conversion of parquet to Delta tables
- Add support for object stores with specific paths
- chore: Refactor SeafowlContext
- Figure out what to do with `table_column` catalog table and bulk schema loading in general HOT 1
- Add support for partitioned tables
- Add Arrow Flight frontend
- Implement eager dropping of tables and schemas
- cassandra or scylladb support
- Enable external metastore implementations
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
š Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ššš
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ā¤ļø Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from seafowl.