Comments (3)
MPizzotti
commented on June 26, 2024
probably this issue is related to the same issue reported here (issue 11845)
from duckdb.
szarnyasg
commented on June 26, 2024
Hi @dforsber, thanks for raising this issue. The S3 bucket is set to private, so I get a 403 error - can you please share the demo.duckdb file with us?
from duckdb.
dforsber
commented on June 26, 2024
You can download the file from here (zstd compressed):
https://isecure.fi/demo.duckdb.zst
I actually found more information about this issue. It seems to happen only when I first try to rely on the existing ~/.duckdbrc S3 credentials (see below), which does not work and I get HTTP 403. Then I use CREATE SECRET and it allows me to mount the db, but fails to read.
If I use the CREATE SECRET directly, then it works.
-- Loading resources from /Users/dforsber/.duckdbrc
v0.10.2 1601d94f94
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
D ATTACH 's3://boilingdata-demo/demo.duckdb' AS demos3 (READ_ONLY);
HTTP Error: HTTP GET error on 'https://boilingdata-demo.s3.amazonaws.com/demo.duckdb' (HTTP 403)
D CREATE SECRET (
TYPE S3,
PROVIDER CREDENTIAL_CHAIN
);
┌─────────┐
│ Success │
│ boolean │
├─────────┤
│ true │
└─────────┘
D ATTACH 's3://boilingdata-demo/demo.duckdb' AS demos3 (READ_ONLY);
D use demos3;
D SELECT * FROM demo LIMIT 10;
[1] 55288 segmentation fault duckdb
dforsber@MacBook-Pro ➜ duckdb
-- Loading resources from /Users/dforsber/.duckdbrc
v0.10.2 1601d94f94
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
D CREATE SECRET (
TYPE S3,
PROVIDER CREDENTIAL_CHAIN
);
┌─────────┐
│ Success │
│ boolean │
├─────────┤
│ true │
└─────────┘
D ATTACH 's3://boilingdata-demo/demo.duckdb' AS demos3 (READ_ONLY);
D use demos3;
D SELECT * FROM demo LIMIT 10;
100% ▕████████████████████████████████████████████████████████████▏
┌──────────┬──────────────────────────┬──────────────────────────┬─────────────────┬───────────────┬────────────┬────────────────────┬──────────────┬──────────────┬──────────────┬─────────────┬────────┬─────────┬────────────┬──────────────┬───────────────────────┬──────────────┬──────────────────────┐
│ VendorID │ tpep_pickup_datetime │ tpep_dropoff_datetime │ passenger_count │ trip_distance │ RatecodeID │ store_and_fwd_flag │ PULocationID │ DOLocationID │ payment_type │ fare_amount │ extra │ mta_tax │ tip_amount │ tolls_amount │ improvement_surcharge │ total_amount │ congestion_surcharge │
│ int32 │ timestamp with time zone │ timestamp with time zone │ int32 │ double │ int32 │ varchar │ int32 │ int32 │ int32 │ double │ double │ double │ double │ double │ double │ double │ double │
├──────────┼──────────────────────────┼──────────────────────────┼─────────────────┼───────────────┼────────────┼────────────────────┼──────────────┼──────────────┼──────────────┼─────────────┼────────┼─────────┼────────────┼──────────────┼───────────────────────┼──────────────┼──────────────────────┤
│ 1 │ 2019-03-01 02:00:00+02 │ 2019-03-01 02:07:54+02 │ 1 │ 1.6 │ 1 │ N │ 113 │ 137 │ 1 │ 7.5 │ 3.0 │ 0.5 │ 2.25 │ 0.0 │ 0.3 │ 13.55 │ 2.5 │
│ 1 │ 2019-03-01 02:00:00+02 │ 2019-03-01 02:14:44+02 │ 2 │ 3.0 │ 1 │ N │ 186 │ 140 │ 1 │ 12.5 │ 3.0 │ 0.5 │ 3.25 │ 0.0 │ 0.3 │ 19.55 │ 2.5 │
│ 1 │ 2019-03-01 02:00:01+02 │ 2019-03-01 02:05:49+02 │ 1 │ 2.2 │ 1 │ N │ 87 │ 33 │ 1 │ 8.0 │ 3.0 │ 0.5 │ 2.35 │ 0.0 │ 0.3 │ 14.15 │ 2.5 │
│ 1 │ 2019-03-01 02:00:02+02 │ 2019-03-01 02:03:19+02 │ 2 │ 0.7 │ 1 │ N │ 249 │ 79 │ 1 │ 4.5 │ 3.0 │ 0.5 │ 1.65 │ 0.0 │ 0.3 │ 9.95 │ 2.5 │
│ 1 │ 2019-03-01 02:00:03+02 │ 2019-03-01 02:03:09+02 │ 1 │ 0.6 │ 1 │ N │ 166 │ 151 │ 2 │ 4.5 │ 0.5 │ 0.5 │ 0.0 │ 0.0 │ 0.3 │ 5.8 │ 0.0 │
│ 1 │ 2019-03-01 02:00:03+02 │ 2019-03-01 02:15:39+02 │ 1 │ 4.2 │ 1 │ N │ 13 │ 100 │ 1 │ 15.5 │ 3.0 │ 0.5 │ 3.85 │ 0.0 │ 0.3 │ 23.15 │ 2.5 │
│ 1 │ 2019-03-01 02:00:03+02 │ 2019-03-01 02:43:11+02 │ 1 │ 9.9 │ 1 │ N │ 90 │ 226 │ 1 │ 36.5 │ 3.0 │ 0.5 │ 1.0 │ 0.0 │ 0.3 │ 41.3 │ 2.5 │
│ 1 │ 2019-03-01 02:00:05+02 │ 2019-03-01 02:03:15+02 │ 1 │ 0.6 │ 1 │ N │ 186 │ 90 │ 1 │ 4.5 │ 3.0 │ 0.5 │ 1.0 │ 0.0 │ 0.3 │ 9.3 │ 2.5 │
│ 1 │ 2019-03-01 02:00:05+02 │ 2019-03-01 02:07:20+02 │ 1 │ 1.7 │ 1 │ N │ 48 │ 239 │ 1 │ 7.5 │ 3.0 │ 0.5 │ 1.0 │ 0.0 │ 0.3 │ 12.3 │ 2.5 │
│ 1 │ 2019-03-01 02:00:05+02 │ 2019-03-01 02:24:01+02 │ 1 │ 3.3 │ 1 │ N │ 113 │ 256 │ 1 │ 17.5 │ 3.0 │ 0.5 │ 4.25 │ 0.0 │ 0.3 │ 25.55 │ 2.5 │
├──────────┴──────────────────────────┴──────────────────────────┴─────────────────┴───────────────┴────────────┴────────────────────┴──────────────┴──────────────┴──────────────┴─────────────┴────────┴─────────┴────────────┴──────────────┴───────────────────────┴──────────────┴──────────────────────┤
│ 10 rows 18 columns │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
D
And I have some settings already in place on ~/.duckdbrc.
% cat ~/.duckdbrc
SET enable_http_metadata_cache = true;
SET enable_object_cache = true;
SET s3_access_key_id='...';
SET s3_secret_access_key='...';
SET s3_region='eu-west-1';
...from duckdb.
Related Issues (20)
- Python: Segmentation Fault when importing duckdb on x86 running MacOS 11 HOT 16
- `duckdb.query` cannot find DataFrames from python global environment HOT 2
- `SUMMARIZE` returns `NULL` in quantile columns for `TIMESTAMP` types
- Cannot declare a CTE inside a Recursive CTE HOT 1
- strptime function not following Posix standard for 2 digit years HOT 2
- [regression in 0.10.3] error: non-virtual member function marked 'override' hides virtual member function HOT 2
- predicate incorrectly pushed into ASOF JOIN
- :bug: `v0.10.3` Regression while adding a new column to a table "because there are entries that depend on it." HOT 4
- Hive partitioned append/write 0.10.3 HOT 3
- fail to query from a Pandas data frame named "df" HOT 2
- Possible downtime on DuckDB Extensions Repository HOT 1
- Empty copy operations does not close the multipart file upload handle
- Unable to COPY to remote Hugging Face dataset repo HOT 2
- View dependency check bypassed on db reload HOT 5
- Version 0.10.3 cannot recognize the pd.DataFrame with the variable name 'df' HOT 1
- Error after reading remote json file HOT 3
- Too many files generated when setting temp_directory property in in-memory mode HOT 5
- [C-API] duckdb_register_table_function terminates with unrecoverable error when registered with a conflicting name
- ARRAY_VALUE(0.5, 0.4, 0.1) --> Unimplemented type "DECIMAL(2,1)[3]"
- Error parsing dataframe from variable name in 0.10.3 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from duckdb.