Cannot read data from a GCS Foreign Table about paradedb HOT 8 OPEN

Hi @parthspatel-mendel, what version of pg_lakehouse are you running?

NOTE: @philippemnoel perhaps a more specific GitHub issue template could be helpful to make sure the environment details are included?

I think the Tokio runtime error could be fixed via #1208 but it's best if the ParadeDB team can confirm, it might be required to restart the postgres service to avoid failures after the crash happens.

OTOH regarding configuration for GCS, this worked for me, please see the docs though as there might be some configuration params that you wish to adapt for your own case https://docs.paradedb.com/analytics/object_stores/gcs:

1. Create the Foreign Data Wrapper

CREATE FOREIGN DATA WRAPPER gcs_fdw
HANDLER gcs_fdw_handler
VALIDATOR gcs_fdw_validator;

2. Create the server using the FDW we just created

In the CREATE SERVER statement we will include the Google Storage API endpoint (https://storage.googleapis.com)

CREATE SERVER gcs_server FOREIGN DATA WRAPPER gcs_fdw
OPTIONS (endpoint 'https://storage.googleapis.com', default_storage_class 'STANDARD', predefined_acl 'authenticatedRead');

3. Create the user mapping for authentication using a Service Account key file that is accessible from the container where postgres is running.

CREATE USER MAPPING FOR postgres
SERVER gcs_server
OPTIONS (
  credential '/path/to/service_account_key_file.json'
);

4. Create the foreign table

Note that unlike in other query engines like duckdb, for a partitioned parquet file you should leave the folder path (ending in '/') instead of specifying '/*.parquet'

CREATE FOREIGN TABLE foreign_table_name ("column_1" dtype, "column_2" dtype ...)
SERVER gcs_server
OPTIONS (path 'gs://bucket_name/bucket_folder/bucket_subfolder_where_parquet_files_are/', extension 'parquet');

from paradedb.

@Mmoncadaisla thank you for pitching in to help here, that is much appreciated.

@parthspatel-mendel, #1224 addresses the tokio executor crash that occurs when there's a connection problem with an object store. I was finding that bad configuration (or bad internet connection etc) would crash the executor and cause there is no reactor running.

It's possible that you were seeing this symptom from a configuration problem. We'll release the executor fix shortly, but in the meantime please try out @Mmoncadaisla's steps to see if it is indeed a GCS config issue.

from paradedb.

@neilyio I was using the docker image from the docs

docker run \
  --name paradedb \
  -e POSTGRESQL_USERNAME=abcd \
  -e POSTGRESQL_PASSWORD=abcd \
  -e POSTGRESQL_DATABASE=abcd \
  -e POSTGRESQL_POSTGRES_PASSWORD=abcd \
  -v paradedb_data:/bitnami/postgresql \
  -p 5432:5432 \
  -v ${HOME}:${HOME} \
  -d \
paradedb/paradedb:latest

from paradedb.

@neilyio I was using the docker image from the docs

docker run \
  --name paradedb \
  -e POSTGRESQL_USERNAME=abcd \
  -e POSTGRESQL_PASSWORD=abcd \
  -e POSTGRESQL_DATABASE=abcd \
  -e POSTGRESQL_POSTGRES_PASSWORD=abcd \
  -v paradedb_data:/bitnami/postgresql \
  -p 5432:5432 \
  -v ${HOME}:${HOME} \
  -d \
paradedb/paradedb:latest

@parthspatel-mendel could you try again with v0.7.5 that we've released?

from paradedb.