Giter VIP home page Giter VIP logo

Comments (2)

geekypilot avatar geekypilot commented on June 25, 2024

I'm able to reproduce the issue as well on Databricks + Azure Data Lake Storage Gen2 storage mounted to /mnt/REDACTED.

from duckdb.

MPizzotti avatar MPizzotti commented on June 25, 2024

related to the same problem:
I've created a DB file on databricks, using the databricks file system, and then i've copied the file inside Azure BLOB Storage.
then, i've tried to connect using a new duckdb BOTH on my local PC and on Databricks, as follows:

# connecting to the COPIED file inside AZURE, NOT IN LOCAL SYSTEM
ddb3 = duckdb.connect(f"{AZ_BLOB_PATH}/mydb.duckdb", read_only=True)
ddb3.execute("PRAGMA disable_progress_bar;")
ddb3.execute("PRAGMA enable_optimizer;")
ddb3.execute("PRAGMA enable_object_cache;")
ddb3.execute(f"""SET GLOBAL temp_directory = '/local_disk0';""")
ddb3.execute("SET errors_as_json = true;")

once i have this ddb3 connection, i've tried to query a table like:
ddb3.sql("select * from persistent_table where col1= 'code1' and col2='2024-04-20'")

but the query hangs, i've waited for over 15 minutes without any results.

if i run the SAME QUERY, ON THE SAME FILE, inside databricks file system ("/local_disk0/mydb.duckdb")
example of connection:

# connecting to the COPIED file inside LOCAL SYSTEM
ddb2 = duckdb.connect(f"/local_disk0/mydb.duckdb", read_only=True)
ddb2.execute("PRAGMA disable_progress_bar;")
ddb2.execute("PRAGMA enable_optimizer;")
ddb2.execute("PRAGMA enable_object_cache;")
ddb2.execute(f"""SET GLOBAL temp_directory = '/local_disk0';""")
ddb2.execute("SET errors_as_json = true;")

the query runs perfectly fine, in less than a second.

I would love to give more details on How it hangs, but i don't have in depth skills regarding lower level APIs and connections.

from duckdb.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.