Comments (4)
hayesgb
commented on May 27, 2024
1
I can bump it tonight.
from adlfs.
raybellwaves
commented on May 27, 2024
Similar thing for me (the import error). Although I realized it wasn't working in my script.
Tried reading from blob:
ddf = dd.read_parquet(
path=remote_folder + "MY_FILE.parquet",
engine="pyarrow",
storage_options=storage_options,
)
and got:
C:\Users\131416\AppData\Local\Continuum\anaconda3\envs\XXX\lib\site-packages\pyarrow\compat.py:24: FutureWarning: pyarrow.compat has been deprecated and will be removed in a future release
warnings.warn("pyarrow.compat has been deprecated and will be removed in a "
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
~\AppData\Local\Continuum\anaconda3\envs\XXX\lib\site-packages\fsspec\registry.py in get_filesystem_class(protocol)
181 try:
--> 182 register_implementation(protocol, _import_class(bit["class"]))
183 except ImportError as e:
~\AppData\Local\Continuum\anaconda3\envs\XXX\lib\site-packages\fsspec\registry.py in _import_class(cls, minv)
196
--> 197 mod = importlib.import_module(mod)
198 if minversion:
~\AppData\Local\Continuum\anaconda3\envs\XXX\lib\importlib\__init__.py in import_module(name, package)
126 level += 1
--> 127 return _bootstrap._gcd_import(name[level:], package, level)
128
~\AppData\Local\Continuum\anaconda3\envs\XXX\lib\importlib\_bootstrap.py in _gcd_import(name, package, level)
~\AppData\Local\Continuum\anaconda3\envs\XXX\lib\importlib\_bootstrap.py in _find_and_load(name, import_)
~\AppData\Local\Continuum\anaconda3\envs\XXX\lib\importlib\_bootstrap.py in _find_and_load_unlocked(name, import_)
~\AppData\Local\Continuum\anaconda3\envs\XXX\lib\importlib\_bootstrap.py in _load_unlocked(spec)
~\AppData\Local\Continuum\anaconda3\envs\XXX\lib\importlib\_bootstrap_external.py in exec_module(self, module)
~\AppData\Local\Continuum\anaconda3\envs\XXX\lib\importlib\_bootstrap.py in _call_with_frames_removed(f, *args, **kwds)
~\AppData\Local\Continuum\anaconda3\envs\XXX\lib\site-packages\adlfs\__init__.py in <module>
----> 1 from .core import AzureDatalakeFileSystem
2 from .core import AzureBlobFileSystem, AzureBlobFile
3 from ._version import get_versions
~\AppData\Local\Continuum\anaconda3\envs\XXX\lib\site-packages\adlfs\core.py in <module>
13 from azure.storage.blob import BlobServiceClient
---> 14 from azure.storage.blob._models import BlobBlock, BlobPrefix
15 from fsspec import AbstractFileSystem
ImportError: cannot import name 'BlobPrefix' from 'azure.storage.blob._models' (C:\Users\131416\AppData\Local\Continuum\anaconda3\envs\XXX\lib\site-packages\azure\storage\blob\_models.py)
The above exception was the direct cause of the following exception:
ImportError Traceback (most recent call last)
<ipython-input-5-b9b0fa34f8cd> in <module>
----> 1 ddf = dd.read_parquet(
2 path=remote_folder + "MY_FILE.parquet",
3 engine="pyarrow",
4 storage_options=storage_options,
5 )
~\AppData\Local\Continuum\anaconda3\envs\XXX\lib\site-packages\dask\dataframe\io\parquet\core.py in read_parquet(path, columns, filters, categories, index, storage_options, engine, gather_statistics, split_row_groups, chunksize, **kwargs)
218 if hasattr(path, "name"):
219 path = stringify_path(path)
--> 220 fs, _, paths = get_fs_token_paths(path, mode="rb", storage_options=storage_options)
221
222 paths = sorted(paths, key=natural_sort_key) # numeric rather than glob ordering
~\AppData\Local\Continuum\anaconda3\envs\XXX\lib\site-packages\fsspec\core.py in get_fs_token_paths(urlpath, mode, num, name_function, storage_options, protocol)
516 protocols, path = split_protocol(urlpath)
517 protocol = protocol or protocols
--> 518 cls = get_filesystem_class(protocol)
519
520 options = cls._get_kwargs_from_urls(urlpath)
~\AppData\Local\Continuum\anaconda3\envs\XXX\lib\site-packages\fsspec\registry.py in get_filesystem_class(protocol)
182 register_implementation(protocol, _import_class(bit["class"]))
183 except ImportError as e:
--> 184 raise ImportError(bit["err"]) from e
185 cls = registry[protocol]
186 if getattr(cls, "protocol", None) in ("abstract", None):
ImportError: Install adlfs to access Azure Datalake Gen2 and Azure Blob Storage
For reference here's my conda env
# Name Version Build Channel
abseil-cpp 20200225.2 ha925a31_2 conda-forge
adal 1.2.4 pyh9f0ad1d_0 conda-forge
adlfs 0.3.2 py_1 conda-forge
aiohttp 3.6.2 py38hfa6e2cd_0 conda-forge
appdirs 1.4.3 py_1 conda-forge
argon2-cffi 20.1.0 py38h1e8a9f7_1 conda-forge
arrow-cpp 1.0.0 py38h1234567_1_cpu conda-forge
async-timeout 3.0.1 py_1000 conda-forge
atomicwrites 1.4.0 pyh9f0ad1d_0 conda-forge
attrs 19.3.0 py_0 conda-forge
aws-sdk-cpp 1.7.164 vc14h84f8083_2 conda-forge
azure-core 1.8.0 pyh9f0ad1d_0 conda-forge
azure-datalake-store 0.0.49 pyh9f0ad1d_0 conda-forge
azure-identity 1.4.0 pyh9f0ad1d_0 conda-forge
azure-storage-blob 12.4.0 pyh9f0ad1d_0 conda-forge
backcall 0.2.0 pyh9f0ad1d_0 conda-forge
backports 1.0 py_2 conda-forge
backports.functools_lru_cache 1.6.1 py_0 conda-forge
black 19.10b0 py_4 conda-forge
blackcellmagic 0.0.2 pyh9f0ad1d_0 conda-forge
blas 1.0 mkl
bleach 3.1.5 pyh9f0ad1d_0 conda-forge
blinker 1.4 py_1 conda-forge
bokeh 2.1.1 py38h32f6830_0 conda-forge
boost-cpp 1.72.0 h89d28cc_2 conda-forge
bottleneck 1.3.2 py38h4b8e87e_1 conda-forge
brotli 1.0.7 ha925a31_1004 conda-forge
brotlipy 0.7.0 py38h1e8a9f7_1000 conda-forge
bzip2 1.0.8 he774522_2 conda-forge
c-ares 1.16.1 h62dcd97_0 conda-forge
ca-certificates 2020.6.20 hecda079_0 conda-forge
certifi 2020.6.20 py38h32f6830_0 conda-forge
cffi 1.14.1 py38hba49e27_0 conda-forge
chardet 3.0.4 py38h32f6830_1006 conda-forge
click 7.1.2 pyh9f0ad1d_0 conda-forge
cloudpickle 1.5.0 py_0 conda-forge
colorama 0.4.3 py_0 conda-forge
colorcet 2.0.1 py_0 conda-forge
cryptography 3.0 py38hba49e27_0 conda-forge
curl 7.71.1 h4b64cdc_4 conda-forge
cx_oracle 8.0.0 py38hd4486f3_0 conda-forge
cycler 0.10.0 py_2 conda-forge
cytoolz 0.10.1 py38hfa6e2cd_0 conda-forge
dask 2.22.0 py_0 conda-forge
dask-core 2.22.0 py_0 conda-forge
dask-glm 0.2.0 py_1 conda-forge
dask-labextension 3.0.0 py_0 conda-forge
dask-ml 1.6.0 py_0 conda-forge
decorator 4.4.2 py_0 conda-forge
defusedxml 0.6.0 py_0 conda-forge
distributed 2.22.0 py38h32f6830_0 conda-forge
doc8 0.8.1 pyh9f0ad1d_0 conda-forge
docutils 0.16 py38h32f6830_1 conda-forge
entrypoints 0.3 py38h32f6830_1001 conda-forge
fastparquet 0.4.1 py38h40bdaab_0 conda-forge
flake8 3.8.3 py_1 conda-forge
freetype 2.10.2 hd328e21_0 conda-forge
fsspec 0.7.4 py_0 conda-forge
gflags 2.2.2 ha925a31_1004 conda-forge
glog 0.4.0 h0174b99_3 conda-forge
grpc-cpp 1.30.1 h45b88af_1 conda-forge
heapdict 1.0.1 py_0 conda-forge
holoviews 1.13.3 pyh9f0ad1d_0 conda-forge
hvplot 0.6.0 pyh9f0ad1d_0 conda-forge
icc_rt 2019.0.0 h0cc432a_1
idna 2.10 pyh9f0ad1d_0 conda-forge
importlib-metadata 1.7.0 py38h32f6830_0 conda-forge
importlib_metadata 1.7.0 0 conda-forge
iniconfig 1.0.1 pyh9f0ad1d_0 conda-forge
intel-openmp 2019.4 245
ipykernel 5.3.4 py38h5ca1d4c_0 conda-forge
ipython 7.17.0 py38h1cdfbd6_0 conda-forge
ipython_genutils 0.2.0 py_1 conda-forge
isodate 0.6.0 py_1 conda-forge
isort 5.4.1 py38h32f6830_0 conda-forge
jedi 0.17.2 py38h32f6830_0 conda-forge
jinja2 2.11.2 pyh9f0ad1d_0 conda-forge
joblib 0.16.0 py_0 conda-forge
jpeg 9d he774522_0 conda-forge
json5 0.9.4 pyh9f0ad1d_0 conda-forge
jsonschema 3.2.0 py38h32f6830_1 conda-forge
jupyter-server-proxy 1.5.0 py_0 conda-forge
jupyter_client 6.1.6 py_0 conda-forge
jupyter_core 4.6.3 py38h32f6830_1 conda-forge
jupyterlab 2.2.4 py_0 conda-forge
jupyterlab_server 1.2.0 py_0 conda-forge
keyring 21.3.0 py38h32f6830_0 conda-forge
kiwisolver 1.2.0 py38heaebd3c_0 conda-forge
krb5 1.17.1 hc04afaa_2 conda-forge
libblas 3.8.0 14_mkl conda-forge
libcblas 3.8.0 14_mkl conda-forge
libcurl 7.71.1 h4b64cdc_4 conda-forge
liblapack 3.8.0 14_mkl conda-forge
libllvm9 9.0.1 hde8ddfe_1 conda-forge
libpng 1.6.37 ha81a0f5_2 conda-forge
libprotobuf 3.12.4 h200bbdf_0 conda-forge
libsodium 1.0.17 h2fa13f4_0 conda-forge
libssh2 1.9.0 hb06d900_5 conda-forge
libtiff 4.1.0 h885aae3_6 conda-forge
libutf8proc 2.5.0 h9e6e254_2 conda-forge
llvmlite 0.33.0 py38ha925a31_0
locket 0.2.0 py_2 conda-forge
lz4-c 1.9.2 h62dcd97_1 conda-forge
m2w64-gcc-libgfortran 5.3.0 6
m2w64-gcc-libs 5.3.0 7
m2w64-gcc-libs-core 5.3.0 7
m2w64-gmp 6.1.0 2
m2w64-libwinpthread-git 5.0.0.4634.697f757 2
markdown 3.2.2 py_0 conda-forge
markupsafe 1.1.1 py38h9de7a3e_1 conda-forge
matplotlib-base 3.3.0 py38hfb9ee82_1 conda-forge
mccabe 0.6.1 py_1 conda-forge
mistune 0.8.4 py38h9de7a3e_1001 conda-forge
mkl 2019.4 245
mkl-service 2.3.0 py38hfa6e2cd_0 conda-forge
more-itertools 8.4.0 py_0 conda-forge
msal 1.4.3 pyh9f0ad1d_0 conda-forge
msal_extensions 0.2.2 pyh9f0ad1d_0 conda-forge
msgpack-python 1.0.0 py38heaebd3c_1 conda-forge
msrest 0.6.18 pyh9f0ad1d_0 conda-forge
msrestazure 0.6.2 py_0 conda-forge
msys2-conda-epoch 20160418 1
multidict 4.7.5 py38h1e8a9f7_1 conda-forge
multipledispatch 0.6.0 py_0 conda-forge
nbconvert 5.6.1 py38h32f6830_1 conda-forge
nbformat 5.0.7 py_0 conda-forge
nodejs 14.8.0 0 conda-forge
notebook 6.1.3 py38h32f6830_0 conda-forge
numba 0.50.1 py38he6e81aa_1 conda-forge
numpy 1.19.1 py38h72c728b_0 conda-forge
oauthlib 3.0.1 py_0 conda-forge
olefile 0.46 py_0 conda-forge
openssl 1.1.1g he774522_1 conda-forge
packaging 20.4 pyh9f0ad1d_0 conda-forge
pandas 1.1.0 py38h7ae7562_0 conda-forge
pandoc 2.10.1 he774522_0 conda-forge
pandocfilters 1.4.2 py_1 conda-forge
panel 0.9.7 py_0 conda-forge
param 1.9.3 py_0 conda-forge
parquet-cpp 1.5.1 2 conda-forge
parso 0.7.1 pyh9f0ad1d_0 conda-forge
partd 1.1.0 py_0 conda-forge
pathspec 0.8.0 pyh9f0ad1d_0 conda-forge
patsy 0.5.1 py_0 conda-forge
pbr 5.4.5 pyh9f0ad1d_0 conda-forge
pickleshare 0.7.5 py38h32f6830_1001 conda-forge
pillow 7.2.0 py38h7011068_1 conda-forge
pip 20.2.2 py_0 conda-forge
pluggy 0.13.1 py38h32f6830_2 conda-forge
portalocker 1.7.0 py38h32f6830_0 conda-forge
prometheus_client 0.8.0 pyh9f0ad1d_0 conda-forge
prompt-toolkit 3.0.6 py_0 conda-forge
properscoring 0.1 py_0 conda-forge
psutil 5.7.2 py38h1e8a9f7_0 conda-forge
py 1.9.0 pyh9f0ad1d_0 conda-forge
pyarrow 1.0.0 py38h1234567_1_cpu conda-forge
pycodestyle 2.6.0 pyh9f0ad1d_0 conda-forge
pycparser 2.20 pyh9f0ad1d_2 conda-forge
pyct 0.4.6 py_0 conda-forge
pyct-core 0.4.6 py_0 conda-forge
pyflakes 2.2.0 pyh9f0ad1d_0 conda-forge
pygments 2.6.1 py_0 conda-forge
pyjwt 1.7.1 py_0 conda-forge
pyopenssl 19.1.0 py_1 conda-forge
pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge
pyrsistent 0.16.0 py38h9de7a3e_0 conda-forge
pysocks 1.7.1 py38h32f6830_1 conda-forge
pytest 6.0.1 py38h32f6830_0 conda-forge
python 3.8.5 h5fd99cc_2_cpython conda-forge
python-dateutil 2.8.1 py_0 conda-forge
python-snappy 0.5.4 py38h9a6de72_1 conda-forge
python_abi 3.8 1_cp38 conda-forge
pytz 2020.1 pyh9f0ad1d_0 conda-forge
pyviz_comms 0.7.6 pyh9f0ad1d_0 conda-forge
pywin32 227 py38hfa6e2cd_0 conda-forge
pywin32-ctypes 0.2.0 py38h32f6830_1001 conda-forge
pywinpty 0.5.7 py38_0 conda-forge
pyyaml 5.3.1 py38h9de7a3e_0 conda-forge
pyzmq 19.0.2 py38h77b9d75_0 conda-forge
re2 2020.08.01 ha925a31_0 conda-forge
regex 2020.7.14 py38h1e8a9f7_0 conda-forge
requests 2.24.0 pyh9f0ad1d_0 conda-forge
requests-oauthlib 1.3.0 pyh9f0ad1d_0 conda-forge
restructuredtext_lint 1.3.1 pyh9f0ad1d_0 conda-forge
scikit-learn 0.24.dev0 pypi_0 pypi
scipy 1.5.0 py38h9439919_0
seaborn 0.10.1 1 conda-forge
seaborn-base 0.10.1 py_1 conda-forge
send2trash 1.5.0 py_0 conda-forge
setuptools 49.3.2 py38h32f6830_0 conda-forge
simpervisor 0.3 py_1 conda-forge
six 1.15.0 pyh9f0ad1d_0 conda-forge
snappy 1.1.8 ha925a31_3 conda-forge
sortedcontainers 2.2.2 pyh9f0ad1d_0 conda-forge
sqlalchemy 1.3.18 py38h1e8a9f7_0 conda-forge
sqlite 3.32.3 he774522_1 conda-forge
statsmodels 0.11.1 py38h1e8a9f7_2 conda-forge
stevedore 3.2.0 py38h32f6830_0 conda-forge
tblib 1.6.0 py_0 conda-forge
terminado 0.8.3 py38h32f6830_1 conda-forge
testpath 0.4.4 py_0 conda-forge
threadpoolctl 2.1.0 pyh5ca1d4c_0 conda-forge
thrift 0.11.0 py38h6538335_1001 conda-forge
thrift-cpp 0.13.0 h1907cbf_2 conda-forge
tk 8.6.10 he774522_0 conda-forge
toml 0.10.1 pyh9f0ad1d_0 conda-forge
toolz 0.10.0 py_0 conda-forge
tornado 6.0.4 py38hfa6e2cd_0 conda-forge
tqdm 4.48.2 pyh9f0ad1d_0 conda-forge
traitlets 4.3.3 py38h32f6830_1 conda-forge
typed-ast 1.4.1 py38hfa6e2cd_0 conda-forge
typing_extensions 3.7.4.2 py_0 conda-forge
urllib3 1.25.10 py_0 conda-forge
vc 14.1 h869be7e_1 conda-forge
vs2015_runtime 14.16.27012 h30e32a0_2 conda-forge
wcwidth 0.2.5 pyh9f0ad1d_1 conda-forge
webencodings 0.5.1 py_1 conda-forge
wheel 0.34.2 py_1 conda-forge
win_inet_pton 1.1.0 py38_0 conda-forge
wincertstore 0.2 py38_1003 conda-forge
winpty 0.4.3 4 conda-forge
xarray 0.16.0 py_0 conda-forge
xskillscore 0.0.16 py_0 conda-forge
xz 5.2.5 h62dcd97_1 conda-forge
yaml 0.2.5 he774522_0 conda-forge
yarl 1.3.0 py38hfa6e2cd_1000 conda-forge
zeromq 4.3.2 ha925a31_3 conda-forge
zict 2.0.0 py_0 conda-forge
zipp 3.1.0 py_0 conda-forge
zlib 1.2.11 h62dcd97_1007 conda-forge
zstd 1.4.5 h1f3a1b7_2 conda-forge
from adlfs.
mattcc82
commented on May 27, 2024
Also just ran into this issue using a dask cluster - my workaround was to pip install a previous version of azure-storage-blob==12.4.0b1 (didn't try any other versions to test)
from adlfs.
hayesgb
commented on May 27, 2024
Done.
from adlfs.
Related Issues (20)
- New release? HOT 1
- Concurrent reads/writes cause BlobNotFound
- TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' HOT 6
- Support for MS Fabric OneLake
- support az cli config ~/.azure/config HOT 1
- UserWarning: Failed to fetch container properties for CONTAINER_NAME. Assume it exists already HOT 1
- "sdk_moniker" key error HOT 9
- Avoid private APIs from azure.storage HOT 2
- InternalServerError while writing large json data.
- await file_obj.credential.close() : TypeError: object NoneType can't be used in 'await' expression HOT 4
- update readme HOT 1
- Support py3.12
- `find` doesn't accept `maxdepth` parameter HOT 1
- Add use_emulator setting to better align with object_store crate HOT 1
- Current state of the library, milestones and current development HOT 1
- Concurrent download of multiple files HOT 1
- Support virtual directory stubs with uppercase "Hdi_isfolder" metadata HOT 1
- Feature Suggestion: Optional content type when for writing file HOT 2
- Support passing url in AzureBlobFileSystem HOT 1
- Add comment why `aiohttp` is required
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from adlfs.