SDK installation on databricks about machinelearningnotebooks HOT 11 CLOSED

databricks 5.0 runtimes should also work, just choose the non-preview ones.

from machinelearningnotebooks.

Try installing using pip install azureml-sdk[databricks]. That should pin the dependent package versions.

from machinelearningnotebooks.

@rastala I did try installing azureml-sdk[databricks] and that gave me the error as follows

Error: java.lang.RuntimeException: Installation failed with message: Traceback (most recent call last): File "/databricks/python/bin/pip", line 7, in from pip._internal import main File "/databricks/python3/lib/python3.5/site-packages/pip/_internal/init.py", line 42, in from pip._internal import cmdoptions File "/databricks/python3/lib/python3.5/site-packages/pip/_internal/cmdoptions.py", line 16, in from pip._internal.index import ( File "/databricks/python3/lib/python3.5/site-packages/pip/_internal/index.py", line 15, in from pip._vendor import html5lib, requests, six File "/databricks/python3/lib/python3.5/site-packages/pip/_vendor/requests/init.py", line 86, in from pip._vendor.urllib3.contrib import pyopenssl File "/databricks/python3/lib/python3.5/site-packages/pip/_vendor/urllib3/contrib/pyopenssl.py", line 46, in import OpenSSL.SSL File "/databricks/python3/lib/python3.5/site-packages/OpenSSL/init.py", line 8, in from OpenSSL import rand, crypto, SSL File "/databricks/python3/lib/python3.5/site-packages/OpenSSL/SSL.py", line 124, in SSL_ST_INIT = _lib.SSL_ST_INIT AttributeError: module 'lib' has no attribute 'SSL_ST_INIT' Traceback (most recent call last): File "/databricks/python/bin/pip", line 7, in from pip._internal import main File "/databricks/python3/lib/python3.5/site-packages/pip/_internal/init.py", line 42, in from pip._internal import cmdoptions File "/databricks/python3/lib/python3.5/site-packages/pip/_internal/cmdoptions.py", line 16, in from pip._internal.index import ( File "/databricks/python3/lib/python3.5/site-packages/pip/_internal/index.py", line 15, in from pip._vendor import html5lib, requests, six File "/databricks/python3/lib/python3.5/site-packages/pip/_vendor/requests/init.py", line 86, in from pip._vendor.urllib3.contrib import pyopenssl File "/databricks/python3/lib/python3.5/site-packages/pip/_vendor/urllib3/contrib/pyopenssl.py", line 46, in import OpenSSL.SSL File "/databricks/python3/lib/python3.5/site-packages/OpenSSL/init.py", line 8, in from OpenSSL import rand, crypto, SSL File "/databricks/python3/lib/python3.5/site-packages/OpenSSL/SSL.py", line 124, in SSL_ST_INIT = _lib.SSL_ST_INIT AttributeError: module 'lib' has no attribute 'SSL_ST_INIT' Traceback (most recent call last): File "/databricks/python/bin/pip", line 7, in from pip._internal import main File "/databricks/python3/lib/python3.5/site-packages/pip/_internal/init.py", line 42, in from pip._internal import cmdoptions File "/databricks/python3/lib/python3.5/site-packages/pip/_internal/cmdoptions.py", line 16, in from pip._internal.index import ( File "/databricks/python3/lib/python3.5/site-packages/pip/_internal/index.py", line 15, in from pip._vendor import html5lib, requests, six File "/databricks/python3/lib/python3.5/site-packages/pip/_vendor/requests/init.py", line 86, in from pip._vendor.urllib3.contrib import pyopenssl File "/databricks/python3/lib/python3.5/site-packages/pip/_vendor/urllib3/contrib/pyopenssl.py", line 46, in import OpenSSL.SSL File "/databricks/python3/lib/python3.5/site-packages/OpenSSL/init.py", line 8, in from OpenSSL import rand, crypto, SSL File "/databricks/python3/lib/python3.5/site-packages/OpenSSL/rand.py", line 12, in from OpenSSL._util import ( File "/databricks/python3/lib/python3.5/site-packages/OpenSSL/_util.py", line 6, in from cryptography.hazmat.bindings.openssl.binding import Binding File "/databricks/python3/lib/python3.5/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 157, in Binding.init_static_locks() File "/databricks/python3/lib/python3.5/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 138, in init_static_locks cls._ensure_ffi_initialized() File "/databricks/python3/lib/python3.5/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 125, in _ensure_ffi_initialized cls.lib = build_conditional_library(lib, CONDITIONAL_NAMES) File "/databricks/python3/lib/python3.5/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 85, in build_conditional_library if not getattr(lib, condition): AttributeError: cffi library '_openssl' has no function, constant or global variable named 'Cryptography_HAS_SCRYPT'

from machinelearningnotebooks.

I think the steps to resolve this are:

1. Remove all the attached libraries from the cluster
2. Restart the cluster, which will reset the cluster to its default packages.
3. Attach azureml-sdk[databricks] again.

from machinelearningnotebooks.

I am getting the following error. is there a specific cluster version I should use?

Error: java.lang.RuntimeException: Installation failed with message: Collecting azureml-sdk[databricks] Downloading https://files.pythonhosted.org/packages/78/1a/a312e326fe491f01a2fa26220066e3e8f2a33fef779f165694b09b522b79/azureml_sdk-0.1.74-py3-none-any.whl Collecting azureml-core==0.1.74.* (from azureml-sdk[databricks]) Downloading https://files.pythonhosted.org/packages/b4/e1/062c248f174389615422ad85c1fbd56b686c185743758fc5983957610bcf/azureml_core-0.1.74-py2.py3-none-any.whl (781kB) Collecting azureml-pipeline==0.1.74.* (from azureml-sdk[databricks]) Downloading https://files.pythonhosted.org/packages/d5/c8/f4ff3c32b3f6dc2291c5ab7f31a4c6a37098fcd46a315b6d08386d7cf891/azureml_pipeline-0.1.74-py3-none-any.whl Collecting azureml-train==0.1.74.* (from azureml-sdk[databricks]) Downloading https://files.pythonhosted.org/packages/6a/3a/442ca39f58c9ea60bd1463d95fd8f10a3db5be77ca251ef7e9277a384cf3/azureml_train-0.1.74-py3-none-any.whl Collecting pyOpenSSL==16.0.0; extra == "databricks" (from azureml-sdk[databricks]) Downloading https://files.pythonhosted.org/packages/b1/4e/54c8995d2de887919272c2b711cd430277ec33e0f7bb88cb564244c102b6/pyOpenSSL-16.0.0-py2.py3-none-any.whl (45kB) Collecting azureml-train-automl==0.1.74.*; extra == "databricks" (from azureml-sdk[databricks]) Downloading https://files.pythonhosted.org/packages/42/7e/a212dbe5585f9d21eac29d014582a1e13727576c46067afff1f54996007c/azureml_train_automl-0.1.74-py3-none-any.whl (146kB) Collecting cryptography==1.5; extra == "databricks" (from azureml-sdk[databricks]) Using cached https://files.pythonhosted.org/packages/6e/96/b8dab146e8be98061dae07e127f80cffa3061ab0e8da0d3d42f3308c6e91/cryptography-1.5.tar.gz Collecting ipython==2.2.0; extra == "databricks" (from azureml-sdk[databricks]) Downloading https://files.pythonhosted.org/packages/95/9a/ed1ea422d5bc77d66c98e4a6bfba5f3e46986b5c4c605ba49271af2f622b/ipython-2.2.0-py3-none-any.whl (2.8MB) Collecting azure-mgmt-keyvault>=0.40.0 (from azureml-core==0.1.74.*->azureml-sdk[databricks]) Downloading https://files.pythonhosted.org/packages/49/de/0d69aedae7c5f6428314640b65947203ab80409c12b5d4e66fb5b7a4182e/azure_mgmt_keyvault-1.1.0-py2.py3-none-any.whl (111kB) Collecting azure-mgmt-authorization>=0.40.0 (from azureml-core==0.1.74.*->azureml-sdk[databricks]) Downloading https://files.pythonhosted.org/packages/20/91/cad6fb2eb9ad106c8d5bd45514af2772c956e921f57b85db6bd0919923e7/azure_mgmt_authorization-0.51.0-py2.py3-none-any.whl (111kB) Collecting azure-cli-profile>=2.0.26 (from azureml-core==0.1.74.*->azureml-sdk *** 2500 first characters above *** Command output was truncated as its original size (28511 characters) exceeds the size limit (5000 characters). *** 2500 last characters below *** ored in directory: /root/.cache/pip/wheels/ad/da/0c/74eb680767247273e2cf2723482cb9c924fe70af57c334513f Running setup.py bdist_wheel for antlr4-python3-runtime: started Running setup.py bdist_wheel for antlr4-python3-runtime: finished with status 'done' Stored in directory: /root/.cache/pip/wheels/ef/f6/18/ad300e691236a3408a99edc750484b56e8d6b11b2c38eacb10 Running setup.py bdist_wheel for JsonForm: started Running setup.py bdist_wheel for JsonForm: finished with status 'done' Stored in directory: /root/.cache/pip/wheels/e8/74/51/42c2d41c02bdc6f0e604476b7e4293b8c98d0bcbfa1dff78c8 Running setup.py bdist_wheel for JsonSir: started Running setup.py bdist_wheel for JsonSir: finished with status 'done' Stored in directory: /root/.cache/pip/wheels/ee/30/5c/3a3b5e1386c8db9a3be5f5c3933644ae0533c1351c6a8eb4b5 Successfully built cryptography SecretStorage pathspec tabulate pyyaml antlr4-python3-runtime JsonForm JsonSir Installing collected packages: urllib3, requests, oauthlib, requests-oauthlib, isodate, msrest, cryptography, PyJWT, adal, msrestazure, azure-common, azure-nspkg, azure-mgmt-nspkg, azure-mgmt-keyvault, azure-mgmt-authorization, tabulate, pyyaml, azure-cli-nspkg, pyOpenSSL, argcomplete, knack, antlr4-python3-runtime, wheel, azure-mgmt-resource, portalocker, applicationinsights, azure-cli-telemetry, humanfriendly, azure-cli-core, azure-cli-command-modules-nspkg, azure-cli-profile, ndg-httpsclient, SecretStorage, azure-mgmt-storage, pathspec, ruamel.yaml, azure-graphrbac, azure-storage-common, azure-storage-blob, azure-mgmt-containerregistry, jsonpickle, contextlib2, azure-storage-nspkg, websocket-client, docker-pycreds, docker, backports.weakref, backports.tempfile, azureml-core, azureml-telemetry, azureml-train-restclients-hyperdrive, azureml-train-core, azureml-pipeline-core, azureml-pipeline-steps, azureml-pipeline, azureml-train, numpy, sklearn-pandas, lightgbm, azureml-dataprep-native, distro, dotnetcore2, azureml-dataprep, python-easyconfig, JsonForm, JsonSir, resource, azureml-train-automl, ipython, azureml-sdk Found existing installation: urllib3 1.22 Uninstalling urllib3-1.22: Successfully uninstalled urllib3-1.22 Found existing installation: requests 2.18.4 Uninstalling requests-2.18.4: Successfully uninstalled requests-2.18.4 Found existing installation: cryptography 2.2.2 Uninstalling cryptography-2.2.2: Successfully uninstalled cryptography-2.2.2 Found existing installation: PyYAML 3.12 Collecting azureml-sdk[databricks] Downloading https://files.pythonhosted.org/packages/78/1a/a312e326fe491f01a2fa26220066e3e8f2a33fef779f165694b09b522b79/azureml_sdk-0.1.74-py3-none-any.whl Collecting azureml-core==0.1.74.* (from azureml-sdk[databricks]) Downloading https://files.pythonhosted.org/packages/b4/e1/062c248f174389615422ad85c1fbd56b686c185743758fc5983957610bcf/azureml_core-0.1.74-py2.py3-none-any.whl (781kB) Collecting azureml-train==0.1.74.* (from azureml-sdk[databricks]) Downloading https://files.pythonhosted.org/packages/6a/3a/442ca39f58c9ea60bd1463d95fd8f10a3db5be77ca251ef7e9277a384cf3/azureml_train-0.1.74-py3-none-any.whl Collecting azureml-pipeline==0.1.74.* (from azureml-sdk[databricks]) Downloading https://files.pythonhosted.org/packages/d5/c8/f4ff3c32b3f6dc2291c5ab7f31a4c6a37098fcd46a315b6d08386d7cf891/azureml_pipeline-0.1.74-py3-none-any.whl Collecting pyOpenSSL==16.0.0; extra == "databricks" (from azureml-sdk[databricks]) Downloading https://files.pythonhosted.org/packages/b1/4e/54c8995d2de887919272c2b711cd430277ec33e0f7bb88cb564244c102b6/pyOpenSSL-16.0.0-py2.py3-none-any.whl (45kB) Collecting cryptography==1.5; extra == "databricks" (from azureml-sdk[databricks]) Using cached https://files.pythonhosted.org/packages/6e/96/b8dab146e8be98061dae07e127f80cffa3061ab0e8da0d3d42f3308c6e91/cryptography-1.5.tar.gz Collecting azureml-train-automl==0.1.74.*; extra == "databricks" (from azureml-sdk[databricks]) Downloading https://files.pythonhosted.org/packages/42/7e/a212dbe5585f9d21eac29d014582a1e13727576c46067afff1f54996007c/azureml_train_automl-0.1.74-py3-none-any.whl (146kB) Collecting ipython==2.2.0; extra == "databricks" (from azureml-sdk[databricks]) Downloading https://files.pythonhosted.org/packages/95/9a/ed1ea422d5bc77d66c98e4a6bfba5f3e46986b5c4c605ba49271af2f622b/ipython-2.2.0-py3-none-any.whl (2.8MB) Collecting azure-cli-core>=2.0.38 (from azureml-core==0.1.74.*->azureml-sdk[databricks]) Downloading https://files.pythonhosted.org/packages/18/fa/4adcb8d2a7683ba972eb2b56d68f21cd0d5b43222c368c5ee6bc38cf2e4d/azure_cli_core-2.0.50-py2.py3-none-any.whl (107kB) Collecting jsonpickle (from azureml-core==0.1.74.*->azureml-sdk[databricks]) Downloading https://files.pythonhosted.org/packages/ca/ce/97404d5aeb58e6155c216825c81b50f6eca8a5345c582317ae48391878f8/jsonpickle-1.0-py2.py3-none-any.whl Collecting azure-storage-common>=1.1.0 (from azureml-core==0.1.74.*->azureml-sdk[databricks]) Downloading https://files.pythonhosted *** 2500 first characters above *** Command output was truncated as its original size (28627 characters) exceeds the size limit (5000 characters). *** 2500 last characters below *** py bdist_wheel for antlr4-python3-runtime: started Running setup.py bdist_wheel for antlr4-python3-runtime: finished with status 'done' Stored in directory: /root/.cache/pip/wheels/ef/f6/18/ad300e691236a3408a99edc750484b56e8d6b11b2c38eacb10 Running setup.py bdist_wheel for JsonForm: started Running setup.py bdist_wheel for JsonForm: finished with status 'done' Stored in directory: /root/.cache/pip/wheels/e8/74/51/42c2d41c02bdc6f0e604476b7e4293b8c98d0bcbfa1dff78c8 Running setup.py bdist_wheel for JsonSir: started Running setup.py bdist_wheel for JsonSir: finished with status 'done' Stored in directory: /root/.cache/pip/wheels/ee/30/5c/3a3b5e1386c8db9a3be5f5c3933644ae0533c1351c6a8eb4b5 Successfully built cryptography SecretStorage pathspec tabulate pyyaml antlr4-python3-runtime JsonForm JsonSir Installing collected packages: wheel, urllib3, requests, PyJWT, cryptography, adal, oauthlib, requests-oauthlib, isodate, msrest, argcomplete, tabulate, azure-nspkg, azure-cli-nspkg, applicationinsights, portalocker, azure-cli-telemetry, msrestazure, pyyaml, humanfriendly, pyOpenSSL, antlr4-python3-runtime, knack, azure-mgmt-nspkg, azure-common, azure-mgmt-resource, azure-cli-core, jsonpickle, azure-storage-common, azure-mgmt-keyvault, azure-storage-nspkg, azure-mgmt-authorization, ndg-httpsclient, azure-graphrbac, azure-storage-blob, SecretStorage, ruamel.yaml, azure-mgmt-containerregistry, azure-cli-command-modules-nspkg, azure-cli-profile, backports.weakref, backports.tempfile, websocket-client, docker-pycreds, docker, contextlib2, azure-mgmt-storage, pathspec, azureml-core, azureml-telemetry, azureml-train-restclients-hyperdrive, azureml-train-core, azureml-train, azureml-pipeline-core, azureml-pipeline-steps, azureml-pipeline, azureml-dataprep-native, distro, dotnetcore2, azureml-dataprep, numpy, lightgbm, JsonForm, JsonSir, python-easyconfig, resource, sklearn-pandas, azureml-train-automl, ipython, azureml-sdk Found existing installation: wheel 0.31.1 Uninstalling wheel-0.31.1: Successfully uninstalled wheel-0.31.1 Found existing installation: urllib3 1.22 Uninstalling urllib3-1.22: Successfully uninstalled urllib3-1.22 Found existing installation: requests 2.18.4 Uninstalling requests-2.18.4: Successfully uninstalled requests-2.18.4 Found existing installation: cryptography 2.2.2 Uninstalling cryptography-2.2.2: Successfully uninstalled cryptography-2.2.2 Found existing installation: PyYAML 3.12 azure-cli-core 2.0.50 has requirement pyopenssl>=17.1.0, but you'll have pyopenssl 16.0.0 which is incompatible. Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall. You are using pip version 10.0.1, however version 18.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command. azure-cli-core 2.0.50 has requirement pyopenssl>=17.1.0, but you'll have pyopenssl 16.0.0 which is incompatible. Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall. You are using pip version 10.0.1, however version 18.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command.

from machinelearningnotebooks.

What cluster version are you using? Please try to use cluster version 4.3

from machinelearningnotebooks.

      What cluster version are you using? Please try to use cluster version 4.3

Thanks! That worked! now on to deploying a spark ml model!

from machinelearningnotebooks.

I have Databricks non-premium cluster and I dont see ML engine below 5.0 - how can I used 4.x on non-premium?

from machinelearningnotebooks.

Thanks that worked for me as well.
But if i try to use the dataprep package, i get no module found error. But i could able to import azureml.core.

from machinelearningnotebooks.

I'm having the exact same stacktrace when I try to install azureml-sdk on my databricks cluster.
Here's my configuration:

• Standard mode cluster
• Databricks 5.3
• Python 3
• Worker_type: Standard_DS3_V2
• Driver type: Standard_DS3_V2

Thank you for your help

from machinelearningnotebooks.

Thanks that worked for me as well.
But if i try to use the dataprep package, i get no module found error. But i could able to import azureml.core.

Did you get this working in the end? I was curious as to using the DataPrep SDK on Databricks - it's another API to learn...

from machinelearningnotebooks.