databrickslabs / blueprint Goto Github PK

View Code? Open in Web Editor NEW

10.0 5.0 6.0 2.9 MB

Baseline for Databricks Labs projects written in Python

Home Page: https://pypi.org/project/databricks-labs-blueprint/

License: Other

Makefile 0.28% Python 99.72%

cli databricks databricks-cli-installable python

blueprint's People

Stargazers

Watchers

Forkers

ggwiebe pritishpai nkvuong grusin-db devbel0 aminmovahed-db

blueprint's Issues

Command-line UI framework

Question + default
Choices from the list
Choices from the dictionary
max attempt handling
colors

No such file or directory when initialising a project

After running databricks labs blueprint init-project --target discoverx I get the following error

> 
> Name of the project (default: discoverx): 
> 11:59:55 ERROR [d.l.blueprint.init-project] FileNotFoundError: [Errno 2] No such file or directory: '.databricks/labs/blueprint/lib'
> User@C02FR21BML85 discoverx % databricks labs blueprint init-project --target discoverx --debug
> 12:00:05  INFO start pid=15666 version=0.212.1 args="databricks, labs, blueprint, init-project, --target, discoverx, --debug"
> 12:00:05 DEBUG Loading installed version info from: /Users/User/.databricks/labs/blueprint/state/version.json pid=15666
> 12:00:05 DEBUG Passing down environment variables:  pid=15666
> 12:00:05 DEBUG Forwarding subprocess: /Users/User/.databricks/labs/blueprint/state/venv/bin/python3 /Users/User/.databricks/labs/blueprint/lib/src/databricks/labs/blueprint/__main__.py {"command":"init-project","flags":{"log_level":"debug","target":"discoverx"},"output_type":""} pid=15666
> 12:00:05 DEBUG starting: /Users/User/.databricks/labs/blueprint/state/venv/bin/python3 /Users/User/.databricks/labs/blueprint/lib/src/databricks/labs/blueprint/__main__.py {"command":"init-project","flags":{"log_level":"debug","target":"discoverx"},"output_type":""} pid=15666
> Name of the project (default: discoverx): 
> 12:00:09 ERROR [d.l.blueprint.init-project] Failed to call init-project: Traceback (most recent call last):
>   File "/Users/User/.databricks/labs/blueprint/lib/src/databricks/labs/blueprint/cli.py", line 74, in _route
>     self._mapping[command].fn(**kwargs)
>   File "/Users/User/.databricks/labs/blueprint/lib/src/databricks/labs/blueprint/__main__.py", line 101, in init_project
>     for file in current.iterdir():
>   File "/usr/local/Cellar/[email protected]/3.10.13/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pathlib.py", line 1017, in iterdir
>     for name in self._accessor.listdir(self):
> FileNotFoundError: [Errno 2] No such file or directory: '.databricks/labs/blueprint/lib'

Integration with Workflow Authoring Toolkit

Test failure: `test_install_folder_custom`

❌ test_install_folder_custom: AssertionError: assert '/custom/folder' == '/custom/folder...' (632ms)

AssertionError: assert '/custom/folder' == '/custom/folder...'
  
  - /custom/folder...
  ?               ---
  + /custom/folder
22:41 DEBUG [databricks.sdk] Loaded from environment
22:41 DEBUG [databricks.sdk] Ignoring pat auth, because metadata-service is preferred
22:41 DEBUG [databricks.sdk] Ignoring basic auth, because metadata-service is preferred
22:41 DEBUG [databricks.sdk] Attempting to configure auth: metadata-service
22:41 INFO [databricks.sdk] Using Databricks Metadata Service authentication
[gw3] linux -- Python 3.10.13 /home/runner/work/blueprint/blueprint/.venv/bin/python
22:41 DEBUG [databricks.sdk] Loaded from environment
22:41 DEBUG [databricks.sdk] Ignoring pat auth, because metadata-service is preferred
22:41 DEBUG [databricks.sdk] Ignoring basic auth, because metadata-service is preferred
22:41 DEBUG [databricks.sdk] Attempting to configure auth: metadata-service
22:41 INFO [databricks.sdk] Using Databricks Metadata Service authentication
22:41 DEBUG [databricks.sdk] Loaded from environment
22:41 DEBUG [databricks.sdk] Ignoring pat auth, because metadata-service is preferred
22:41 DEBUG [databricks.sdk] Ignoring basic auth, because metadata-service is preferred
22:41 DEBUG [databricks.sdk] Attempting to configure auth: metadata-service
22:41 INFO [databricks.sdk] Using Databricks Metadata Service authentication
[gw3] linux -- Python 3.10.13 /home/runner/work/blueprint/blueprint/.venv/bin/python

_{Running from acceptance #87}

Nice log formatter that is consistent for all Labs products

Create `ProductInfo.for_testing(klass)` factory

at the moment, it's a bit cumbersome to do

        product_name = make_random(4)
        product_info = ProductInfo.from_class(WorkspaceConfig)
        product_info.product_name = lambda: product_name

Add file logging capabilities

When writing notebooks, there is no way to print all logs to the UI, there's a limit in terms of Bytes.

Having a way to dump logs to a file will help the field to debug production tasks easily

Inject prompts to CLI if command needs it

This will simplify the unit testing for commands

Support for integration testing the installer results

Time rotating log file configuration for workspace-fs

Utility to migrate through installed version application upgrades

Sometimes, upgrading an installation requires changes not only to config format, but also application state in the form of data assets and workspace assets, like cluster policies, cluster or dashboard configurations, which results in outages while rolling new versions out.

We need:

in-wheel package name to iterate over submodules
every submodule to be named as v0.12.5.py (verify if this works)
every submodule providing upgrade(WorkspaceClient) function to deploy this version upgrade
current app version determined from ~/.app/version.json file created by the WheelsV2.upload() method
figure out if we execute version migrations atomically or just rely on a version diff and have the upgrader code be resilient to re-applies, possibly via exception ignores and multiple files per release

Common workspace-level installer handling

Project built using Blueprint cannot function as standalone python applications

When attempting to run the Python API of a project built using Blueprint, there is an error associated when initializing the logger. Specifically, it is unable to find the project root.

Python applications extending the blueprint library should be able to run without explicitly having a project root. Note this is run on databricks-labs-remorph==0.2.1 with databricks-labs-blueprint==0.1.0

Command framework for Databricks CLI launcher frontend

`installation.load()` does not handle dict correctly

simple example

from databricks.sdk.service.compute import ClusterDetails

installation = MockInstallation(...)
installation.load(ClusterDetails, filename=...)

fails with

databricks.labs.blueprint.installation.SerdeError: spark_conf: unknown: <class 'dict'>

Configuration file handling framework

file format versioning
file format migrations
databricks client initialization
product info and product version spec

databrickslabs / blueprint Goto Github PK

blueprint's People

Stargazers

Watchers

Forkers

blueprint's Issues

Recommend Projects

Recommend Topics

Recommend Org