BUG: kedro neptune init fail about neptune-client HOT 10 CLOSED

Sure, here it is:

CONFIG_LOADER_ARGS = {
    "custom_resolvers": {
        "merge": merge_dicts,
    },
    "config_patterns": {
        "credentials_neptune": ["credentials_neptune*"],
        "neptune": ["neptune*"],
    }
}

from neptune-client.

Unfortunately I cannot share the code and the project I am using as there are sensible information.

However, I just tried the simpliest thing: I removed and reinstalled kedro and kedro-telemetry (always the 0.18.14, due to catalog compatibility issues), run kedro neptune init... and it worked. So, I could imagine that the problem was some dirty in the python venv.

from neptune-client.

Glad that you were able to get this resolved! 🎉

Since it seems to be an issue with kedro/kedro-telemetry, can you try raising an issue in kedro?

Meanwhile, I am closing this. But please feel free to reopen or comment if need be 🤗

from neptune-client.

Hey @netphantom 👋

Looks like you have not added the config patterns needed to load the Neptune config to your project's settings.py -> CONFIG_LOADER_ARGS.

You will need to update CONFIG_LOADER_ARGS as below:

CONFIG_LOADER_ARGS = {
    ...,
    "config_patterns": {
        ...,
        "credentials_neptune" : ["credentials_neptune*"],
        "neptune": ["neptune*"],
    }
}

Here is the step-by-step installation and setup guide for kedro-neptune: https://docs.neptune.ai/integrations/kedro/

Please let me know if this helps, or if there is anything else I can help you with.

from neptune-client.

Hi @SiddhantSadangi , thank you for your welcome :).
I have updated the CONFIG_LOADER_ARGS, however as the init command fails, it does not create the yaml files in the config folders with the required credentials. As a matter of fact, if I add the config_patterns and I run the kedro neptune init, I have the following error:

Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in run_code
File "C:\Users\kedro-project\venv\Scripts\kedro.exe_main.py", line 7, in
File "C:\Users\kedro-project\venv\Lib\site-packages\kedro\framework\cli\cli.py", line 211, in main
cli_collection()
File "C:\Users\kedro-project\venv\Lib\site-packages\click\core.py", line 1157, in call
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\kedro\framework\cli\cli.py", line 139, in main
super().main(
File "C:\Users\kedro-project\venv\Lib\site-packages\click\core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\click\core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\click\core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\click\core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\click\core.py", line 783, in invoke
return _callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\click\decorators.py", line 45, in new_func
return f(get_current_context().obj, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\kedro_neptune_init.py", line 177, in init
context = session.load_context()
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\kedro\framework\session\session.py", line 284, in load_context
self._hook_manager.hook.after_context_created(context=context)
File "C:\Users\kedro-project\venv\Lib\site-packages\pluggy_hooks.py", line 433, in call
return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\pluggy_manager.py", line 112, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\pluggy_manager.py", line 418, in traced_hookexec
return outcome.get_result()
^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\pluggy_result.py", line 108, in get_result
raise exc.with_traceback(exc.traceback)
File "C:\Users\kedro-project\venv\Lib\site-packages\pluggy_result.py", line 70, in from_call
result = func()
^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\pluggy_manager.py", line 415, in
lambda: oldcall(hook_name, hook_impls, caller_kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\pluggy_callers.py", line 116, in _multicall
raise exception.with_traceback(exception.traceback)
File "C:\Users\kedro-project\venv\Lib\site-packages\pluggy_callers.py", line 80, in _multicall
res = hook_impl.function(args)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\kedro_telemetry\plugin.py", line 128, in after_context_created
catalog = context.catalog
^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\kedro\framework\context\context.py", line 226, in catalog
return self._get_catalog()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\kedro\framework\context\context.py", line 287, in _get_catalog
self._hook_manager.hook.after_catalog_created(
File "C:\Users\kedro-project\venv\Lib\site-packages\pluggy_hooks.py", line 433, in call
return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\pluggy_manager.py", line 112, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\pluggy_manager.py", line 418, in traced_hookexec
return outcome.get_result()
^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\pluggy_result.py", line 108, in get_result
raise exc.with_traceback(exc.traceback)
File "C:\Users\kedro-project\venv\Lib\site-packages\pluggy_result.py", line 70, in from_call
result = func()
^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\pluggy_manager.py", line 415, in
lambda: oldcall(hook_name, hook_impls, caller_kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\pluggy_callers.py", line 116, in _multicall
raise exception.with_traceback(exception.traceback)
File "C:\Users\kedro-project\venv\Lib\site-packages\pluggy_callers.py", line 80, in multicall
res = hook_impl.function(*args)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\kedro_neptune_init.py", line 425, in after_catalog_created
config = get_neptune_config(settings)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\kedro_neptune\config.py", line 37, in get_neptune_config
config = config_loader["neptune"]
~~~~~~~~~~~~~^^^^^^^^^^^
File "C:\Users\kedro-project\venv\Lib\site-packages\kedro\config\omegaconf_config.py", line 244, in getitem
raise MissingConfigException(
kedro.config.abstract_config.MissingConfigException: No files of YAML or JSON format found in conf\base or conf\local matching the glob pattern(s): ['neptune']

If I manually create the files, it keeps going with other errors/configurations missing

from neptune-client.

Could you share the config_patterns you are using?
Looks like you might have missed adding the * at the end of "neptune*" :)

from neptune-client.

Thats'weird.

I am not able to reproduce this issue.
Could you take me through the steps you followed, in order, while creating your kedro project, and share the structure of your project?

The settings.py file is used only for running the kedro project, and does not need to be updated before you run kedro neptune init.

Also, from your previous traceback, I would assume that the pattern in your config_patterns was ["neptune"], and not ["neptune*"], but that doesn't seem to be the case. Can you reconfirm that you are not able to run your project using the config_patterns you shared?

from neptune-client.

Hi @SiddhantSadangi, the steps I did were simple:

• Install with pip install -U kedro-neptune (I also tried pip install -U "neptune[kedro]" but it didn't change the procedure outcome)
• Then I run kedro neptune init, receiving the error I mentioned.

I cloned the same repository I am using on a Linux (Manjaro) machine, run kedro neptune init and everything worked perfectly.

It seems to be some weirdo Windows problem, I don't know where to look to solve this. By the way this is the project structure:

.
├── conf
│   ├── base
│   │   ├── catalog.yml
│   │   ├── logging.yml
│   │   ├── parameters_pipeline1.yml
│   │   ├── parameters_pipeline2.yml
│   │   ├── parameters_pipeline3.yml
│   │   ├── parameters_pipeline4.yml
│   │   └── parameters.yml
│   ├── local
│   │   └── credentials.yml
│   └── README.md
├── docs
│   └── source
│       ├── conf.py
│       └── index.rst
├── info.log
├── notebooks
├── pyproject.toml
├── README.md
├── src
│   ├── lstm
│   │   ├── extras
│   │   │   ├── dataset
│   │   │   │   ├── __init__.py
│   │   │   │   └── torch_set.py
│   │   │   └── __init__.py
│   │   ├── __init__.py
│   │   ├── __main__.py
│   │   ├── pipeline_registry.py
│   │   ├── pipelines
│   │   │   ├── pipeline1
│   │   │   │   ├── accessories.py
│   │   │   │   ├── __init__.py
│   │   │   │   ├── nodes.py
│   │   │   │   └── pipeline.py
│   │   │   ├── __init__.py
│   │   │   ├── pipeline2
│   │   │   │   ├── __init__.py
│   │   │   │   ├── nodes.py
│   │   │   │   └── pipeline.py
│   │   │   ├── pipeline3
│   │   │   │   ├── __init__.py
│   │   │   │   ├── nodes.py
│   │   │   │   └── pipeline.py
│   │   │   └── pipeline4
│   │   │       ├── __init__.py
│   │   │       ├── nodes.py
│   │   │       └── pipeline.py
│   │   └── settings.py
│   ├── pyproject.toml
│   ├── requirements.txt
│   ├── tests
│   │   ├── __init__.py
│   │   ├── pipelines
│   │   │   ├── pipeline1
│   │   │   │   ├── __init__.py
│   │   │   │   └── test_pipeline.py
│   │   │   ├── __init__.py
│   │   │   ├── pipeline2
│   │   │   │   ├── __init__.py
│   │   │   │   └── test_pipeline.py
│   │   │   ├── pipeline3
│   │   │   │   ├── __init__.py
│   │   │   │   └── test_pipeline.py
│   │   │   └── pipeline4
│   │   │       ├── __init__.py
│   │   │       └── test_pipeline.py
│   │   └── test_run.py
│   └── utilities
│       ├── data.py
│       ├── __init__.py
│       ├── model.py
│       ├── plotting.py
│       ├── testing.py
│       └── training.py
└── stats.json

from neptune-client.

I am not able to reproduce this error on a Windows machine too 😔

Would it be possible for you to share a minimal project (as a ZIP file, or a GitHub repo) that I can use to reproduce this?

from neptune-client.

Can you also try upgrading kedro to the latest version - 0.19.2?

from neptune-client.