Dynamic import resolution about nickel HOT 5 OPEN

@yannham I don't disagree. It is completely possibly to just create a tmp file or a symlink every time. But assuming I have many files to process, that means significant I/O overhead and many files to cleanup afterwards.

I agree that dynamic imports are useful in general, and can't always be just replaced with a static import and symlinks. I just wanted to make sure you knew about this simplest option first, which could fit the bill in some situations 🙂

Thanks for the example, @Jasha10. As always there's a tradeoff there between static and dynamic: while dynamic is more flexible, it also disallows a whole class of optimizations and static analysis. In general most compiled, structured languages don't allow dynamic imports (think Rust, OCaml, Haskell, Swift, C, C++, etc.). It's a bit different in scripting languages, as I reckon javascript has the import() function, and because in general in Nickel import is currently used both as "import a module/library" and "import data (read + parse)", and the latter is always possible to do dynamically in most languages out there.

The example given here isn't entirely compelling IMHO, as it trades a bit of boilerplate for a more fragile dynamic idiom (for example, it might be harder to get the LSP to work on dynamic import, it's also not clear how this interact with static typing, etc.). But I can see how it could be, on a larger example, or a different example when you might not want to touch the source of the config but still add files or datasets.

Dynamic imports aren't out of the question, but they need to be carefully designed - in particular I'm thinking right now that they should be separate from normal imports (if only using a std function or a slightly different syntax) so that normal imports are still guaranteed to be static, and dynamic imports are used only when needed (and it's obvious they're dynamic). In any case, I would say let's first finish the line of work around lazy imports (as dynamic imports have to be lazy anyway, it's a required first step) and then decide on a design.

from nickel.

Do you really need your file name data.json to be configurable? Because otherwise, you can import directly JSON, YAML and TOML file from within Nickel by just simply writing import "data.json". So, if you're ok to always provide a file named data.json (or copy/rename the actual source in a prep terminal command just before), you can do that.

#1694 (lazy imports), while not necessarily implying dynamic imports, are I believe a first step toward potentital dynamic imports - work is being done on this front in e.g. #1807.

from nickel.

I think dynamic imports could be useful. The hydra configuration system (to which I am a contributor) has a feature called the defaults list which is basically an import system that allows for some degree of dynamic behavior.

from nickel.

As a motivating example of how dynamic imports could be useful, I'll give a rough translation into nickel-lang of Hydra's specializing configuration example.

Let's say we want to configure a machine learning application that will train a given model on a given dataset. Suppose we have the following tree of files:

$ tree
.
├── config.ncl
├── dataset
│   ├── cifar10.ncl
│   └── imagenet.ncl
├── dataset_model
│   ├── cifar10_alexnet.ncl
│   ├── cifar10_resnet.ncl
│   ├── imagenet_alexnet.ncl
│   └── imagenet_resnet.ncl
└── model
    ├── alexnet.ncl
    └── resnet.ncl

3 directories, 9 files

Here cifar10.ncl would contain settings for the cifar10 dataset, resnet.ncl contains settings for the resnet model, and cifar10_resnet.ncl contains special settings that are only relevant when using the combination of cifar10 and resnet.

If dynamic imports were possible in nickel, we could write the following in config.ncl:

# config.ncl
{
  defaults = {
    dataset = "imagenet",
    model = "alexnet",
    dataset_model = "%{dataset}_%{model}",
  },
  local = {
    dataset = import "dataset/%{defaults.dataset}.ncl",
    model = import "model/%{defaults.model}.ncl",
    dataset_model = import "dataset_model/%{defaults.dataset_model}.ncl",
  },
  output = { dataset = local.dataset } & { model = local.model } & local.dataset_model,
}

Piping the configuration to the ML application then looks like this:

nickel eval config.ncl --field=output | python my_ml_application.py

And switching from alexnet to resnet becomes as simple as this, with all imports being automatically adjusted:

nickel eval config.ncl --field=output -- --override 'defaults.model="resnet"' | python my_ml_application.py

I think it's possible to achieve something similar using Nickel's if/else construct to decide which imports to merge based on the defaults settings. This would require a fair amount of boilerplate, however, and that boilerplate code would need to be updated if we e.g. want to add more models or more datasets to our file tree.

from nickel.

@yannham I don't disagree. It is completely possibly to just create a tmp file or a symlink every time. But assuming I have many files to process, that means significant I/O overhead and many files to cleanup afterwards.

from nickel.