Is your feature request related to a problem? Please describe.

The API for defining materializer nodes could be streamlined. Currently, materializer usage is very verbose and contains redundant information.

to.json(
  dependencies=["statistics"],
  id="statistics__json",  # <- `__FORMAT` or `.FORMAT` won't conflict with other nodes
  path=f"{materializer_path}/statistics.json",
)

# essentially
to.$FORMAT(
  dependencies=[$NODE_NAME]
  id=$NODE_NAME__$FORMAT
  path=$MATERIALIZER_PATH/$NODE_NAME.$FORMAT
)

Side problem: using from_ can point to inexistent files, which makes the DAG fail early, but to can also point to inexistent directories (or no write access) which fail at the end of the pipeline and requires a rerun.

Describe the solution you'd like

Instead, we could implement smart defaults to allow an API such as:

dr = (
  driver.Builder()
  .with_destination(...)
  ...
)

to.json("statistics")

# essentially
dr = (
  driver.Builder()
  .with_destination($MATERIALIZER_PATH)
  ...
)

to.$FORMAT($NODE_NAME)

Features

• .with_destination() sets a base path for materialization. At driver instantiation, paths can be resolved to absolute path and check if they exist or create them (should follow pathlib API docs). This also provide a simple interface to pass credentials to an S3 bucket via fsspec or a dlt Destination
• smart defaults for from_ and to won't conflict with the full API and will be backward compatible. It's slightly more challenging for materializers with combine=, but we could simply sort, concat, truncate the node names. The behavior would be implemented at the level of individual Saver/Loader to allow for example to.plt, which is a render engine, not a format, to work without issue

Hmm, the issue is that not every one has path or something equivalent-- the structure isn't there to unify it. That said, I agree that we should enable it to be part of the driver. Then we can just call execute, knowing that we've already appended saver nodes... And visualize it staticaly.

from hamilton.