birkhoffg / relax Goto Github PK

Plugin to #50

Plugin to #50

• #85
  

  ReLax/relax/data/module.py

  Line 404 in 729cacf

  def load_data(

• #86
• #87

We might want to provide a hook to the data_module for accessing cat_idx and cat_array.

https://arxiv.org/abs/1905.07697

Create a Jupyter Notebook to benchmark methods in Carla

Pytorch is only needed for loading data. Our library mainly handles tabular data, so data loading would not be a bottleneck to most scenarios. Pytorch Dataloader is overkill for our project in most use cases.

Purpose

Write a drop-in NumpyLoader.

ToDo

Delete the Pytorch Dependency

https://github.com/BirkhoffG/cfnet/blob/24783713dc787cd5b13e70aa483e455c4856198f/settings.ini#L18

https://github.com/BirkhoffG/cfnet/blob/24783713dc787cd5b13e70aa483e455c4856198f/cfnet/datasets.py#L9

Next, modify the following code to make them not inherent Pytorch Dataset and DataLoader:

https://github.com/BirkhoffG/cfnet/blob/24783713dc787cd5b13e70aa483e455c4856198f/cfnet/datasets.py#L12-L22

https://github.com/BirkhoffG/cfnet/blob/24783713dc787cd5b13e70aa483e455c4856198f/cfnet/datasets.py#L35-L51

Expected Functionalities

NumpyDataset should contain all the input data.

# x, y are jax.numpy.array, such that len(x) == len(y)
dataset = NumpyDataset(x, y)

x, y = dataset[:] # access all the data of x, y
x_5, y_5 = dataset[:5] # access first five data of x, y

NumpyLoader iterates the NumpyDataset. See Pytorch Docs.

batch_size = 128
dataloader = NumpyLoader(
    dataset, # a `NumpyDataset`
    batchsize=batch_size,
    shuffle=True, # if True, shuffle the data; else, return the data in order
    drop_last=False # if True, discard the last batch (if len(dataset) % batchsize != 0); else, return the last batch
)

for x, y in dataloader:
    assert len(x) == batch_size
    assert len(y) == batch_size
    ...

Move

• binary_cross_entropy in cfnet.methods.vanilla
• grad_update, cat_normalize in cfnet.training_module

into cfnet.utils

https://github.com/BirkhoffG/cfnet/blob/6d5c3ec28351de79e575bbded5e67ab1bab48b03/cfnet/datasets.py#L95

You can write something like:

 batch_data = self.dataset[self.indices] 

This line seems to slow the entire training

https://github.com/BirkhoffG/cfnet/blob/6d5c3ec28351de79e575bbded5e67ab1bab48b03/cfnet/module.py#L249-L250

Proposal

data_module = TabularDataModule('adult')

As such, TabularDataModule will automatically load data_configs of the adult dataset.

We should also allow TabularDataModule to pass user-defined configs (i.e., current argument data_configs: str | dict).

We don't support PredictiveTrainingModuleConfigs

There are some outdated links (i.e., some links are still written in cfnet). Please try to fix ALL of them to the correct links.

For example,
https://github.com/BirkhoffG/ReLax/blob/v0.1/relax/data/module.py#L415-L416

Seems to run some unnecessary tests (e.g., train some models) during the testing

Check monitor_metrics before actually finding the metric in logs.

Before this line:
https://github.com/BirkhoffG/cfnet/blob/2ee1a3203a9935e89b2ed8adf175ee1150fd9960/cfnet/_ckpt_manager.py#L54

Check monitor_metrics.

raise ValueError(...)

if monitor_metrics is not appropriately configured.

We use cat_normalize for encoding features, and clip continuous features to [0, 1]. This is because we use one-hot encoding for cat features, and min-max scalar for cont features.

If a user wants to use other encoding methods (e.g., standardized cont features), our current way of handing normalized data is not applicable.

Proposed features:

• #74
• #73

1. Pass seed and batch_size to TabularDataModule.train_dataloader, TabularDataModule.val_dataloader, and TabularDataModule.test_dataloader.

2. batch_size should also be an argument in TrainingConfigs
   https://github.com/BirkhoffG/cfnet/blob/2ee1a3203a9935e89b2ed8adf175ee1150fd9960/cfnet/train.py#L15

3. Deprecated batch_size in DataConfigs

4. Finally, pass appropriate arguments:
   https://github.com/BirkhoffG/cfnet/blob/2ee1a3203a9935e89b2ed8adf175ee1150fd9960/cfnet/train.py#L58-L59

For example, remove this function:

https://github.com/NVIDIA-Merlin/dataloader

Plugin to #50

Something like:

If monitor_metrics is None:
    # no checkpoint manage

Currently, we assume pred_fn is a function of only one input x. E.g., it is something like:

pred_fn = lambda x: 2 * x + 1

However, it is possible that user-defined pred_fn takes other arguments.

Hence, I propose

def generate_cf_explanations(
    cf_module: BaseCFModule,
    datamodule: TabularDataModule,
    pred_fn: Callable[[jnp.DeviceArray], jnp.DeviceArray] = None,
    *,
    t_configs=None,
    pred_fn_args: dict=None
)

where inside, we call pred_fn as

pred_fn(x, **pred_fn_args)

This offers additional flexibility for models that are not implemented using our framework.

Supporting hyper-parameter searching enables us to properly benchmark the algorithms. This issue is a thread discussing how to support hyperparameter searching in CF explanation methods.

In essence, this problem is a multi-objective problem (i.e., minimizing invalidity and cost).

Some open-sourced libraries of hyper-parameter searching:

• Optuna [doc]
• Ax [doc]
• Dragonfly [link]
• Spearmint [link]
• BoTorch [link]

https://arxiv.org/abs/1907.02584

Reference:
https://stackoverflow.com/a/68293931