PackMetric is a lightweight, easy-to-use metric management tool built on top of torchmetrics. This tool simplifies the process of integrating, updating, and managing metrics, also enabling the separation of evaluation and computation code in your ML projects. PackMetric can be seamlessly intergrated into a lightning project, and it also supports configuration via Hydra, offering a flexible and scalable way to manage metrics throughout the lifecycle of a model.
- DDP Support: Ensures metrics work correctly under distributed training conditions through
torchmetrics. - Seamless Integration with PyTorch Lightning: Adds easily into the PyTorch Lightning workflow as a callback.
- Code Separation: Facilitates the separation of metric evaluation from computation, enhancing code modularity.
- Hydra Configuration: Supports using Hydra for dynamic configuration of metrics and their management.
Install PackMetric via pip with the following command:
pip install packmetricUsing PackMetric in your projects involves constructing a MetricGroup and then integrating it within your PyTorch Lightning model, either with or without a callback. Here's how to do it:
You can create a MetricGroup directly in your code by instantiating it with your desired metrics from torchmetrics.
import packmetric
from packmetric import MetricGroup, BaseMetricAdapter, BaseMetaMetricAdapter, STAGE_TRAIN, STAGE_VAL, STAGE_TEST
from packmetric.utils.template import MeanMetricAdapter
from torchmetrics import Accuracy, MaxMetric
def custom_metric(x, y, some_parameter):
return x ** 2 + y * some_parameter - 1
# Define your metrics
accuracy = BaseMetricAdapter(name='acc',
metric_init_fn=lambda: Accuracy(task="multiclass", num_classes=4),
input_pos_args=['pred', 'target'],
stages=[STAGE_TRAIN, STAGE_VAL])
max_accuracy = BaseMetaMetricAdapter(name='max_acc',
metric_init_fn=MaxMetric,
input_pos_args=['acc'])
cm = MeanMetricAdapter('cm')
# Create a metric group
metric_group = MetricGroup(input_metrics=[accuracy, max_accuracy, cm])An example config could be as bellow. For more details, please refer to hydra.
metrics:
accuracy:
_target_: packmetric.BaseMetricAdapter
name: acc
metric_init_fn:
_target_: torchmetrics.Accuracy
task: "multiclass"
num_classes: 4
_partial_: true
input_pos_args: ["pred", "target"]
stages: ["train", "val"]
max_accuracy:
_target_: packmetric.BaseMetaMetricAdapter
name: max_acc
metric_init_fn:
_target_: torchmetrics.MaxMetric
_partial_: true
input_pos_args: ["acc"]
custom_metric:
_target_: packmetric.utils.template.MeanMetricAdapter
name: cmfrom typing import List
import hydra
from omegaconf import DictConfig
from packmetric import MGMetric, MetricGroup
def instantiate_metrics(metric_cfg: DictConfig) -> List[MGMetric]:
"""Instantiates metrics from config."""
metrics: List[MGMetric] = []
if not metric_cfg:
print("Metric config is empty.")
return metrics
if not isinstance(metric_cfg, DictConfig):
raise TypeError("Metric config must be a DictConfig!")
for _, m_conf in metric_cfg.items():
if isinstance(m_conf, DictConfig) and "_target_" in m_conf:
print(f'Instantiating metric "{m_conf.name}"')
metrics.append(hydra.utils.instantiate(m_conf))
return metrics
metric_cfg = ...
metric_group = MetricGroup(instantiate_metrics(metric_cfg))Once your MetricGroup is configured, you can integrate it directly into your training loops. Here’s an example of how to use MetricGroup to track metrics during training:
n_epoch = ...
dataloaders = ...
model = ...
metric_group.reset(level=packmetric.LEVEL_RUN)
for epoch in range(n_epoch):
for batch in dataloaders['train']:
y_hat = model(batch.x)
step_metrics = metric_group.batch_step(
{'pred': y_hat, 'target': batch.y, 'cm': custom_metric(batch.x, batch.y, batch.some_parameter)},
stage=STAGE_TRAIN
)
epoch_metrics = metric_group.epoch_step(STAGE_TRAIN)
print(epoch_metrics)from packmetric.utils.lightning import LogMetricsCallback
from pytorch_lightning import Trainer
# Initialize your model and MetricGroup as shown above
# Add the PackMetric callback
metric_callback = LogMetricsCallback(metric_group=metric_group)
# Create a trainer and pass the callback
trainer = Trainer(callbacks=[metric_callback])
trainer.fit(model, train_dataloader)
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent