Comments (11)
The above Traceback makes me think that you didn't add EncoderDecoder._get_loss
to skipped_methods
. Can you tell me if that is the case?
EDIT: I see, so EncoderDecider is not a submodule, sorry, if so, you'll need to refactor the loss function into not using .update
for dicts, since that is what makes it untraceable
EDIT 2: Or, alternatively, factor the dict handling out of the class and decorate it with @torch.fx.wrap
EDIT 3: You might also need to refactor and skip the refactored code from the decoder head and auxiliary head losses when they also handle dictionaries.
from mmrazor.
Thank you! Indeed, it works by refactoring the dict handling the batch preparation in respectively the
loss
andpredict
method of theEncoderDecoder
class and thepostprocess_result
method of the classBaseSegmentor
and decorating it with@torch.fx.wrap
. I also put theBaseDecodeHead.predict_by_feat
, thePSPHead.loss_by_feat
and theFCNHead.loss
methods in theskipped_method
argument.
Do make sure that the FCNHead.loss
doesn't have any nodes in common i
with EncoderDecoder.forward
, or the fake quants won't calibrate correctly.
What is the difference between the use of the
@torch.fx.wrap
decorator and theskipped_method
argument if both try to handle untraceable code? When using one instead of the other?
@torch.fx.wrap
is mainly for functions, and I use it for things that either repeat across classes or that are on the root class I'm trying to trace. In contrast, skipped_methods
works only on submodule methods, but theoretically if you can skip it without refactoring it is more convenient.
from mmrazor.
This might be tangentially related to what I encountered in the mmpose TopdownEstimator in issue #3012
You might need to refactor the model so that there is no self-referencing methods within it, and instead point to wrapped outer methods.
I haven't checked if thats the case for mmseg but it might point you in the right direction.
from mmrazor.
Hi, I have the same problem with the class EncoderDecoder from the segmentors of MMSegmentation (line 208). Did you manage to refactor your model and how?
from mmrazor.
Hi, I have the same problem with the class EncoderDecoder from the segmentors of MMSegmentation (line 208). Did you manage to refactor your model and how?
Yes, I haven't posted an issue yet, but you should mimic the structure in mmpretrain.models.heads.cls_head.ClsHead where there is an additional _get_loss
and _get_predict
that handle all the untraceable methods, and only trace the code where forward is being called on the input.
from mmrazor.
Thank you. I have changed the following argument of the MMRazor CustomTracer to fit with the EncoderDecoder class:
skipped_methods=[
'mmseg.models.decode_heads.decode_head.BaseDecodeHead.predict_by_feat',
'mmseg.models.decode_heads.decode_head.BaseDecodeHead.loss_by_feat']
Both auxiliary head (FCNHead) and decode head (PSPHead) use the the same predict and loss functions.
Moreover, I have take the whole code of the EncoderDecoder predict method out of the class (except from the self.inference() call), by creating functions with a @torch.fx.wrap decorator.
def predict(self,
inputs: Tensor,
data_samples: OptSampleList = None) -> SampleList:
"""Predict results from a batch of inputs and data samples with post-
processing.
Args:
inputs (Tensor): Inputs with shape (N, C, H, W).
data_samples (List[:obj:`SegDataSample`], optional): The seg data
samples. It usually includes information such as `metainfo`
and `gt_sem_seg`.
Returns:
list[:obj:`SegDataSample`]: Segmentation results of the
input images. Each SegDataSample usually contain:
- ``pred_sem_seg``(PixelData): Prediction of semantic segmentation.
- ``seg_logits``(PixelData): Predicted logits of semantic
segmentation before normalization.
"""
batch_img_metas = _prepare_batch(inputs, data_samples)
seg_logits = self.inference(inputs, batch_img_metas)
return postprocess_result(self.decode_head, seg_logits, data_samples)
The problem now is when calling the EncoderDecoder loss function, it calls the EncoderDecoder _decode_head_forward_train and _auxiliary_head_forward_train functions which try to update a dictionnary of losses. I can't make the same changes you have made in mmpose TopdownEstimator for the loss function, as the latter two functions update the dictionnary.
Do I have to pass the EncoderDecoder loss function entirely to skipped_methods
, or is this a bigger issue?
Here is the full log of the issue:
/opt/conda/lib/python3.10/site-packages/mmseg/models/backbones/resnet.py:431: UserWarning: DeprecationWarning: pretrained is a deprecated, please use "init_cfg" instead
warnings.warn('DeprecationWarning: pretrained is a deprecated, '
/opt/conda/lib/python3.10/site-packages/mmseg/models/builder.py:36: UserWarning: ``build_loss`` would be deprecated soon, please use ``mmseg.registry.MODELS.build()``
warnings.warn('``build_loss`` would be deprecated soon, please use '
/opt/conda/lib/python3.10/site-packages/mmseg/models/losses/cross_entropy_loss.py:235: UserWarning: Default ``avg_non_ignore`` is False, if you would like to ignore the certain label and average loss over non-ignore labels, which is the same with PyTorch official cross_entropy, set ``avg_non_ignore=True``.
warnings.warn(
Loads checkpoint by local backend from path: /workspace/mmlab/MMR/qat/seg/pspnet_r18-d8_512x1024_80k_cityscapes_20201225_021458-09ffa746.pth
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/XXX/.vscode-server/extensions/ms-python.debugpy-2024.0.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 39, in <module>
cli.main()
File "/home/XXX/.vscode-server/extensions/ms-python.debugpy-2024.0.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main
run()
File "/home/XXX/.vscode-server/extensions/ms-python.debugpy-2024.0.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file
runpy.run_path(target, run_name="__main__")
File "/home/XXX/.vscode-server/extensions/ms-python.debugpy-2024.0.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 321, in run_path
return _run_module_code(code, init_globals, run_name,
File "/home/XXX/.vscode-server/extensions/ms-python.debugpy-2024.0.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 135, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "/home/XXX/.vscode-server/extensions/ms-python.debugpy-2024.0.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
exec(code, run_globals)
File "/workspace/mmlab/mmrazor/tools/train.py", line 121, in <module>
main()
File "/workspace/mmlab/mmrazor/tools/train.py", line 114, in main
runner = Runner.from_cfg(cfg)
File "/opt/conda/lib/python3.10/site-packages/mmengine/runner/runner.py", line 462, in from_cfg
runner = cls(
File "/opt/conda/lib/python3.10/site-packages/mmengine/runner/runner.py", line 429, in __init__
self.model = self.build_model(model)
File "/opt/conda/lib/python3.10/site-packages/mmengine/runner/runner.py", line 836, in build_model
model = MODELS.build(model)
File "/opt/conda/lib/python3.10/site-packages/mmengine/registry/registry.py", line 570, in build
return self.build_func(cfg, *args, **kwargs, registry=self)
File "/opt/conda/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 232, in build_model_from_cfg
return build_from_cfg(cfg, registry, default_args)
File "/opt/conda/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg
obj = obj_cls(**args) # type: ignore
File "/workspace/mmlab/mmrazor/mmrazor/models/algorithms/quantization/mm_architecture.py", line 90, in __init__
self.qmodels = self._build_qmodels(self.architecture)
File "/workspace/mmlab/mmrazor/mmrazor/models/algorithms/quantization/mm_architecture.py", line 300, in _build_qmodels
observed_module = self.quantizer.prepare(model, concrete_args)
File "/workspace/mmlab/mmrazor/mmrazor/models/quantizers/native_quantizer.py", line 231, in prepare
traced_graph = self.tracer.trace(model, concrete_args=concrete_args)
File "/workspace/mmlab/mmrazor/mmrazor/models/task_modules/tracer/fx/custom_tracer.py", line 422, in trace
'output', (self.create_arg(fn(*args)), ), {},
File "/opt/conda/lib/python3.10/site-packages/mmseg/models/segmentors/base.py", line 94, in forward
return self.loss(inputs, data_samples)
File "/opt/conda/lib/python3.10/site-packages/mmseg/models/segmentors/encoder_decoder.py", line 179, in loss
loss_decode = self._decode_head_forward_train(x, data_samples)
File "/opt/conda/lib/python3.10/site-packages/mmseg/models/segmentors/encoder_decoder.py", line 143, in _decode_head_forward_train
losses.update(add_prefix(loss_decode, 'decode'))
File "/opt/conda/lib/python3.10/site-packages/mmseg/utils/misc.py", line 24, in add_prefix
for name, value in inputs.items():
File "/opt/conda/lib/python3.10/site-packages/torch/fx/proxy.py", line 274, in __iter__
return self.tracer.iter(self)
File "/opt/conda/lib/python3.10/site-packages/torch/fx/proxy.py", line 183, in iter
raise TraceError('Proxy object cannot be iterated. This can be '
torch.fx.proxy.TraceError: Proxy object cannot be iterated. This can be attempted when the Proxy is used in a loop or as a *args or **kwargs function argument. See the torch.fx docs on pytorch.org for a more detailed explanation of what types of control flow can be traced, and check out the Proxy docstring for help troubleshooting Proxy iteration errors
from mmrazor.
Passing the entire loss function to skipped_methods
will disallow the fake quantize observers to be calibrated, but anything that is inside the loss function that is not calling the head forward call can be refactored in another method which you can then skip. Basically you want the tracer to trace all nodes that are common between forward, predict, and loss, but not anything else necessarily.
In this case something like this should work:
def _get_loss(self, x: Tensor, data_samples: SampleList) -> dict:
"""Calculate losses from a batch of inputs and data samples.
Args:
x (Tensor): forward call result.
data_samples (list[:obj:`SegDataSample`]): The seg data samples.
It usually includes information such as `metainfo` and
`gt_sem_seg`.
Returns:
dict[str, Tensor]: a dictionary of loss components
"""
losses = dict()
loss_decode = self._decode_head_forward_train(x, data_samples)
losses.update(loss_decode)
if self.with_auxiliary_head:
loss_aux = self._auxiliary_head_forward_train(x, data_samples)
losses.update(loss_aux)
return losses
def loss(self, inputs: Tensor, data_samples: SampleList) -> dict:
"""Calculate losses from a batch of inputs and data samples.
Args:
inputs (Tensor): Input images.
data_samples (list[:obj:`SegDataSample`]): The seg data samples.
It usually includes information such as `metainfo` and
`gt_sem_seg`.
Returns:
dict[str, Tensor]: a dictionary of loss components
"""
x = self.extract_feat(inputs)
losses = self._get_loss(x, data_samples)
return losses
with a config that skips _get_loss
from mmrazor.
Thank you for your answer. Unfortunately, this doesn't work (see traceback below). It seems to be a malfunction in the trace function when dealing with the 'loss' mode.
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/XXX/.vscode-server/extensions/ms-python.debugpy-2024.0.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 39, in <module>
cli.main()
File "/home/XXX/.vscode-server/extensions/ms-python.debugpy-2024.0.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main
run()
File "/home/XXX/.vscode-server/extensions/ms-python.debugpy-2024.0.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file
runpy.run_path(target, run_name="__main__")
File "/home/XXX/.vscode-server/extensions/ms-python.debugpy-2024.0.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 321, in run_path
return _run_module_code(code, init_globals, run_name,
File "/home/XXX/.vscode-server/extensions/ms-python.debugpy-2024.0.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 135, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "/home/XXX/.vscode-server/extensions/ms-python.debugpy-2024.0.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
exec(code, run_globals)
File "/workspace/mmlab/mmrazor/tools/train.py", line 121, in <module>
main()
File "/workspace/mmlab/mmrazor/tools/train.py", line 114, in main
runner = Runner.from_cfg(cfg)
File "/opt/conda/lib/python3.10/site-packages/mmengine/runner/runner.py", line 462, in from_cfg
runner = cls(
File "/opt/conda/lib/python3.10/site-packages/mmengine/runner/runner.py", line 429, in __init__
self.model = self.build_model(model)
File "/opt/conda/lib/python3.10/site-packages/mmengine/runner/runner.py", line 836, in build_model
model = MODELS.build(model)
File "/opt/conda/lib/python3.10/site-packages/mmengine/registry/registry.py", line 570, in build
return self.build_func(cfg, *args, **kwargs, registry=self)
File "/opt/conda/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 232, in build_model_from_cfg
return build_from_cfg(cfg, registry, default_args)
File "/opt/conda/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg
obj = obj_cls(**args) # type: ignore
File "/workspace/mmlab/mmrazor/mmrazor/models/algorithms/quantization/mm_architecture.py", line 90, in __init__
self.qmodels = self._build_qmodels(self.architecture)
File "/workspace/mmlab/mmrazor/mmrazor/models/algorithms/quantization/mm_architecture.py", line 300, in _build_qmodels
observed_module = self.quantizer.prepare(model, concrete_args)
File "/workspace/mmlab/mmrazor/mmrazor/models/quantizers/native_quantizer.py", line 231, in prepare
traced_graph = self.tracer.trace(model, concrete_args=concrete_args)
File "/workspace/mmlab/mmrazor/mmrazor/models/task_modules/tracer/fx/custom_tracer.py", line 422, in trace
'output', (self.create_arg(fn(*args)), ), {},
File "/opt/conda/lib/python3.10/site-packages/mmseg/models/segmentors/base.py", line 94, in forward
return self.loss(inputs, data_samples)
File "/opt/conda/lib/python3.10/site-packages/mmseg/models/segmentors/encoder_decoder.py", line 205, in loss
losses = self._get_loss(x, data_samples)
File "/workspace/mmlab/mmrazor/mmrazor/models/task_modules/tracer/fx/custom_tracer.py", line 72, in wrapped_method
return self.tracer.call_method(mod, self.name, method, args,
File "/workspace/mmlab/mmrazor/mmrazor/models/task_modules/tracer/fx/custom_tracer.py", line 317, in call_method
return self.create_proxy('call_method', name, args, kwargs)
File "/opt/conda/lib/python3.10/site-packages/torch/fx/proxy.py", line 66, in create_proxy
args_ = self.create_arg(args)
File "/opt/conda/lib/python3.10/site-packages/torch/fx/_symbolic_trace.py", line 344, in create_arg
return super().create_arg(a)
File "/opt/conda/lib/python3.10/site-packages/torch/fx/proxy.py", line 140, in create_arg
return type(a)(self.create_arg(elem) for elem in a)
File "/opt/conda/lib/python3.10/site-packages/torch/fx/proxy.py", line 140, in <genexpr>
return type(a)(self.create_arg(elem) for elem in a)
File "/opt/conda/lib/python3.10/site-packages/torch/fx/_symbolic_trace.py", line 298, in create_arg
return self.create_node("get_attr", n_, (), {})
File "/opt/conda/lib/python3.10/site-packages/torch/ao/quantization/fx/tracer.py", line 114, in create_node
node = super().create_node(kind, target, args, kwargs, name, type_expr)
File "/opt/conda/lib/python3.10/site-packages/torch/fx/proxy.py", line 46, in create_node
return self.graph.create_node(kind, target, args, kwargs, name, type_expr)
File "/opt/conda/lib/python3.10/site-packages/torch/fx/graph.py", line 777, in create_node
name = self._graph_namespace.create_name(candidate, None)
File "/opt/conda/lib/python3.10/site-packages/torch/fx/graph.py", line 137, in create_name
if candidate[0].isdigit():
IndexError: string index out of range
When the trace function of CustomTracer is called, it calls the create_arg method of torch fx for the forward method of EncoderDecoder and several of its modules. However, one of these modules is the EncoderDecoder itself (not submodules), which should not. It enters in create_arg and crashes in this condition because the EncoderDecoder module has no name n_
(empty string).
I think the problem comes from the fact that the _get_loss function is still in the EncoderDecoder class: this makes the EncoderDecoder model appear in the arguments of the create_arg method. I had the same issue and traceback with the tracing of the 'predict' mode and I made some changes (see in this comment). I take the _prepare_batch
and postprocess_result
functions out of the class and put the @torch.fx.wrap decorator on top, which enables the tracing for the 'predict' mode.
from mmrazor.
Thank you! Indeed, it works by refactoring the dict handling the batch preparation in respectively the loss
and predict
method of the EncoderDecoder
class and the postprocess_result
method of the class BaseSegmentor
and decorating it with @torch.fx.wrap
. I also put the BaseDecodeHead.predict_by_feat
, the PSPHead.loss_by_feat
and the FCNHead.loss
methods in the skipped_method
argument.
What is the difference between the use of the @torch.fx.wrap
decorator and the skipped_method
argument if both try to handle untraceable code? When using one instead of the other?
from mmrazor.
How do you check if methods have nodes in common? FCNHead
and PSPHead
both inherit from the same loss method in the BaseDecodeHead
class, that only do the forward of the head and the computation of the loss. But these heads are submodules inside the EncoderDecoder
model.
from mmrazor.
Anything that has a forward calculation would need to not be skipped. one way to check is adding a printout of the JIT graph within mmrazor's CustomTracer
from mmrazor.
Related Issues (20)
- I can't reproduce dfad results
- How to get started??
- [Bug] TypeError: 'NoneType' object is not iterable
- Try to reproduce CWD in VOC data set
- [Bug] (suggested temporary fix) Pytorch >= 2 causes mmrazor.engine to fail HOT 4
- [Bug] (suggested fix) `nn.Parameter` are not added to root after being traced in `mmrazor.models.task_utils.tracer.fx.custom_tracer.build_graphmodule()` HOT 2
- [Bug] (suggested fix) `mmrazor.models.algorithms.quantization.mm_architecture.MMArchitectureQuant.sync_qparams()` fails if there are modules present in other modes but not in forward `mode='tensor'` HOT 4
- I want to obtain the current epoch value and associate it with the custom distillation loss
- cannot use recorder to obtain panoptic_head info from mask2former
- [Bug] `mmrazor.engine.runner.quantization_loops.QATValLoop` calls `after_val_epoch` hook twice with different keys, causing `mmengine.hooks.checkpoint_hook._save_best_checkpoint()` to fail with `KeyError` for the `save_best` config
- [Bug] Custom Distillation MMSeg CWD loss nan problem
- When I use methodoutputs to access the results of assigner, I only obtain one sample
- Regarding tables and accuracy
- [Bug] (suggested fix) `mmrazor.models.algorithms.mm_architecture.MMArchitectureQuant.get_deploy_model()` fails if `predict` mode lacks nodes from the `model.quantizer.tracer.skipped_methods` configuration, but the architecture `quantizer.prepare(fp32_model)` has these nodes. HOT 4
- Is this a dead project ? HOT 1
- 我在用mmrazor通过yolov5-x蒸馏yolov5-s时候遇到了问题 HOT 1
- No Sign of activation quantization with QAT HOT 1
- MAP is stucked at 0 for Mobilenet V2 SSD QAT without pretrained model [Bug]
- [Docs] A100算力加持!书生大模型实战营第3期全面升级,趣味闯关模式等你开启
- Missing keys after RTMDET knowledge distillation HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mmrazor.