lyken17 / pytorch-opcounter Goto Github PK
View Code? Open in Web Editor NEWCount the MACs / FLOPs of your PyTorch model.
License: MIT License
Count the MACs / FLOPs of your PyTorch model.
License: MIT License
When I used pip install thop, I got
Could not find a version that satisfies the requirement thop (from versions: )
When I installed thop from source and used it to resnet50 model, I got the following error message:
AttributeError Traceback (most recent call last)
in ()
----> 1 flops, params = profile(model, input_size=(1, 3, 224,224))
in profile(model, input_size, custom_ops, device)
73 x = torch.zeros(input_size).to(device)
74 with torch.no_grad():
---> 75 model(x)
76
77 total_ops = 0
/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py in call(self, *input, **kwargs)
475 result = self._slow_forward(*input, **kwargs)
476 else:
--> 477 result = self.forward(*input, **kwargs)
478 for hook in self._forward_hooks.values():
479 hook_result = hook(self, input, result)
/anaconda3/envs/pytorch/lib/python3.6/site-packages/torchvision/models/resnet.py in forward(self, x)
137
138 def forward(self, x):
--> 139 x = self.conv1(x)
140 x = self.bn1(x)
141 x = self.relu(x)
/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py in call(self, *input, **kwargs)
477 result = self.forward(*input, **kwargs)
478 for hook in self._forward_hooks.values():
--> 479 hook_result = hook(self, input, result)
480 if hook_result is not None:
481 raise RuntimeError(
in count_convNd(m, x, y)
10 batch_size = x.size(0)
11
---> 12 kernel_ops = m.weight.size()[2:].numel()
13 bias_ops = 1 if m.bias is not None else 0
14 ops_per_element = kernel_ops + bias_ops
AttributeError: 'torch.Size' object has no attribute 'numel'
When I measure FLOPs of a Convolution layer with 3x3 kernel, 128 output channels, (1, 1, 128, 128) input data, "VALID" padding, thop and Tensorflow give me different results.
Tensorflow: 38612737
Torch: 20321280.0
Here is my code:
graph = tf.Graph()
with graph.as_default():
print("* Initialize network")
v = tf.keras.layers.Conv2D(128, (3, 3), padding="VALID", data_format="channels_first")(tf.placeholder(tf.float32, (1, 1, 128, 128)))
print("Tensorflow:", tf.profiler.profile(graph, options=tf.profiler.ProfileOptionBuilder.float_operation()).total_float_ops)
print("Torch:", thop.profile(torch.nn.Conv2d(1, 128, (3, 3)), inputs=(torch.zeros((1, 1, 128, 128)),), verbose=False)[0])
Why?
Hi,
Thank you for this repo. when I use this repo to count flops and params on pytorch0.4.1. there is an error occur like:
File "~/site-packages/thop/count_hooks.py", line 20, in count_convNd
kernel_ops = m.weight.size()[2:].numel() # Kw x Kh
AttributeError: 'torch.Size' object has no attribute 'numel'
I only use conv2d so I change this line like:
kernel_ops = m.weight.size()[2]*m.weight.size()[3]
I think you can offer different pytorch-OpCounter Branch for different pytorch version if there is someone needs. Or remind others who are in need.
Anyway, thanks and BRs.
Hello, when I run your code. it appears the error"No Module "utils" and "denselink_imagenet"", how to solve it .Thank you !
Thanks!
Firstly, I sincerely thank you for THOP, a much needed product for the community. However, I would like to know how to find the FLOPs when nodes are dropped. A followup question would be, if we freeze the layers, the forward propagation will be executed but weights wont be updated during the back propagation. How to calculate FLOPs when layers are frozen.
Hello!
I want to use the code to count the Multi-Adds operations, but I don't know if the Multi-Adds operations and MACs and FLOPs are the same thing.
I'm confused about the definition of MACs and FLOPs, and I'm also not clearly know if the code you provided is to count the MACs or FLOPs .
Could you tell me the definition of them and the difference between them?
Thank you!
Hi, first of all, thank you for your code. It really helps my work.
I am curious about the flops for depthwise convolutional layer, which should be different from the standard conv.
But it seems there is no implementation for such layer in your code. Could you please add support for it?
Thanks!
I have a model which is needed to be profiled with inputs in different sizes. I tested the following example code in jupyter notebook, where the second run without reloading the model threw a warning and no result returned:
import sys
pyversion = sys.version.split(' ')[0]
print('current python version: {:s}'.format(pyversion))
>> current python version: 3.7.2
import torch
torchversion = torch.__version__
print('current pytorch version: {:s}'.format(torch.__version__))
import torchvision
from torchvision import models
print('current torchvision version: {:s}'.format(torchvision.__version__))
>> current pytorch version: 1.1.0.post2
>> current torchvision version: 0.3.0
# VGG16
vgg16 = models.vgg16()
from thop import profile
inp = torch.randn(1, 3, 224, 224, )
profile(vgg16, inputs=(inp, ))
>> .....(normal output, too long to list)
inp = torch.randn(1, 3, 448, 448, )
profile(vgg16, inputs=(inp, ))
---------------------------------------------------------------------------
Warning Traceback (most recent call last)
<ipython-input-4-9bd5a4920d3c> in <module>
1 inp = torch.randn(1, 3, 448, 448, )
----> 2 profile(vgg16, inputs=(inp, ))
/usr/local/lib/python3.7/site-packages/thop/profile.py in profile(model, inputs, custom_ops, verbose)
67
68 model.eval()
---> 69 model.apply(add_hooks)
70
71 with torch.no_grad():
/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py in apply(self, fn)
245 """
246 for module in self.children():
--> 247 module.apply(fn)
248 fn(self)
249 return self
/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py in apply(self, fn)
245 """
246 for module in self.children():
--> 247 module.apply(fn)
248 fn(self)
249 return self
/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py in apply(self, fn)
246 for module in self.children():
247 module.apply(fn)
--> 248 fn(self)
249 return self
250
/usr/local/lib/python3.7/site-packages/thop/profile.py in add_hooks(m)
38 if hasattr(m, "total_ops") or hasattr(m, "total_params"):
39 raise Warning("Either .total_ops or .total_params is already defined in %s.\n"
---> 40 "Be careful, it might change your code's behavior." % str(m))
41
42 m.register_buffer('total_ops', torch.zeros(1))
Warning: Either .total_ops or .total_params is already defined in Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)).
Be careful, it might change your code's behavior.
Since I have a lot of inputs in different sizes (like images in a dataset), it is much slower if I reload the model everytime an input comes in. Is this a PyTorch design feature or this repo's? Is there any workarounds?
Could you please give a closer explanation for the calculation of the AvgPool FLOPs? Despite I know that the impact of pooling layers is minimal, I'd like to understand the calculation. As I get from the code the formular for 2dAvgPooling you use is: (kernel_size + 1) * H_out * W_out * in_channels * out_channels
But if I consider the formular on the pytorch website I'd calculate them as follows:
For each kernel beeing applied on your input you have kernel_size * kernel_size - 1 additions plus one division plus one multiplication of your kernel_size inside the denominator. You apply out_channels kernels on in_channels input channels which result in a H_out * W_out output. This is the computation for all multiplications and additions. To be compliant with the regular understanding of MACs and FLOPs you devide by 2 as two MACs result in one FLOP. So finally I get the formular: FLOPs = (kernel_size * kernel_size - 1) * H_out * W_out * in_channels * out_channels / 2
Hi. What environment is needed to install THOP? When I run the command 'pip install thop', it shows 'ERROR: No matching distribution found for thop'. Thanks.
First of all, thanks to the author's selfless dedication, your code is well written!
Now 3D convolution is widely used. Have you considered supporting 3D convolution?
First, nice work. But I found something wrong when calculating flops in torch.nn.Conv2d, in count_hooks.py file, the out_w = y.size(2) // m.stride[0]
is wrong, it must be out_w = y.size(2)
. The same as out_h
.
So the flops result is less than standard ones.
I use the tools to count FLOPs for EXTD-pytorch
The FLOPs result is different from the result computed by EXTD-pytorch which has code for computing flops as follows:
def compute_flops(model, image_size):
import torch.nn as nn
flops = 0.
input_size = image_size
for m in model.modules():
if isinstance(m, nn.AvgPool2d) or isinstance(m, nn.MaxPool2d):
input_size = input_size / 2.
if isinstance(m, nn.Conv2d):
if m.groups == 1:
flop = (input_size[0] / m.stride[0] * input_size[1] / m.stride[1]) * m.kernel_size[0] ** 2 * m.in_channels * m.out_channels
else:
flop = (input_size[0] / m.stride[0] * input_size[1] / m.stride[1]) * m.kernel_size[0] ** 2 * ((m.in_channels/m.groups) * (m.out_channels/m.groups) * m.groups)
flops += flop
if m.stride[0] == 2: input_size = input_size / 2.
return flops / 1000000000., flops / 1000000
The result by OpCounter is 1.084G, while 11.15G by above code.
Input size is 640x640
Hi,
The tool you developed is very helpful.
But I am a little confused about the unit used in your result.
For example, here Params(M) should be the number of parameters, not the size?
And how about FLOPs(G), 1x10^9 flops the total forward and backward for a model? But the result you got is quite different from the tensorflow profiler's result. So, I am a little confused.
Hi, thank you so much for your awesome work! I run the following code:
from torchvision.models import resnet50 from thop import profile model = resnet50() flops, params = profile(model, input_size=(1, 3, 224,224)) print(flops, params)
But I get the output:
4142713856.0 25557032.0 ,
which is different from your results in the table of README. And I'am using pytorch 1.0.1.
Could you help me explain that? Thank you!
It seems that thop only support python3.
In python2.7, there will be syntax error.
thop-0.0.22-py2.7.egg/thop/count_hooks.py, line 119
kernel = torch.Tensor([*(x[0].shape[2:])]) // torch.Tensor(list((m.output_size,))).squeeze()
SyntaxError: invalid syntax
If I get my own network from some baseline models which is modified somewhat, and how can I use the thop, can you give me an example for reference, thanks a lot
kernel_ops = m.weight.size()[2:].numel() # Kw x Kh
AttributeError: 'torch.Size' object has no attribute 'numel'
The original output of profile is too ugly and it seems that you add a visualization tool in utils
pytorch-OpCounter/thop/utils.py
Lines 2 to 8 in 1f4ddb7
However, the clever_format
tool is not included in the __init__
so it can not be smoothly called. My recommendation is add an clever_format
in your profile
arguments, and make it True
by default.
pytorch-OpCounter/thop/profile.py
Lines 42 to 43 in 1f4ddb7
First of all, thanks to the author's selfless dedication, your code is well written!
I have implemented two versions of 3D CNN(separable 3D and P3D) and want to calculate their FLOPs. I found that the FLOPs of separable 3D is much smaller than P3D version using your code (283.237G FLOPs VS 3.246T FLOPs). But when I calculate the inference time averaging in 100 forwards, the P3D version is much faster (0.8194s VS 0.3823s). Is there any problem when calculating computation of 3D CNN?
Firstly thank you for your exellent job.
When I try to calculate the FLOPs of MixNet which uses swish as non-linearity. And swish equals to x*sigmoid(x). But there are no hooks for both sigmoid or swish. So would you consider to add hooks for sigmoid or swish?
I know it is a little complex since Sigmoid(x) = 1/(1+exp(x)). The problem is how to calculate the FLOPs of exponetial operation. Maybe we can use multinomial to appoximate it? According to Maclaurin Series, exp(x) = 1 + x + x^2/2! + x^3/3! + .. + x^n/n!. The real problem becomes how should I choose the length of the infinite sequence.
Any one can help?
Hi, thank you for your work! The output is number of params, how to convert it to memory usage just like in README
?
Thanks for your open source work,it's great!
I'm wondering how your bn layer flops is calculated,because your calculation result is not the same as the bn layer of ptflop package.
Also,compared to ptflops, I find you are not having to compute the relu layer flops.......
Hi
currently no counting method for LSTM layer! Please put it on road map.
thanks
Hi Lyken, thank you for your powerful tool, however, I met some difficult with it and get error as
'torch.Size' object has no attribute 'numel'
My environment is python 3.6 with torch 0.4.1.
It seems that the reuse of module is banned by THOP tools
pytorch-OpCounter/thop/profile.py
Lines 49 to 51 in 1f4ddb7
However, it is not quite reasonable. The re-use of ReLU-type module is very common, it can make your codes neater and causes no harm. e.g.
class Model(nn.Module):
def __init__(self):
super().__init__()
self.dropout = nn.Dropout(0.5)
self.relu = nn.ReLU()
self.lin1 = nn.Linear(4096, 4096)
self.lin2 = nn.Linear(4096, 4096)
self.lin3 = nn.Linear(4096, 4096)
def forward(self, x):
output = self.relu(self.lin1(x))
output = self.relu(self.lin2(output))
output = self.dropout(self.relu(self.lin3(output)))
....
Besides, I think the network legality check is not THOP's work to do. This kind of restriction is unnecessary and will cause troublesome Error raise as I can see.
It seems that the OpCounter doesn't take into account the skip connection structure.
My model has two inputs. How can I use the code for a multiple-input model? Thanks.
pytorch-OpCounter/thop/count_hooks.py
Line 15 in 1f4ddb7
pytorch-OpCounter/thop/count_hooks.py
Line 84 in 1f4ddb7
May I ask why do you disable all the maxpooling part of your codes?
pytorch-OpCounter/thop/profile.py
Lines 23 to 28 in 1f4ddb7
pytorch-OpCounter/thop/count_hooks.py
Lines 111 to 125 in 1f4ddb7
logs:
Either .total_ops or .total_params is already defined in Conv2d(3, 16, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False).Be careful, it might change your code's beh avior. Either .total_ops or .total_params is already defined in BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True).Be careful, it might change your code's behavior.
The bias of a Linear layer is not taken into account for the FLOPs computation, as shown below:
Example:
import torch
import torch.nn as nn
from thop import profile
class Simple_FC_Net(nn.Module):
def __init__(self, bias=True):
super(Simple_FC_Net, self).__init__()
self.fc = nn.Linear(in_features=17, out_features=49, bias=bias)
def forward(self, x):
x = self.fc(x)
return x
input_simple_fc = torch.randn(1, 17)
# Model without bias
print("Model without bias:")
model = Simple_FC_Net(bias=False)
flops, params = profile(model, inputs=(input_simple_fc,))
print("Flops: {}, Params: {}".format(flops, params))
# Model with bias
print("Model with bias:")
model = Simple_FC_Net(bias=True)
flops, params = profile(model, inputs=(input_simple_fc,))
print("Flops: {}, Params: {}".format(flops, params))
Output:
Model without bias:
Register FLOP counter for module Linear(in_features=17, out_features=49, bias=False)
Flops: 1617.0, Params: 833.0
Model with bias:
Register FLOP counter for module Linear(in_features=17, out_features=49, bias=True)
Flops: 1617.0, Params: 882.0
Beside the entire model's flops and params, is there anyway to output Flops and Params for each component of the network?
Traceback (most recent call last):
File "/home/gpu/chen/IMIXNet-PyTorch/code/get_Netpara.py", line 21, in
print("Torch:", thop.profile(torch.nn.Conv2d(1, 128, (3, 3)), inputs=(torch.zeros((1, 1, 128, 128)),), verbose=False)[0])
File "/home/gpu/.conda/envs/pytorch4/lib/python3.6/site-packages/thop/profile.py", line 92, in profile
model(*inputs)
File "/home/gpu/.conda/envs/pytorch4/lib/python3.6/site-packages/torch/nn/modules/module.py", line 479, in call
hook_result = hook(self, input, result)
File "/home/gpu/.conda/envs/pytorch4/lib/python3.6/site-packages/thop/count_hooks.py", line 20, in count_convNd
kernel_ops = m.weight.size()[2:].numel() # Kw x Kh
AttributeError: 'torch.Size' object has no attribute 'numel'
what‘s the problem here ?
the version of thop is wrong?
kernel = torch.Tensor([*(x[0].shape[2:])]) // torch.Tensor(list((m.output_size,))).squeeze()
^
SyntaxError: invalid syntax
Hi!When my convolution kernel value is only 1 and -1 or when my input value is only 1 and -1, will the GFLOPS of the convolution operation be small? Thanks!
In my model, nn.functional.interpolate
is used, whose computation should be also calculated in my case.
nn.functional.interpolate: count_upsample_bilinear
was appended in register_hooks
in profile.py
, and count_upsample_bilinear
was implemented in count_hooks.py
, but it did not work.
I also tried to print m_type
in add_hooks
in profile
, but the output did not include nn.functional.interpolate
. It seems that we cannot add hooks to some APIs like nn.functional.interpolate
which work differently from APIs from torch.nn
in terms of parameters and handler. I wonder if we can only wrap them into 3rd party modules to set FLOPs counters for them like the example in README.md.
Is there any workarounds introducing minimum modification to the model definition code?
like the title
It has multiple errors:
File "/home/jhy/mf_py3.5/pytorch-OpCounter-master/thop/count_hooks.py", line 14, in count_convNd
kernel_ops = m.weight.size()[2:].numel()
AttributeError: 'torch.Size' object has no attribute 'numel'
Hi, thanks for your great work. I found an error in the following codes:
pytorch-OpCounter/thop/count_hooks.py
Lines 66 to 71 in 41d9d15
ops_per_element
with two different methods. The first is kh * kw * cin, while the second is kh * kw * cin * cout. Obviously, the first one is right, but it's overwrited by the second one.I think the count_conv2d function is for MACC or Multiplications.
In this function, total_ops is calculated by K x K x Cin x Wout x Hout X Cout.
Isn't it for the MACC calculation?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.