Unused operations seem to be included in FLOPS calculation about torchprofile HOT 8 CLOSED

Could you please set reduction to None to show the computation for all the modules?

from torchprofile.

Sure, here's the result of execution of line 3 of my original message, but with no reduction (profile_macs(copy.deepcopy(self.engine.get_active_subnet(True)), inputs.cuda(), reduction=None))

{%input.2: float[1, 16, 96, 96] = aten::_convolution(%input.1: float[1, 3, 192, 192], %1: float[16, 3, 3, 3], %696: none, %699: list[int], %702: list[int], %705: list[int], %706: bool, %709: list[int], %710: int, %711: bool, %712: bool, %713: bool, %714: bool): 3981312, %739: float[1, 16, 96, 96] = aten::batch_norm(%input.2: float[1, 16, 96, 96], %2: float[16], %3: float[16], %4: float[16], %5: float[16], %735: bool, %736: float, %737: float, %738: bool): 0, %746: float[1, 16, 96, 96] = aten::mul(%739: float[1, 16, 96, 96], %745: float[1, 16, 96, 96]): 147456, %input.5: float[1, 16, 96, 96] = aten::_convolution(%input.4: float[1, 16, 96, 96], %7: float[16, 1, 3, 3], %749: none, %752: list[int], %755: list[int], %758: list[int], %759: bool, %762: list[int], %763: int, %764: bool, %765: bool, %766: bool, %767: bool): 1327104, %input.6: float[1, 16, 96, 96] = aten::batch_norm(%input.5: float[1, 16, 96, 96], %8: float[16], %9: float[16], %10: float[16], %11: float[16], %788: bool, %789: float, %790: float, %791: bool): 0, %input.8: float[1, 16, 96, 96] = aten::_convolution(%input.7: float[1, 16, 96, 96], %13: float[16, 16, 1, 1], %794: none, %797: list[int], %800: list[int], %803: list[int], %804: bool, %807: list[int], %808: int, %809: bool, %810: bool, %811: bool, %812: bool): 2359296, %837: float[1, 16, 96, 96] = aten::batch_norm(%input.8: float[1, 16, 96, 96], %14: float[16], %15: float[16], %16: float[16], %17: float[16], %833: bool, %834: float, %835: float, %836: bool): 0, %input.10: float[1, 48, 96, 96] = aten::_convolution(%input.9: float[1, 16, 96, 96], %19: float[48, 16, 1, 1], %840: none, %843: list[int], %846: list[int], %849: list[int], %850: bool, %853: list[int], %854: int, %855: bool, %856: bool, %857: bool, %858: bool): 7077888, %input.11: float[1, 48, 96, 96] = aten::batch_norm(%input.10: float[1, 48, 96, 96], %20: float[48], %21: float[48], %22: float[48], %23: float[48], %879: bool, %880: float, %881: float, %882: bool): 0, %input.13: float[1, 48, 48, 48] = aten::_convolution(%input.12: float[1, 48, 96, 96], %25: float[48, 1, 3, 3], %885: none, %888: list[int], %891: list[int], %894: list[int], %895: bool, %898: list[int], %899: int, %900: bool, %901: bool, %902: bool, %903: bool): 995328, %input.14: float[1, 48, 48, 48] = aten::batch_norm(%input.13: float[1, 48, 48, 48], %26: float[48], %27: float[48], %28: float[48], %29: float[48], %924: bool, %925: float, %926: float, %927: bool): 0, %input.16: float[1, 24, 48, 48] = aten::_convolution(%input.15: float[1, 48, 48, 48], %31: float[24, 48, 1, 1], %930: none, %933: list[int], %936: list[int], %939: list[int], %940: bool, %943: list[int], %944: int, %945: bool, %946: bool, %947: bool, %948: bool): 2654208, %input.17: float[1, 24, 48, 48] = aten::batch_norm(%input.16: float[1, 24, 48, 48], %32: float[24], %33: float[24], %34: float[24], %35: float[24], %969: bool, %970: float, %971: float, %972: bool): 0, %input.18: float[1, 72, 48, 48] = aten::_convolution(%input.17: float[1, 24, 48, 48], %37: float[72, 24, 1, 1], %974: none, %977: list[int], %980: list[int], %983: list[int], %984: bool, %987: list[int], %988: int, %989: bool, %990: bool, %991: bool, %992: bool): 3981312, %input.19: float[1, 72, 48, 48] = aten::batch_norm(%input.18: float[1, 72, 48, 48], %38: float[72], %39: float[72], %40: float[72], %41: float[72], %1013: bool, %1014: float, %1015: float, %1016: bool): 0, %input.21: float[1, 72, 48, 48] = aten::_convolution(%input.20: float[1, 72, 48, 48], %43: float[72, 1, 3, 3], %1019: none, %1022: list[int], %1025: list[int], %1028: list[int], %1029: bool, %1032: list[int], %1033: int, %1034: bool, %1035: bool, %1036: bool, %1037: bool): 1492992, %input.22: float[1, 72, 48, 48] = aten::batch_norm(%input.21: float[1, 72, 48, 48], %44: float[72], %45: float[72], %46: float[72], %47: float[72], %1058: bool, %1059: float, %1060: float, %1061: bool): 0, %input.24: float[1, 24, 48, 48] = aten::_convolution(%input.23: float[1, 72, 48, 48], %49: float[24, 72, 1, 1], %1064: none, %1067: list[int], %1070: list[int], %1073: list[int], %1074: bool, %1077: list[int], %1078: int, %1079: bool, %1080: bool, %1081: bool, %1082: bool): 3981312, %1107: float[1, 24, 48, 48] = aten::batch_norm(%input.24: float[1, 24, 48, 48], %50: float[24], %51: float[24], %52: float[24], %53: float[24], %1103: bool, %1104: float, %1105: float, %1106: bool): 0, %input.26: float[1, 72, 48, 48] = aten::_convolution(%input.25: float[1, 24, 48, 48], %55: float[72, 24, 1, 1], %1110: none, %1113: list[int], %1116: list[int], %1119: list[int], %1120: bool, %1123: list[int], %1124: int, %1125: bool, %1126: bool, %1127: bool, %1128: bool): 3981312, %input.27: float[1, 72, 48, 48] = aten::batch_norm(%input.26: float[1, 72, 48, 48], %56: float[72], %57: float[72], %58: float[72], %59: float[72], %1149: bool, %1150: float, %1151: float, %1152: bool): 0, %input.29: float[1, 72, 24, 24] = aten::_convolution(%input.28: float[1, 72, 48, 48], %61: float[72, 1, 3, 3], %1155: none, %1158: list[int], %1161: list[int], %1164: list[int], %1165: bool, %1168: list[int], %1169: int, %1170: bool, %1171: bool, %1172: bool, %1173: bool): 373248, %input.30: float[1, 72, 24, 24] = aten::batch_norm(%input.29: float[1, 72, 24, 24], %62: float[72], %63: float[72], %64: float[72], %65: float[72], %1194: bool, %1195: float, %1196: float, %1197: bool): 0, %1204: float[1, 72, 24, 1] = aten::mean(%1199: float[1, 72, 24, 24], %1201: list[int], %1202: bool, %1203: none): 1728, %input.31: float[1, 72, 1, 1] = aten::mean(%1204: float[1, 72, 24, 1], %1206: list[int], %1207: bool, %1208: none): 72, %input.32: float[1, 24, 1, 1] = aten::_convolution(%input.31: float[1, 72, 1, 1], %67: float[24, 72, 1, 1], %68: float[24], %1212: list[int], %1215: list[int], %1218: list[int], %1219: bool, %1222: list[int], %1223: int, %1224: bool, %1225: bool, %1226: bool, %1227: bool): 1728, %1248: float[1, 72, 1, 1] = aten::_convolution(%input.33: float[1, 24, 1, 1], %69: float[72, 24, 1, 1], %70: float[72], %1232: list[int], %1235: list[int], %1238: list[int], %1239: bool, %1242: list[int], %1243: int, %1244: bool, %1245: bool, %1246: bool, %1247: bool): 1728, %input.35: float[1, 72, 24, 24] = aten::mul(%1199: float[1, 72, 24, 24], %1256: float[1, 72, 1, 1]): 41472, %input.36: float[1, 40, 24, 24] = aten::_convolution(%input.35: float[1, 72, 24, 24], %71: float[40, 72, 1, 1], %1258: none, %1261: list[int], %1264: list[int], %1267: list[int], %1268: bool, %1271: list[int], %1272: int, %1273: bool, %1274: bool, %1275: bool, %1276: bool): 1658880, %input.37: float[1, 40, 24, 24] = aten::batch_norm(%input.36: float[1, 40, 24, 24], %72: float[40], %73: float[40], %74: float[40], %75: float[40], %1297: bool, %1298: float, %1299: float, %1300: bool): 0, %input.38: float[1, 120, 24, 24] = aten::_convolution(%input.37: float[1, 40, 24, 24], %77: float[120, 40, 1, 1], %1302: none, %1305: list[int], %1308: list[int], %1311: list[int], %1312: bool, %1315: list[int], %1316: int, %1317: bool, %1318: bool, %1319: bool, %1320: bool): 2764800, %input.39: float[1, 120, 24, 24] = aten::batch_norm(%input.38: float[1, 120, 24, 24], %78: float[120], %79: float[120], %80: float[120], %81: float[120], %1341: bool, %1342: float, %1343: float, %1344: bool): 0, %input.41: float[1, 120, 24, 24] = aten::_convolution(%input.40: float[1, 120, 24, 24], %83: float[120, 1, 3, 3], %1347: none, %1350: list[int], %1353: list[int], %1356: list[int], %1357: bool, %1360: list[int], %1361: int, %1362: bool, %1363: bool, %1364: bool, %1365: bool): 622080, %input.42: float[1, 120, 24, 24] = aten::batch_norm(%input.41: float[1, 120, 24, 24], %84: float[120], %85: float[120], %86: float[120], %87: float[120], %1386: bool, %1387: float, %1388: float, %1389: bool): 0, %1396: float[1, 120, 24, 1] = aten::mean(%1391: float[1, 120, 24, 24], %1393: list[int], %1394: bool, %1395: none): 2880, %input.43: float[1, 120, 1, 1] = aten::mean(%1396: float[1, 120, 24, 1], %1398: list[int], %1399: bool, %1400: none): 120, %input.44: float[1, 32, 1, 1] = aten::_convolution(%input.43: float[1, 120, 1, 1], %89: float[32, 120, 1, 1], %90: float[32], %1404: list[int], %1407: list[int], %1410: list[int], %1411: bool, %1414: list[int], %1415: int, %1416: bool, %1417: bool, %1418: bool, %1419: bool): 3840, %1440: float[1, 120, 1, 1] = aten::_convolution(%input.45: float[1, 32, 1, 1], %91: float[120, 32, 1, 1], %92: float[120], %1424: list[int], %1427: list[int], %1430: list[int], %1431: bool, %1434: list[int], %1435: int, %1436: bool, %1437: bool, %1438: bool, %1439: bool): 3840, %input.47: float[1, 120, 24, 24] = aten::mul(%1391: float[1, 120, 24, 24], %1448: float[1, 120, 1, 1]): 69120, %input.48: float[1, 40, 24, 24] = aten::_convolution(%input.47: float[1, 120, 24, 24], %93: float[40, 120, 1, 1], %1450: none, %1453: list[int], %1456: list[int], %1459: list[int], %1460: bool, %1463: list[int], %1464: int, %1465: bool, %1466: bool, %1467: bool, %1468: bool): 2764800, %1493: float[1, 40, 24, 24] = aten::batch_norm(%input.48: float[1, 40, 24, 24], %94: float[40], %95: float[40], %96: float[40], %97: float[40], %1489: bool, %1490: float, %1491: float, %1492: bool): 0, %input.50: float[1, 120, 24, 24] = aten::_convolution(%input.49: float[1, 40, 24, 24], %99: float[120, 40, 1, 1], %1496: none, %1499: list[int], %1502: list[int], %1505: list[int], %1506: bool, %1509: list[int], %1510: int, %1511: bool, %1512: bool, %1513: bool, %1514: bool): 2764800, %1539: float[1, 120, 24, 24] = aten::batch_norm(%input.50: float[1, 120, 24, 24], %100: float[120], %101: float[120], %102: float[120], %103: float[120], %1535: bool, %1536: float, %1537: float, %1538: bool): 0, %1546: float[1, 120, 24, 24] = aten::mul(%1539: float[1, 120, 24, 24], %1545: float[1, 120, 24, 24]): 69120, %input.53: float[1, 120, 12, 12] = aten::_convolution(%input.52: float[1, 120, 24, 24], %105: float[120, 1, 3, 3], %1549: none, %1552: list[int], %1555: list[int], %1558: list[int], %1559: bool, %1562: list[int], %1563: int, %1564: bool, %1565: bool, %1566: bool, %1567: bool): 155520, %1592: float[1, 120, 12, 12] = aten::batch_norm(%input.53: float[1, 120, 12, 12], %106: float[120], %107: float[120], %108: float[120], %109: float[120], %1588: bool, %1589: float, %1590: float, %1591: bool): 0, %1599: float[1, 120, 12, 12] = aten::mul(%1592: float[1, 120, 12, 12], %1598: float[1, 120, 12, 12]): 17280, %input.56: float[1, 80, 12, 12] = aten::_convolution(%input.55: float[1, 120, 12, 12], %111: float[80, 120, 1, 1], %1602: none, %1605: list[int], %1608: list[int], %1611: list[int], %1612: bool, %1615: list[int], %1616: int, %1617: bool, %1618: bool, %1619: bool, %1620: bool): 1382400, %input.57: float[1, 80, 12, 12] = aten::batch_norm(%input.56: float[1, 80, 12, 12], %112: float[80], %113: float[80], %114: float[80], %115: float[80], %1641: bool, %1642: float, %1643: float, %1644: bool): 0, %input.58: float[1, 240, 12, 12] = aten::_convolution(%input.57: float[1, 80, 12, 12], %117: float[240, 80, 1, 1], %1646: none, %1649: list[int], %1652: list[int], %1655: list[int], %1656: bool, %1659: list[int], %1660: int, %1661: bool, %1662: bool, %1663: bool, %1664: bool): 2764800, %1689: float[1, 240, 12, 12] = aten::batch_norm(%input.58: float[1, 240, 12, 12], %118: float[240], %119: float[240], %120: float[240], %121: float[240], %1685: bool, %1686: float, %1687: float, %1688: bool): 0, %1696: float[1, 240, 12, 12] = aten::mul(%1689: float[1, 240, 12, 12], %1695: float[1, 240, 12, 12]): 34560, %input.61: float[1, 240, 12, 12] = aten::_convolution(%input.60: float[1, 240, 12, 12], %123: float[240, 1, 3, 3], %1699: none, %1702: list[int], %1705: list[int], %1708: list[int], %1709: bool, %1712: list[int], %1713: int, %1714: bool, %1715: bool, %1716: bool, %1717: bool): 311040, %1742: float[1, 240, 12, 12] = aten::batch_norm(%input.61: float[1, 240, 12, 12], %124: float[240], %125: float[240], %126: float[240], %127: float[240], %1738: bool, %1739: float, %1740: float, %1741: bool): 0, %1749: float[1, 240, 12, 12] = aten::mul(%1742: float[1, 240, 12, 12], %1748: float[1, 240, 12, 12]): 34560, %input.64: float[1, 80, 12, 12] = aten::_convolution(%input.63: float[1, 240, 12, 12], %129: float[80, 240, 1, 1], %1752: none, %1755: list[int], %1758: list[int], %1761: list[int], %1762: bool, %1765: list[int], %1766: int, %1767: bool, %1768: bool, %1769: bool, %1770: bool): 2764800, %1795: float[1, 80, 12, 12] = aten::batch_norm(%input.64: float[1, 80, 12, 12], %130: float[80], %131: float[80], %132: float[80], %133: float[80], %1791: bool, %1792: float, %1793: float, %1794: bool): 0, %input.66: float[1, 480, 12, 12] = aten::_convolution(%input.65: float[1, 80, 12, 12], %135: float[480, 80, 1, 1], %1798: none, %1801: list[int], %1804: list[int], %1807: list[int], %1808: bool, %1811: list[int], %1812: int, %1813: bool, %1814: bool, %1815: bool, %1816: bool): 5529600, %1841: float[1, 480, 12, 12] = aten::batch_norm(%input.66: float[1, 480, 12, 12], %136: float[480], %137: float[480], %138: float[480], %139: float[480], %1837: bool, %1838: float, %1839: float, %1840: bool): 0, %1848: float[1, 480, 12, 12] = aten::mul(%1841: float[1, 480, 12, 12], %1847: float[1, 480, 12, 12]): 69120, %input.69: float[1, 480, 12, 12] = aten::_convolution(%input.68: float[1, 480, 12, 12], %141: float[480, 1, 7, 7], %1851: none, %1854: list[int], %1857: list[int], %1860: list[int], %1861: bool, %1864: list[int], %1865: int, %1866: bool, %1867: bool, %1868: bool, %1869: bool): 3386880, %1894: float[1, 480, 12, 12] = aten::batch_norm(%input.69: float[1, 480, 12, 12], %142: float[480], %143: float[480], %144: float[480], %145: float[480], %1890: bool, %1891: float, %1892: float, %1893: bool): 0, %1901: float[1, 480, 12, 12] = aten::mul(%1894: float[1, 480, 12, 12], %1900: float[1, 480, 12, 12]): 69120, %1908: float[1, 480, 12, 1] = aten::mean(%1903: float[1, 480, 12, 12], %1905: list[int], %1906: bool, %1907: none): 5760, %input.71: float[1, 480, 1, 1] = aten::mean(%1908: float[1, 480, 12, 1], %1910: list[int], %1911: bool, %1912: none): 480, %input.72: float[1, 120, 1, 1] = aten::_convolution(%input.71: float[1, 480, 1, 1], %147: float[120, 480, 1, 1], %148: float[120], %1916: list[int], %1919: list[int], %1922: list[int], %1923: bool, %1926: list[int], %1927: int, %1928: bool, %1929: bool, %1930: bool, %1931: bool): 57600, %1952: float[1, 480, 1, 1] = aten::_convolution(%input.73: float[1, 120, 1, 1], %149: float[480, 120, 1, 1], %150: float[480], %1936: list[int], %1939: list[int], %1942: list[int], %1943: bool, %1946: list[int], %1947: int, %1948: bool, %1949: bool, %1950: bool, %1951: bool): 57600, %input.75: float[1, 480, 12, 12] = aten::mul(%1903: float[1, 480, 12, 12], %1960: float[1, 480, 1, 1]): 69120, %input.76: float[1, 112, 12, 12] = aten::_convolution(%input.75: float[1, 480, 12, 12], %151: float[112, 480, 1, 1], %1962: none, %1965: list[int], %1968: list[int], %1971: list[int], %1972: bool, %1975: list[int], %1976: int, %1977: bool, %1978: bool, %1979: bool, %1980: bool): 7741440, %input.77: float[1, 112, 12, 12] = aten::batch_norm(%input.76: float[1, 112, 12, 12], %152: float[112], %153: float[112], %154: float[112], %155: float[112], %2001: bool, %2002: float, %2003: float, %2004: bool): 0, %input.78: float[1, 672, 12, 12] = aten::_convolution(%input.77: float[1, 112, 12, 12], %157: float[672, 112, 1, 1], %2006: none, %2009: list[int], %2012: list[int], %2015: list[int], %2016: bool, %2019: list[int], %2020: int, %2021: bool, %2022: bool, %2023: bool, %2024: bool): 10838016, %2049: float[1, 672, 12, 12] = aten::batch_norm(%input.78: float[1, 672, 12, 12], %158: float[672], %159: float[672], %160: float[672], %161: float[672], %2045: bool, %2046: float, %2047: float, %2048: bool): 0, %2056: float[1, 672, 12, 12] = aten::mul(%2049: float[1, 672, 12, 12], %2055: float[1, 672, 12, 12]): 96768, %input.81: float[1, 672, 12, 12] = aten::_convolution(%input.80: float[1, 672, 12, 12], %163: float[672, 1, 7, 7], %2059: none, %2062: list[int], %2065: list[int], %2068: list[int], %2069: bool, %2072: list[int], %2073: int, %2074: bool, %2075: bool, %2076: bool, %2077: bool): 4741632, %2102: float[1, 672, 12, 12] = aten::batch_norm(%input.81: float[1, 672, 12, 12], %164: float[672], %165: float[672], %166: float[672], %167: float[672], %2098: bool, %2099: float, %2100: float, %2101: bool): 0, %2109: float[1, 672, 12, 12] = aten::mul(%2102: float[1, 672, 12, 12], %2108: float[1, 672, 12, 12]): 96768, %2116: float[1, 672, 12, 1] = aten::mean(%2111: float[1, 672, 12, 12], %2113: list[int], %2114: bool, %2115: none): 8064, %input.83: float[1, 672, 1, 1] = aten::mean(%2116: float[1, 672, 12, 1], %2118: list[int], %2119: bool, %2120: none): 672, %input.84: float[1, 168, 1, 1] = aten::_convolution(%input.83: float[1, 672, 1, 1], %169: float[168, 672, 1, 1], %170: float[168], %2124: list[int], %2127: list[int], %2130: list[int], %2131: bool, %2134: list[int], %2135: int, %2136: bool, %2137: bool, %2138: bool, %2139: bool): 112896, %2160: float[1, 672, 1, 1] = aten::_convolution(%input.85: float[1, 168, 1, 1], %171: float[672, 168, 1, 1], %172: float[672], %2144: list[int], %2147: list[int], %2150: list[int], %2151: bool, %2154: list[int], %2155: int, %2156: bool, %2157: bool, %2158: bool, %2159: bool): 112896, %input.87: float[1, 672, 12, 12] = aten::mul(%2111: float[1, 672, 12, 12], %2168: float[1, 672, 1, 1]): 96768, %input.88: float[1, 112, 12, 12] = aten::_convolution(%input.87: float[1, 672, 12, 12], %173: float[112, 672, 1, 1], %2170: none, %2173: list[int], %2176: list[int], %2179: list[int], %2180: bool, %2183: list[int], %2184: int, %2185: bool, %2186: bool, %2187: bool, %2188: bool): 10838016, %2213: float[1, 112, 12, 12] = aten::batch_norm(%input.88: float[1, 112, 12, 12], %174: float[112], %175: float[112], %176: float[112], %177: float[112], %2209: bool, %2210: float, %2211: float, %2212: bool): 0, %input.90: float[1, 672, 12, 12] = aten::_convolution(%input.89: float[1, 112, 12, 12], %179: float[672, 112, 1, 1], %2216: none, %2219: list[int], %2222: list[int], %2225: list[int], %2226: bool, %2229: list[int], %2230: int, %2231: bool, %2232: bool, %2233: bool, %2234: bool): 10838016, %2259: float[1, 672, 12, 12] = aten::batch_norm(%input.90: float[1, 672, 12, 12], %180: float[672], %181: float[672], %182: float[672], %183: float[672], %2255: bool, %2256: float, %2257: float, %2258: bool): 0, %2266: float[1, 672, 12, 12] = aten::mul(%2259: float[1, 672, 12, 12], %2265: float[1, 672, 12, 12]): 96768, %input.93: float[1, 672, 6, 6] = aten::_convolution(%input.92: float[1, 672, 12, 12], %185: float[672, 1, 7, 7], %2269: none, %2272: list[int], %2275: list[int], %2278: list[int], %2279: bool, %2282: list[int], %2283: int, %2284: bool, %2285: bool, %2286: bool, %2287: bool): 1185408, %2312: float[1, 672, 6, 6] = aten::batch_norm(%input.93: float[1, 672, 6, 6], %186: float[672], %187: float[672], %188: float[672], %189: float[672], %2308: bool, %2309: float, %2310: float, %2311: bool): 0, %2319: float[1, 672, 6, 6] = aten::mul(%2312: float[1, 672, 6, 6], %2318: float[1, 672, 6, 6]): 24192, %2326: float[1, 672, 6, 1] = aten::mean(%2321: float[1, 672, 6, 6], %2323: list[int], %2324: bool, %2325: none): 4032, %input.95: float[1, 672, 1, 1] = aten::mean(%2326: float[1, 672, 6, 1], %2328: list[int], %2329: bool, %2330: none): 672, %input.96: float[1, 168, 1, 1] = aten::_convolution(%input.95: float[1, 672, 1, 1], %191: float[168, 672, 1, 1], %192: float[168], %2334: list[int], %2337: list[int], %2340: list[int], %2341: bool, %2344: list[int], %2345: int, %2346: bool, %2347: bool, %2348: bool, %2349: bool): 112896, %2370: float[1, 672, 1, 1] = aten::_convolution(%input.97: float[1, 168, 1, 1], %193: float[672, 168, 1, 1], %194: float[672], %2354: list[int], %2357: list[int], %2360: list[int], %2361: bool, %2364: list[int], %2365: int, %2366: bool, %2367: bool, %2368: bool, %2369: bool): 112896, %input.99: float[1, 672, 6, 6] = aten::mul(%2321: float[1, 672, 6, 6], %2378: float[1, 672, 1, 1]): 24192, %input.100: float[1, 160, 6, 6] = aten::_convolution(%input.99: float[1, 672, 6, 6], %195: float[160, 672, 1, 1], %2380: none, %2383: list[int], %2386: list[int], %2389: list[int], %2390: bool, %2393: list[int], %2394: int, %2395: bool, %2396: bool, %2397: bool, %2398: bool): 3870720, %input.101: float[1, 160, 6, 6] = aten::batch_norm(%input.100: float[1, 160, 6, 6], %196: float[160], %197: float[160], %198: float[160], %199: float[160], %2419: bool, %2420: float, %2421: float, %2422: bool): 0, %input.102: float[1, 960, 6, 6] = aten::_convolution(%input.101: float[1, 160, 6, 6], %201: float[960, 160, 1, 1], %2424: none, %2427: list[int], %2430: list[int], %2433: list[int], %2434: bool, %2437: list[int], %2438: int, %2439: bool, %2440: bool, %2441: bool, %2442: bool): 5529600, %2467: float[1, 960, 6, 6] = aten::batch_norm(%input.102: float[1, 960, 6, 6], %202: float[960], %203: float[960], %204: float[960], %205: float[960], %2463: bool, %2464: float, %2465: float, %2466: bool): 0, %2474: float[1, 960, 6, 6] = aten::mul(%2467: float[1, 960, 6, 6], %2473: float[1, 960, 6, 6]): 34560, %input.105: float[1, 960, 6, 6] = aten::_convolution(%input.104: float[1, 960, 6, 6], %207: float[960, 1, 7, 7], %2477: none, %2480: list[int], %2483: list[int], %2486: list[int], %2487: bool, %2490: list[int], %2491: int, %2492: bool, %2493: bool, %2494: bool, %2495: bool): 1693440, %2520: float[1, 960, 6, 6] = aten::batch_norm(%input.105: float[1, 960, 6, 6], %208: float[960], %209: float[960], %210: float[960], %211: float[960], %2516: bool, %2517: float, %2518: float, %2519: bool): 0, %2527: float[1, 960, 6, 6] = aten::mul(%2520: float[1, 960, 6, 6], %2526: float[1, 960, 6, 6]): 34560, %2534: float[1, 960, 6, 1] = aten::mean(%2529: float[1, 960, 6, 6], %2531: list[int], %2532: bool, %2533: none): 5760, %input.107: float[1, 960, 1, 1] = aten::mean(%2534: float[1, 960, 6, 1], %2536: list[int], %2537: bool, %2538: none): 960, %input.108: float[1, 240, 1, 1] = aten::_convolution(%input.107: float[1, 960, 1, 1], %213: float[240, 960, 1, 1], %214: float[240], %2542: list[int], %2545: list[int], %2548: list[int], %2549: bool, %2552: list[int], %2553: int, %2554: bool, %2555: bool, %2556: bool, %2557: bool): 230400, %2578: float[1, 960, 1, 1] = aten::_convolution(%input.109: float[1, 240, 1, 1], %215: float[960, 240, 1, 1], %216: float[960], %2562: list[int], %2565: list[int], %2568: list[int], %2569: bool, %2572: list[int], %2573: int, %2574: bool, %2575: bool, %2576: bool, %2577: bool): 230400, %input.111: float[1, 960, 6, 6] = aten::mul(%2529: float[1, 960, 6, 6], %2586: float[1, 960, 1, 1]): 34560, %input.112: float[1, 160, 6, 6] = aten::_convolution(%input.111: float[1, 960, 6, 6], %217: float[160, 960, 1, 1], %2588: none, %2591: list[int], %2594: list[int], %2597: list[int], %2598: bool, %2601: list[int], %2602: int, %2603: bool, %2604: bool, %2605: bool, %2606: bool): 5529600, %2631: float[1, 160, 6, 6] = aten::batch_norm(%input.112: float[1, 160, 6, 6], %218: float[160], %219: float[160], %220: float[160], %221: float[160], %2627: bool, %2628: float, %2629: float, %2630: bool): 0, %input.114: float[1, 960, 6, 6] = aten::_convolution(%input.113: float[1, 160, 6, 6], %223: float[960, 160, 1, 1], %2634: none, %2637: list[int], %2640: list[int], %2643: list[int], %2644: bool, %2647: list[int], %2648: int, %2649: bool, %2650: bool, %2651: bool, %2652: bool): 5529600, %2677: float[1, 960, 6, 6] = aten::batch_norm(%input.114: float[1, 960, 6, 6], %224: float[960], %225: float[960], %226: float[960], %227: float[960], %2673: bool, %2674: float, %2675: float, %2676: bool): 0, %2684: float[1, 960, 6, 6] = aten::mul(%2677: float[1, 960, 6, 6], %2683: float[1, 960, 6, 6]): 34560, %2691: float[1, 960, 6, 1] = aten::mean(%2686: float[1, 960, 6, 6], %2688: list[int], %2689: bool, %2690: none): 5760, %input.116: float[1, 960, 1, 1] = aten::mean(%2691: float[1, 960, 6, 1], %2693: list[int], %2694: bool, %2695: none): 960, %2716: float[1, 1280, 1, 1] = aten::_convolution(%input.116: float[1, 960, 1, 1], %229: float[1280, 960, 1, 1], %2697: none, %2700: list[int], %2703: list[int], %2706: list[int], %2707: bool, %2710: list[int], %2711: int, %2712: bool, %2713: bool, %2714: bool, %2715: bool): 1228800, %2723: float[1, 1280, 1, 1] = aten::mul(%2716: float[1, 1280, 1, 1], %2722: float[1, 1280, 1, 1]): 1280, %2736: float[1, 10] = aten::addmm(%231: float[10], %input: float[1, 1280], %2733: float[1280, 10], %2734: int, %2735: int): 12800}

And here's for line 6 (profile_macs(copy.deepcopy(self.engine.get_active_subnet(True)), inputs.cuda(), reduction=None)):

{%input.2: float[1, 16, 96, 96] = aten::_convolution(%input.1: float[1, 3, 192, 192], %1: float[16, 3, 3, 3], %696: none, %699: list[int], %702: list[int], %705: list[int], %706: bool, %709: list[int], %710: int, %711: bool, %712: bool, %713: bool, %714: bool): 3981312, %739: float[1, 16, 96, 96] = aten::batch_norm(%input.2: float[1, 16, 96, 96], %2: float[16], %3: float[16], %4: float[16], %5: float[16], %735: bool, %736: float, %737: float, %738: bool): 0, %746: float[1, 16, 96, 96] = aten::mul(%739: float[1, 16, 96, 96], %745: float[1, 16, 96, 96]): 147456, %input.5: float[1, 16, 96, 96] = aten::_convolution(%input.4: float[1, 16, 96, 96], %7: float[16, 1, 3, 3], %749: none, %752: list[int], %755: list[int], %758: list[int], %759: bool, %762: list[int], %763: int, %764: bool, %765: bool, %766: bool, %767: bool): 1327104, %input.6: float[1, 16, 96, 96] = aten::batch_norm(%input.5: float[1, 16, 96, 96], %8: float[16], %9: float[16], %10: float[16], %11: float[16], %788: bool, %789: float, %790: float, %791: bool): 0, %input.8: float[1, 16, 96, 96] = aten::_convolution(%input.7: float[1, 16, 96, 96], %13: float[16, 16, 1, 1], %794: none, %797: list[int], %800: list[int], %803: list[int], %804: bool, %807: list[int], %808: int, %809: bool, %810: bool, %811: bool, %812: bool): 2359296, %837: float[1, 16, 96, 96] = aten::batch_norm(%input.8: float[1, 16, 96, 96], %14: float[16], %15: float[16], %16: float[16], %17: float[16], %833: bool, %834: float, %835: float, %836: bool): 0, %input.10: float[1, 48, 96, 96] = aten::_convolution(%input.9: float[1, 16, 96, 96], %19: float[48, 16, 1, 1], %840: none, %843: list[int], %846: list[int], %849: list[int], %850: bool, %853: list[int], %854: int, %855: bool, %856: bool, %857: bool, %858: bool): 7077888, %input.11: float[1, 48, 96, 96] = aten::batch_norm(%input.10: float[1, 48, 96, 96], %20: float[48], %21: float[48], %22: float[48], %23: float[48], %879: bool, %880: float, %881: float, %882: bool): 0, %input.13: float[1, 48, 48, 48] = aten::_convolution(%input.12: float[1, 48, 96, 96], %25: float[48, 1, 3, 3], %885: none, %888: list[int], %891: list[int], %894: list[int], %895: bool, %898: list[int], %899: int, %900: bool, %901: bool, %902: bool, %903: bool): 995328, %input.14: float[1, 48, 48, 48] = aten::batch_norm(%input.13: float[1, 48, 48, 48], %26: float[48], %27: float[48], %28: float[48], %29: float[48], %924: bool, %925: float, %926: float, %927: bool): 0, %input.16: float[1, 24, 48, 48] = aten::_convolution(%input.15: float[1, 48, 48, 48], %31: float[24, 48, 1, 1], %930: none, %933: list[int], %936: list[int], %939: list[int], %940: bool, %943: list[int], %944: int, %945: bool, %946: bool, %947: bool, %948: bool): 2654208, %input.17: float[1, 24, 48, 48] = aten::batch_norm(%input.16: float[1, 24, 48, 48], %32: float[24], %33: float[24], %34: float[24], %35: float[24], %969: bool, %970: float, %971: float, %972: bool): 0, %input.18: float[1, 72, 48, 48] = aten::_convolution(%input.17: float[1, 24, 48, 48], %37: float[72, 24, 1, 1], %974: none, %977: list[int], %980: list[int], %983: list[int], %984: bool, %987: list[int], %988: int, %989: bool, %990: bool, %991: bool, %992: bool): 3981312, %input.19: float[1, 72, 48, 48] = aten::batch_norm(%input.18: float[1, 72, 48, 48], %38: float[72], %39: float[72], %40: float[72], %41: float[72], %1013: bool, %1014: float, %1015: float, %1016: bool): 0, %input.21: float[1, 72, 48, 48] = aten::_convolution(%input.20: float[1, 72, 48, 48], %43: float[72, 1, 3, 3], %1019: none, %1022: list[int], %1025: list[int], %1028: list[int], %1029: bool, %1032: list[int], %1033: int, %1034: bool, %1035: bool, %1036: bool, %1037: bool): 1492992, %input.22: float[1, 72, 48, 48] = aten::batch_norm(%input.21: float[1, 72, 48, 48], %44: float[72], %45: float[72], %46: float[72], %47: float[72], %1058: bool, %1059: float, %1060: float, %1061: bool): 0, %input.24: float[1, 24, 48, 48] = aten::_convolution(%input.23: float[1, 72, 48, 48], %49: float[24, 72, 1, 1], %1064: none, %1067: list[int], %1070: list[int], %1073: list[int], %1074: bool, %1077: list[int], %1078: int, %1079: bool, %1080: bool, %1081: bool, %1082: bool): 3981312, %1107: float[1, 24, 48, 48] = aten::batch_norm(%input.24: float[1, 24, 48, 48], %50: float[24], %51: float[24], %52: float[24], %53: float[24], %1103: bool, %1104: float, %1105: float, %1106: bool): 0, %input.26: float[1, 72, 48, 48] = aten::_convolution(%input.25: float[1, 24, 48, 48], %55: float[72, 24, 1, 1], %1110: none, %1113: list[int], %1116: list[int], %1119: list[int], %1120: bool, %1123: list[int], %1124: int, %1125: bool, %1126: bool, %1127: bool, %1128: bool): 3981312, %input.27: float[1, 72, 48, 48] = aten::batch_norm(%input.26: float[1, 72, 48, 48], %56: float[72], %57: float[72], %58: float[72], %59: float[72], %1149: bool, %1150: float, %1151: float, %1152: bool): 0, %input.29: float[1, 72, 24, 24] = aten::_convolution(%input.28: float[1, 72, 48, 48], %61: float[72, 1, 3, 3], %1155: none, %1158: list[int], %1161: list[int], %1164: list[int], %1165: bool, %1168: list[int], %1169: int, %1170: bool, %1171: bool, %1172: bool, %1173: bool): 373248, %input.30: float[1, 72, 24, 24] = aten::batch_norm(%input.29: float[1, 72, 24, 24], %62: float[72], %63: float[72], %64: float[72], %65: float[72], %1194: bool, %1195: float, %1196: float, %1197: bool): 0, %1204: float[1, 72, 24, 1] = aten::mean(%1199: float[1, 72, 24, 24], %1201: list[int], %1202: bool, %1203: none): 1728, %input.31: float[1, 72, 1, 1] = aten::mean(%1204: float[1, 72, 24, 1], %1206: list[int], %1207: bool, %1208: none): 72, %input.32: float[1, 24, 1, 1] = aten::_convolution(%input.31: float[1, 72, 1, 1], %67: float[24, 72, 1, 1], %68: float[24], %1212: list[int], %1215: list[int], %1218: list[int], %1219: bool, %1222: list[int], %1223: int, %1224: bool, %1225: bool, %1226: bool, %1227: bool): 1728, %1248: float[1, 72, 1, 1] = aten::_convolution(%input.33: float[1, 24, 1, 1], %69: float[72, 24, 1, 1], %70: float[72], %1232: list[int], %1235: list[int], %1238: list[int], %1239: bool, %1242: list[int], %1243: int, %1244: bool, %1245: bool, %1246: bool, %1247: bool): 1728, %input.35: float[1, 72, 24, 24] = aten::mul(%1199: float[1, 72, 24, 24], %1256: float[1, 72, 1, 1]): 41472, %input.36: float[1, 40, 24, 24] = aten::_convolution(%input.35: float[1, 72, 24, 24], %71: float[40, 72, 1, 1], %1258: none, %1261: list[int], %1264: list[int], %1267: list[int], %1268: bool, %1271: list[int], %1272: int, %1273: bool, %1274: bool, %1275: bool, %1276: bool): 1658880, %input.37: float[1, 40, 24, 24] = aten::batch_norm(%input.36: float[1, 40, 24, 24], %72: float[40], %73: float[40], %74: float[40], %75: float[40], %1297: bool, %1298: float, %1299: float, %1300: bool): 0, %input.38: float[1, 120, 24, 24] = aten::_convolution(%input.37: float[1, 40, 24, 24], %77: float[120, 40, 1, 1], %1302: none, %1305: list[int], %1308: list[int], %1311: list[int], %1312: bool, %1315: list[int], %1316: int, %1317: bool, %1318: bool, %1319: bool, %1320: bool): 2764800, %input.39: float[1, 120, 24, 24] = aten::batch_norm(%input.38: float[1, 120, 24, 24], %78: float[120], %79: float[120], %80: float[120], %81: float[120], %1341: bool, %1342: float, %1343: float, %1344: bool): 0, %input.41: float[1, 120, 24, 24] = aten::_convolution(%input.40: float[1, 120, 24, 24], %83: float[120, 1, 3, 3], %1347: none, %1350: list[int], %1353: list[int], %1356: list[int], %1357: bool, %1360: list[int], %1361: int, %1362: bool, %1363: bool, %1364: bool, %1365: bool): 622080, %input.42: float[1, 120, 24, 24] = aten::batch_norm(%input.41: float[1, 120, 24, 24], %84: float[120], %85: float[120], %86: float[120], %87: float[120], %1386: bool, %1387: float, %1388: float, %1389: bool): 0, %1396: float[1, 120, 24, 1] = aten::mean(%1391: float[1, 120, 24, 24], %1393: list[int], %1394: bool, %1395: none): 2880, %input.43: float[1, 120, 1, 1] = aten::mean(%1396: float[1, 120, 24, 1], %1398: list[int], %1399: bool, %1400: none): 120, %input.44: float[1, 32, 1, 1] = aten::_convolution(%input.43: float[1, 120, 1, 1], %89: float[32, 120, 1, 1], %90: float[32], %1404: list[int], %1407: list[int], %1410: list[int], %1411: bool, %1414: list[int], %1415: int, %1416: bool, %1417: bool, %1418: bool, %1419: bool): 3840, %1440: float[1, 120, 1, 1] = aten::_convolution(%input.45: float[1, 32, 1, 1], %91: float[120, 32, 1, 1], %92: float[120], %1424: list[int], %1427: list[int], %1430: list[int], %1431: bool, %1434: list[int], %1435: int, %1436: bool, %1437: bool, %1438: bool, %1439: bool): 3840, %input.47: float[1, 120, 24, 24] = aten::mul(%1391: float[1, 120, 24, 24], %1448: float[1, 120, 1, 1]): 69120, %input.48: float[1, 40, 24, 24] = aten::_convolution(%input.47: float[1, 120, 24, 24], %93: float[40, 120, 1, 1], %1450: none, %1453: list[int], %1456: list[int], %1459: list[int], %1460: bool, %1463: list[int], %1464: int, %1465: bool, %1466: bool, %1467: bool, %1468: bool): 2764800, %1493: float[1, 40, 24, 24] = aten::batch_norm(%input.48: float[1, 40, 24, 24], %94: float[40], %95: float[40], %96: float[40], %97: float[40], %1489: bool, %1490: float, %1491: float, %1492: bool): 0, %input.50: float[1, 120, 24, 24] = aten::_convolution(%input.49: float[1, 40, 24, 24], %99: float[120, 40, 1, 1], %1496: none, %1499: list[int], %1502: list[int], %1505: list[int], %1506: bool, %1509: list[int], %1510: int, %1511: bool, %1512: bool, %1513: bool, %1514: bool): 2764800, %1539: float[1, 120, 24, 24] = aten::batch_norm(%input.50: float[1, 120, 24, 24], %100: float[120], %101: float[120], %102: float[120], %103: float[120], %1535: bool, %1536: float, %1537: float, %1538: bool): 0, %1546: float[1, 120, 24, 24] = aten::mul(%1539: float[1, 120, 24, 24], %1545: float[1, 120, 24, 24]): 69120, %input.53: float[1, 120, 12, 12] = aten::_convolution(%input.52: float[1, 120, 24, 24], %105: float[120, 1, 3, 3], %1549: none, %1552: list[int], %1555: list[int], %1558: list[int], %1559: bool, %1562: list[int], %1563: int, %1564: bool, %1565: bool, %1566: bool, %1567: bool): 155520, %1592: float[1, 120, 12, 12] = aten::batch_norm(%input.53: float[1, 120, 12, 12], %106: float[120], %107: float[120], %108: float[120], %109: float[120], %1588: bool, %1589: float, %1590: float, %1591: bool): 0, %1599: float[1, 120, 12, 12] = aten::mul(%1592: float[1, 120, 12, 12], %1598: float[1, 120, 12, 12]): 17280, %input.56: float[1, 80, 12, 12] = aten::_convolution(%input.55: float[1, 120, 12, 12], %111: float[80, 120, 1, 1], %1602: none, %1605: list[int], %1608: list[int], %1611: list[int], %1612: bool, %1615: list[int], %1616: int, %1617: bool, %1618: bool, %1619: bool, %1620: bool): 1382400, %input.57: float[1, 80, 12, 12] = aten::batch_norm(%input.56: float[1, 80, 12, 12], %112: float[80], %113: float[80], %114: float[80], %115: float[80], %1641: bool, %1642: float, %1643: float, %1644: bool): 0, %input.58: float[1, 240, 12, 12] = aten::_convolution(%input.57: float[1, 80, 12, 12], %117: float[240, 80, 1, 1], %1646: none, %1649: list[int], %1652: list[int], %1655: list[int], %1656: bool, %1659: list[int], %1660: int, %1661: bool, %1662: bool, %1663: bool, %1664: bool): 2764800, %1689: float[1, 240, 12, 12] = aten::batch_norm(%input.58: float[1, 240, 12, 12], %118: float[240], %119: float[240], %120: float[240], %121: float[240], %1685: bool, %1686: float, %1687: float, %1688: bool): 0, %1696: float[1, 240, 12, 12] = aten::mul(%1689: float[1, 240, 12, 12], %1695: float[1, 240, 12, 12]): 34560, %input.61: float[1, 240, 12, 12] = aten::_convolution(%input.60: float[1, 240, 12, 12], %123: float[240, 1, 3, 3], %1699: none, %1702: list[int], %1705: list[int], %1708: list[int], %1709: bool, %1712: list[int], %1713: int, %1714: bool, %1715: bool, %1716: bool, %1717: bool): 311040, %1742: float[1, 240, 12, 12] = aten::batch_norm(%input.61: float[1, 240, 12, 12], %124: float[240], %125: float[240], %126: float[240], %127: float[240], %1738: bool, %1739: float, %1740: float, %1741: bool): 0, %1749: float[1, 240, 12, 12] = aten::mul(%1742: float[1, 240, 12, 12], %1748: float[1, 240, 12, 12]): 34560, %input.64: float[1, 80, 12, 12] = aten::_convolution(%input.63: float[1, 240, 12, 12], %129: float[80, 240, 1, 1], %1752: none, %1755: list[int], %1758: list[int], %1761: list[int], %1762: bool, %1765: list[int], %1766: int, %1767: bool, %1768: bool, %1769: bool, %1770: bool): 2764800, %1795: float[1, 80, 12, 12] = aten::batch_norm(%input.64: float[1, 80, 12, 12], %130: float[80], %131: float[80], %132: float[80], %133: float[80], %1791: bool, %1792: float, %1793: float, %1794: bool): 0, %input.66: float[1, 240, 12, 12] = aten::_convolution(%input.65: float[1, 80, 12, 12], %135: float[240, 80, 1, 1], %1798: none, %1801: list[int], %1804: list[int], %1807: list[int], %1808: bool, %1811: list[int], %1812: int, %1813: bool, %1814: bool, %1815: bool, %1816: bool): 2764800, %1841: float[1, 240, 12, 12] = aten::batch_norm(%input.66: float[1, 240, 12, 12], %136: float[240], %137: float[240], %138: float[240], %139: float[240], %1837: bool, %1838: float, %1839: float, %1840: bool): 0, %1848: float[1, 240, 12, 12] = aten::mul(%1841: float[1, 240, 12, 12], %1847: float[1, 240, 12, 12]): 34560, %input.69: float[1, 240, 12, 12] = aten::_convolution(%input.68: float[1, 240, 12, 12], %141: float[240, 1, 3, 3], %1851: none, %1854: list[int], %1857: list[int], %1860: list[int], %1861: bool, %1864: list[int], %1865: int, %1866: bool, %1867: bool, %1868: bool, %1869: bool): 311040, %1894: float[1, 240, 12, 12] = aten::batch_norm(%input.69: float[1, 240, 12, 12], %142: float[240], %143: float[240], %144: float[240], %145: float[240], %1890: bool, %1891: float, %1892: float, %1893: bool): 0, %1901: float[1, 240, 12, 12] = aten::mul(%1894: float[1, 240, 12, 12], %1900: float[1, 240, 12, 12]): 34560, %1908: float[1, 240, 12, 1] = aten::mean(%1903: float[1, 240, 12, 12], %1905: list[int], %1906: bool, %1907: none): 2880, %input.71: float[1, 240, 1, 1] = aten::mean(%1908: float[1, 240, 12, 1], %1910: list[int], %1911: bool, %1912: none): 240, %input.72: float[1, 64, 1, 1] = aten::_convolution(%input.71: float[1, 240, 1, 1], %147: float[64, 240, 1, 1], %148: float[64], %1916: list[int], %1919: list[int], %1922: list[int], %1923: bool, %1926: list[int], %1927: int, %1928: bool, %1929: bool, %1930: bool, %1931: bool): 15360, %1952: float[1, 240, 1, 1] = aten::_convolution(%input.73: float[1, 64, 1, 1], %149: float[240, 64, 1, 1], %150: float[240], %1936: list[int], %1939: list[int], %1942: list[int], %1943: bool, %1946: list[int], %1947: int, %1948: bool, %1949: bool, %1950: bool, %1951: bool): 15360, %input.75: float[1, 240, 12, 12] = aten::mul(%1903: float[1, 240, 12, 12], %1960: float[1, 240, 1, 1]): 34560, %input.76: float[1, 112, 12, 12] = aten::_convolution(%input.75: float[1, 240, 12, 12], %151: float[112, 240, 1, 1], %1962: none, %1965: list[int], %1968: list[int], %1971: list[int], %1972: bool, %1975: list[int], %1976: int, %1977: bool, %1978: bool, %1979: bool, %1980: bool): 3870720, %input.77: float[1, 112, 12, 12] = aten::batch_norm(%input.76: float[1, 112, 12, 12], %152: float[112], %153: float[112], %154: float[112], %155: float[112], %2001: bool, %2002: float, %2003: float, %2004: bool): 0, %input.78: float[1, 336, 12, 12] = aten::_convolution(%input.77: float[1, 112, 12, 12], %157: float[336, 112, 1, 1], %2006: none, %2009: list[int], %2012: list[int], %2015: list[int], %2016: bool, %2019: list[int], %2020: int, %2021: bool, %2022: bool, %2023: bool, %2024: bool): 5419008, %2049: float[1, 336, 12, 12] = aten::batch_norm(%input.78: float[1, 336, 12, 12], %158: float[336], %159: float[336], %160: float[336], %161: float[336], %2045: bool, %2046: float, %2047: float, %2048: bool): 0, %2056: float[1, 336, 12, 12] = aten::mul(%2049: float[1, 336, 12, 12], %2055: float[1, 336, 12, 12]): 48384, %input.81: float[1, 336, 12, 12] = aten::_convolution(%input.80: float[1, 336, 12, 12], %163: float[336, 1, 3, 3], %2059: none, %2062: list[int], %2065: list[int], %2068: list[int], %2069: bool, %2072: list[int], %2073: int, %2074: bool, %2075: bool, %2076: bool, %2077: bool): 435456, %2102: float[1, 336, 12, 12] = aten::batch_norm(%input.81: float[1, 336, 12, 12], %164: float[336], %165: float[336], %166: float[336], %167: float[336], %2098: bool, %2099: float, %2100: float, %2101: bool): 0, %2109: float[1, 336, 12, 12] = aten::mul(%2102: float[1, 336, 12, 12], %2108: float[1, 336, 12, 12]): 48384, %2116: float[1, 336, 12, 1] = aten::mean(%2111: float[1, 336, 12, 12], %2113: list[int], %2114: bool, %2115: none): 4032, %input.83: float[1, 336, 1, 1] = aten::mean(%2116: float[1, 336, 12, 1], %2118: list[int], %2119: bool, %2120: none): 336, %input.84: float[1, 88, 1, 1] = aten::_convolution(%input.83: float[1, 336, 1, 1], %169: float[88, 336, 1, 1], %170: float[88], %2124: list[int], %2127: list[int], %2130: list[int], %2131: bool, %2134: list[int], %2135: int, %2136: bool, %2137: bool, %2138: bool, %2139: bool): 29568, %2160: float[1, 336, 1, 1] = aten::_convolution(%input.85: float[1, 88, 1, 1], %171: float[336, 88, 1, 1], %172: float[336], %2144: list[int], %2147: list[int], %2150: list[int], %2151: bool, %2154: list[int], %2155: int, %2156: bool, %2157: bool, %2158: bool, %2159: bool): 29568, %input.87: float[1, 336, 12, 12] = aten::mul(%2111: float[1, 336, 12, 12], %2168: float[1, 336, 1, 1]): 48384, %input.88: float[1, 112, 12, 12] = aten::_convolution(%input.87: float[1, 336, 12, 12], %173: float[112, 336, 1, 1], %2170: none, %2173: list[int], %2176: list[int], %2179: list[int], %2180: bool, %2183: list[int], %2184: int, %2185: bool, %2186: bool, %2187: bool, %2188: bool): 5419008, %2213: float[1, 112, 12, 12] = aten::batch_norm(%input.88: float[1, 112, 12, 12], %174: float[112], %175: float[112], %176: float[112], %177: float[112], %2209: bool, %2210: float, %2211: float, %2212: bool): 0, %input.90: float[1, 336, 12, 12] = aten::_convolution(%input.89: float[1, 112, 12, 12], %179: float[336, 112, 1, 1], %2216: none, %2219: list[int], %2222: list[int], %2225: list[int], %2226: bool, %2229: list[int], %2230: int, %2231: bool, %2232: bool, %2233: bool, %2234: bool): 5419008, %2259: float[1, 336, 12, 12] = aten::batch_norm(%input.90: float[1, 336, 12, 12], %180: float[336], %181: float[336], %182: float[336], %183: float[336], %2255: bool, %2256: float, %2257: float, %2258: bool): 0, %2266: float[1, 336, 12, 12] = aten::mul(%2259: float[1, 336, 12, 12], %2265: float[1, 336, 12, 12]): 48384, %input.93: float[1, 336, 6, 6] = aten::_convolution(%input.92: float[1, 336, 12, 12], %185: float[336, 1, 3, 3], %2269: none, %2272: list[int], %2275: list[int], %2278: list[int], %2279: bool, %2282: list[int], %2283: int, %2284: bool, %2285: bool, %2286: bool, %2287: bool): 108864, %2312: float[1, 336, 6, 6] = aten::batch_norm(%input.93: float[1, 336, 6, 6], %186: float[336], %187: float[336], %188: float[336], %189: float[336], %2308: bool, %2309: float, %2310: float, %2311: bool): 0, %2319: float[1, 336, 6, 6] = aten::mul(%2312: float[1, 336, 6, 6], %2318: float[1, 336, 6, 6]): 12096, %2326: float[1, 336, 6, 1] = aten::mean(%2321: float[1, 336, 6, 6], %2323: list[int], %2324: bool, %2325: none): 2016, %input.95: float[1, 336, 1, 1] = aten::mean(%2326: float[1, 336, 6, 1], %2328: list[int], %2329: bool, %2330: none): 336, %input.96: float[1, 88, 1, 1] = aten::_convolution(%input.95: float[1, 336, 1, 1], %191: float[88, 336, 1, 1], %192: float[88], %2334: list[int], %2337: list[int], %2340: list[int], %2341: bool, %2344: list[int], %2345: int, %2346: bool, %2347: bool, %2348: bool, %2349: bool): 29568, %2370: float[1, 336, 1, 1] = aten::_convolution(%input.97: float[1, 88, 1, 1], %193: float[336, 88, 1, 1], %194: float[336], %2354: list[int], %2357: list[int], %2360: list[int], %2361: bool, %2364: list[int], %2365: int, %2366: bool, %2367: bool, %2368: bool, %2369: bool): 29568, %input.99: float[1, 336, 6, 6] = aten::mul(%2321: float[1, 336, 6, 6], %2378: float[1, 336, 1, 1]): 12096, %input.100: float[1, 160, 6, 6] = aten::_convolution(%input.99: float[1, 336, 6, 6], %195: float[160, 336, 1, 1], %2380: none, %2383: list[int], %2386: list[int], %2389: list[int], %2390: bool, %2393: list[int], %2394: int, %2395: bool, %2396: bool, %2397: bool, %2398: bool): 1935360, %input.101: float[1, 160, 6, 6] = aten::batch_norm(%input.100: float[1, 160, 6, 6], %196: float[160], %197: float[160], %198: float[160], %199: float[160], %2419: bool, %2420: float, %2421: float, %2422: bool): 0, %input.102: float[1, 480, 6, 6] = aten::_convolution(%input.101: float[1, 160, 6, 6], %201: float[480, 160, 1, 1], %2424: none, %2427: list[int], %2430: list[int], %2433: list[int], %2434: bool, %2437: list[int], %2438: int, %2439: bool, %2440: bool, %2441: bool, %2442: bool): 2764800, %2467: float[1, 480, 6, 6] = aten::batch_norm(%input.102: float[1, 480, 6, 6], %202: float[480], %203: float[480], %204: float[480], %205: float[480], %2463: bool, %2464: float, %2465: float, %2466: bool): 0, %2474: float[1, 480, 6, 6] = aten::mul(%2467: float[1, 480, 6, 6], %2473: float[1, 480, 6, 6]): 17280, %input.105: float[1, 480, 6, 6] = aten::_convolution(%input.104: float[1, 480, 6, 6], %207: float[480, 1, 3, 3], %2477: none, %2480: list[int], %2483: list[int], %2486: list[int], %2487: bool, %2490: list[int], %2491: int, %2492: bool, %2493: bool, %2494: bool, %2495: bool): 155520, %2520: float[1, 480, 6, 6] = aten::batch_norm(%input.105: float[1, 480, 6, 6], %208: float[480], %209: float[480], %210: float[480], %211: float[480], %2516: bool, %2517: float, %2518: float, %2519: bool): 0, %2527: float[1, 480, 6, 6] = aten::mul(%2520: float[1, 480, 6, 6], %2526: float[1, 480, 6, 6]): 17280, %2534: float[1, 480, 6, 1] = aten::mean(%2529: float[1, 480, 6, 6], %2531: list[int], %2532: bool, %2533: none): 2880, %input.107: float[1, 480, 1, 1] = aten::mean(%2534: float[1, 480, 6, 1], %2536: list[int], %2537: bool, %2538: none): 480, %input.108: float[1, 120, 1, 1] = aten::_convolution(%input.107: float[1, 480, 1, 1], %213: float[120, 480, 1, 1], %214: float[120], %2542: list[int], %2545: list[int], %2548: list[int], %2549: bool, %2552: list[int], %2553: int, %2554: bool, %2555: bool, %2556: bool, %2557: bool): 57600, %2578: float[1, 480, 1, 1] = aten::_convolution(%input.109: float[1, 120, 1, 1], %215: float[480, 120, 1, 1], %216: float[480], %2562: list[int], %2565: list[int], %2568: list[int], %2569: bool, %2572: list[int], %2573: int, %2574: bool, %2575: bool, %2576: bool, %2577: bool): 57600, %input.111: float[1, 480, 6, 6] = aten::mul(%2529: float[1, 480, 6, 6], %2586: float[1, 480, 1, 1]): 17280, %input.112: float[1, 160, 6, 6] = aten::_convolution(%input.111: float[1, 480, 6, 6], %217: float[160, 480, 1, 1], %2588: none, %2591: list[int], %2594: list[int], %2597: list[int], %2598: bool, %2601: list[int], %2602: int, %2603: bool, %2604: bool, %2605: bool, %2606: bool): 2764800, %2631: float[1, 160, 6, 6] = aten::batch_norm(%input.112: float[1, 160, 6, 6], %218: float[160], %219: float[160], %220: float[160], %221: float[160], %2627: bool, %2628: float, %2629: float, %2630: bool): 0, %input.114: float[1, 960, 6, 6] = aten::_convolution(%input.113: float[1, 160, 6, 6], %223: float[960, 160, 1, 1], %2634: none, %2637: list[int], %2640: list[int], %2643: list[int], %2644: bool, %2647: list[int], %2648: int, %2649: bool, %2650: bool, %2651: bool, %2652: bool): 5529600, %2677: float[1, 960, 6, 6] = aten::batch_norm(%input.114: float[1, 960, 6, 6], %224: float[960], %225: float[960], %226: float[960], %227: float[960], %2673: bool, %2674: float, %2675: float, %2676: bool): 0, %2684: float[1, 960, 6, 6] = aten::mul(%2677: float[1, 960, 6, 6], %2683: float[1, 960, 6, 6]): 34560, %2691: float[1, 960, 6, 1] = aten::mean(%2686: float[1, 960, 6, 6], %2688: list[int], %2689: bool, %2690: none): 5760, %input.116: float[1, 960, 1, 1] = aten::mean(%2691: float[1, 960, 6, 1], %2693: list[int], %2694: bool, %2695: none): 960, %2716: float[1, 1280, 1, 1] = aten::_convolution(%input.116: float[1, 960, 1, 1], %229: float[1280, 960, 1, 1], %2697: none, %2700: list[int], %2703: list[int], %2706: list[int], %2707: bool, %2710: list[int], %2711: int, %2712: bool, %2713: bool, %2714: bool, %2715: bool): 1228800, %2723: float[1, 1280, 1, 1] = aten::mul(%2716: float[1, 1280, 1, 1], %2722: float[1, 1280, 1, 1]): 1280, %2736: float[1, 10] = aten::addmm(%231: float[10], %input: float[1, 1280], %2733: float[1280, 10], %2734: int, %2735: int): 12800}

from torchprofile.

sorry for the huge text dumps; if you tell me how to get it in a nicer format, I'll do it. For now the only thing I was able to understand that these 2 outputs are not the same (different lengths at the very least)

from torchprofile.

Thanks! It would be great if you could upload them to https://pastebin.com/ and share the link here.

from torchprofile.

Sure, here's the first one (https://pastebin.com/1g8fHcCf) and the second one (https://pastebin.com/2nX1hu9T)

from torchprofile.

I think it might not really because of torchprofile. In each case, you set active networks by two times: the first time with a larger config, and the second time with a smaller config. This might cause an issue in OFA's implementation: these kernel sizes that are not covered in the second one might still use the value from the first one.

from torchprofile.

Btw, I think you need to have the same length of ks for config and config_small_long/config_large_long.

from torchprofile.

In each case, you set active networks by two times: the first time with a larger config, and the second time with a smaller config. This might cause an issue in OFA's implementation: these kernel sizes that are not covered in the second one might still use the value from the first one.

Yes, I now see that I misunderstood how OFA implementation works: I thought that the values from the longer configs will not be used in shorter configs (after all, if the depth = 2, the code will not read the config values for layers 3 and 4), but it isn't so. The correct way is to pad the shorter config with None, then the result is as expected:

> configv ={'ks': [3, 3, None, None, 3, 3, None, None, 3, 3, None, None, 3, 3, None, None, 3, 3, None, None], 'e': [3, 3, None, None, 3, 3, None, None, 3, 3, None, None, 3, 3, None, None, 3, 3, None, None], 'd': [2, 2, 2, 2, 2], 'r': 192}
> self.engine.set_active_subnet(ks=config_big_long['ks'], e=config_big_long['e'], d=config_big_long['d'])
> self.engine.set_active_subnet(ks=configv['ks'], e=configv['e'], d=configv['d'])
> int(profile_macs(copy.deepcopy(self.engine.get_active_subnet(True)), inputs.cuda()))
89421520
> self.engine.set_active_subnet(ks=config_small_long['ks'], e=config_small_long['e'], d=config_small_long['d'])
> self.engine.set_active_subnet(ks=configv['ks'], e=configv['e'], d=configv['d'])
> int(profile_macs(copy.deepcopy(self.engine.get_active_subnet(True)), inputs.cuda()))
89421520

Thanks for the help

from torchprofile.