同一份训练模型的代码在使用CPU和GPU运行时梯度存在明显差异 about paddle HOT 7 OPEN

从你的输出结果看不出来啥，是没有把梯度之类的放上去么

from paddle.

从你的输出结果看不出来啥，是没有把梯度之类的放上去么

对不起，昨天贴漏了。这是用chebyshev距离，得到的每层的梯度差值。
conv2_mutated.bias: 梯度数据一致
conv2_mutated.weight: 梯度数据不一致, 差值:5.5
grad_cpu[conv2_mutated.weight]: [[[[-4.93047656e+04 -2.00046797e+04 -1.62369312e+03 7.59652783e+03
-5.48374883e+04]
[-2.40742212e+03 -2.21753438e+04 -1.43452900e+04 -1.93090723e+04
-1.27709398e+05]
[ 2.43435410e+04 -7.57103906e+03 -9.44760000e+04 -2.01418578e+05
-1.52792062e+05]
[ 7.07376758e+03 1.35068535e+04 -1.40223578e+05 -2.37257938e+05
-2.44707812e+03]
[-6.05375928e+03 -1.56677861e+04 -1.24804016e+05 -1.57144484e+05
1.09178383e+05]]

[[-3.72414969e+05 -3.73191812e+05 -2.26085250e+05 7.64871500e+05
4.97877281e+05]
[-3.47680719e+05 -3.89100438e+05 -3.51337438e+05 5.48387688e+05
8.64763438e+05]
[-2.70533875e+05 -3.60679156e+05 -3.66796094e+05 1.15022633e+05
8.48278188e+05]
[-2.34205922e+05 -3.20030219e+05 -3.39195125e+05 -1.02370727e+05
9.38136875e+05]
[-2.68674812e+05 -3.85438812e+05 -2.50196875e+05 -2.68981562e+05
9.93834875e+05]]

[[ 5.23475703e+04 5.49324570e+04 -1.76759875e+05 1.30762588e+06
1.24978750e+06]
[-2.12730527e+04 6.70919922e+04 -1.06152617e+05 1.45183050e+06
1.10874450e+06]
[-6.74717969e+04 -2.08249316e+04 -1.37789707e+04 1.46574575e+06
6.68378000e+05]
[-5.87182344e+04 -1.46308789e+04 -1.61792141e+05 1.22808212e+06
1.15342712e+06]
[ 7.15725625e+04 1.50465039e+04 -1.45600344e+05 5.29076312e+05
8.43831500e+05]]

[[ 6.55417252e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 6.55417252e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]]

[[-1.50522844e+05 -2.35194312e+05 -1.37334016e+05 -6.06970469e+04
5.99827375e+05]
[-1.92600547e+05 -2.72019969e+05 -2.48360828e+05 -2.65046812e+05
2.52831797e+05]
[-1.68689031e+05 -2.54523062e+05 -1.90627234e+05 -4.63715031e+05
1.61123250e+05]
[-2.40060688e+05 -1.36298906e+05 -1.50286172e+05 -6.30319562e+05
-1.12596094e+04]
[-2.59171109e+05 -1.17558016e+05 -1.12840672e+05 -6.09054438e+05
-1.14436312e+06]]

[[-2.62650875e+05 -1.16864781e+05 -2.15904000e+05 1.72069212e+06
1.94082650e+06]
[-2.53180797e+05 -2.93392219e+05 -1.46332250e+05 1.56495088e+06
1.69055150e+06]
[-2.48327797e+05 -3.12065312e+05 -2.37632547e+05 1.11853650e+06
1.74028262e+06]
[-1.73032359e+05 -2.89476625e+05 -3.02409438e+05 6.62414250e+05
1.57581350e+06]
[-1.86105266e+05 -2.06310969e+05 -2.10907578e+05 -7.55981406e+04
1.25849225e+06]]]

[[[ 2.69577531e+05 2.92178375e+05 1.19407650e+06 1.26323875e+06
9.03335062e+05]
[ 3.85040438e+05 3.34183562e+05 2.97996656e+05 1.23551725e+06
9.29394062e+05]
[ 3.06816500e+05 2.32322734e+05 3.01556219e+05 3.59404125e+05
4.37201281e+05]
[ 6.38243562e+05 1.76048453e+05 2.99430156e+05 4.63545656e+05
2.81540719e+05]
[ 1.25954912e+06 5.14187406e+05 5.08445844e+05 6.36706875e+05
6.04375312e+05]]

[[ 3.86893200e+06 4.05173050e+06 4.45501050e+06 4.21305000e+06
3.39742900e+06]
[ 4.48155150e+06 4.25043850e+06 4.33898650e+06 4.47497000e+06
3.67779075e+06]
[ 4.72280900e+06 4.95814600e+06 4.07699825e+06 4.42552650e+06
3.77526050e+06]
[ 4.83503350e+06 5.12743500e+06 4.12285075e+06 4.53620300e+06
3.67631925e+06]
[ 4.77345950e+06 4.84331450e+06 4.48067000e+06 4.04141900e+06
3.70734275e+06]]

[[ 7.03271688e+05 2.69933469e+05 -4.23030078e+04 -5.50020898e+03
1.76843066e+03]
[ 1.23826400e+06 2.12564141e+05 -4.16835859e+04 1.05213877e+04
-1.23053943e+03]
[ 1.34406138e+06 1.40702172e+05 -1.23120605e+04 5.62885156e+04
2.11702285e+04]
[ 1.40762350e+06 1.92766781e+05 2.50885609e+05 5.62290352e+04
2.75023438e+04]
[ 1.23924200e+06 1.30267900e+06 3.66235188e+05 1.42298531e+05
1.46462781e+05]]

[[-4.24267139e+03 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[-4.24267188e+03 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]]

[[ 2.88902300e+06 2.85713100e+06 1.98817400e+06 2.31171900e+06
2.93669525e+06]
[ 2.88224100e+06 3.37476100e+06 2.37467850e+06 2.53130900e+06
2.85721725e+06]
[ 2.03649662e+06 2.96671000e+06 2.87354950e+06 1.30968825e+06
2.20598125e+06]
[ 1.68180438e+06 2.90120000e+06 3.35519125e+06 2.22286375e+06
1.64376200e+06]
[ 9.88009750e+05 2.37718425e+06 3.19369625e+06 2.03298588e+06
1.51434762e+06]]

[[ 3.93839175e+06 2.78740325e+06 2.82867150e+06 3.36569650e+06
2.41877075e+06]
[ 3.81958700e+06 2.87860000e+06 2.10766100e+06 3.14558650e+06
2.53024850e+06]
[ 4.31836000e+06 3.76848725e+06 1.55136875e+06 3.38057250e+06
2.86755175e+06]
[ 4.11898125e+06 4.54822500e+06 2.07924288e+06 2.66027525e+06
2.71566175e+06]
[ 3.63501600e+06 4.39152950e+06 2.42983625e+06 2.26300450e+06
2.27944675e+06]]]

[[[-1.05050531e+05 -1.31717109e+05 -6.34260188e+05 -9.56963438e+05
-6.74814375e+05]
[-4.52676523e+04 -7.48202656e+04 -1.50089391e+05 -8.61050438e+05
-5.14429969e+05]
[-7.90492109e+04 -2.52166328e+04 -1.00111812e+05 -2.55426734e+05
-1.77222375e+05]
[-3.93504781e+05 -2.51343531e+05 -5.58612422e+04 -9.72137695e+03
-1.15256219e+05]
[-6.20196312e+05 -2.23881969e+05 -1.22978680e+05 -1.30920500e+05
-3.12577250e+05]]

[[-2.55336250e+06 -2.88503975e+06 -3.17217325e+06 -2.82195600e+06
-2.56058800e+06]
[-3.03622050e+06 -3.00843250e+06 -2.97326750e+06 -3.10830600e+06
-2.77870450e+06]
[-3.48695700e+06 -3.44389400e+06 -2.88144050e+06 -2.87904425e+06
-2.74157325e+06]
[-3.89350325e+06 -3.88699925e+06 -2.91640400e+06 -2.78343650e+06
-2.81675900e+06]
[-3.90409850e+06 -3.72164925e+06 -2.94660375e+06 -2.61625925e+06
-2.59234125e+06]]

[[ 4.85538672e+04 -1.86045266e+05 -3.41200219e+05 -1.41402119e+04
7.70520898e+03]
[-9.20572344e+04 1.09843438e+05 -2.94303906e+05 1.23711777e+04
8.47314746e+03]
[-4.05993828e+04 2.74464781e+05 -3.07190875e+05 -9.89758398e+03
-1.53972500e+04]
[-1.71568500e+05 2.72754188e+05 -1.20289203e+05 -1.70353965e+04
-1.57783828e+04]
[-5.17239656e+05 -3.38249031e+05 -3.41111406e+04 -6.68130000e+04
-7.20203672e+04]]

[[ 1.26366481e-01 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 1.26366481e-01 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]]

[[-2.00877638e+06 -1.62695862e+06 -1.55209325e+06 -2.06723350e+06
-1.95982362e+06]
[-1.79910900e+06 -1.77862662e+06 -1.60656762e+06 -1.93451488e+06
-2.18434625e+06]
[-1.47162238e+06 -1.34126988e+06 -1.59201450e+06 -1.24523125e+06
-1.86062212e+06]
[-1.02599238e+06 -1.42478938e+06 -1.62417075e+06 -1.62493750e+06
-1.51670512e+06]
[-1.17623412e+06 -1.50951075e+06 -1.79247950e+06 -1.36786138e+06
-1.04653788e+06]]

[[-2.32106925e+06 -1.82371925e+06 -2.55469250e+06 -2.14721675e+06
-1.33473812e+06]
[-2.09480162e+06 -1.58117225e+06 -2.11945175e+06 -2.26724525e+06
-1.54018638e+06]
[-2.39881125e+06 -1.97004888e+06 -1.61074050e+06 -2.45722800e+06
-1.88083612e+06]
[-2.18387850e+06 -2.45760475e+06 -1.64466275e+06 -1.96643775e+06
-1.69771025e+06]
[-1.98372688e+06 -2.44572050e+06 -1.76979675e+06 -1.49244875e+06
-1.72228388e+06]]]

...

[[[ 5.20250469e+05 6.22927938e+05 3.58193945e+04 4.60490586e+04
-3.99414038e+03]
[ 4.23788906e+05 5.95167000e+05 3.62784688e+04 2.53242637e+04
-2.57375176e+04]
[ 5.38334250e+05 2.15557500e+05 -1.05876309e+04 -1.01525859e+05
-2.78481504e+04]
[ 6.43328312e+05 7.03256562e+05 3.77605688e+05 2.47941766e+05
4.37818938e+05]
[ 1.02748306e+06 1.01651394e+06 7.03210125e+05 7.34159125e+05
7.76076938e+05]]

[[ 5.95240300e+06 5.61964050e+06 4.41002150e+06 3.72228450e+06
3.33303125e+06]
[ 5.86676500e+06 5.70572550e+06 4.64509650e+06 3.95739675e+06
3.57594225e+06]
[ 6.28857300e+06 5.91527600e+06 5.18006750e+06 4.09302800e+06
3.98143650e+06]
[ 6.14971750e+06 5.83138850e+06 5.43428250e+06 4.34911350e+06
4.45576550e+06]
[ 5.75666950e+06 5.80031350e+06 5.27774850e+06 4.86971650e+06
4.73697450e+06]]

[[ 4.63367656e+05 1.77958109e+05 -3.07492031e+04 -7.00024658e+02
-2.24081268e+01]
[ 4.29497375e+05 4.58173781e+05 -2.56045957e+04 -2.67829803e+02
1.97553268e+02]
[ 4.79467500e+05 7.29339375e+05 1.96105137e+04 3.22385625e+04
1.96344590e+04]
[ 8.61398812e+05 8.11065625e+05 -9.00205312e+04 4.09493633e+04
3.24354219e+04]
[ 1.49420462e+06 9.17384312e+05 1.82592312e+05 1.98473250e+05
2.21392922e+05]]

[[ 1.45108031e+05 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 1.45108031e+05 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]]

[[ 3.72800200e+06 2.85056000e+06 2.25826700e+06 2.19885075e+06
2.05793662e+06]
[ 3.81520425e+06 3.29211200e+06 2.47528650e+06 2.01414488e+06
2.12070225e+06]
[ 3.39744250e+06 3.86205375e+06 2.25771950e+06 1.79806575e+06
1.84930988e+06]
[ 3.48766650e+06 4.14789375e+06 2.83664650e+06 2.65714750e+06
2.32061550e+06]
[ 2.64678625e+06 3.41338375e+06 2.59947725e+06 2.62394375e+06
2.04897200e+06]]

[[ 3.16691350e+06 2.29182500e+06 2.74246425e+06 2.57032525e+06
2.37738325e+06]
[ 3.77226375e+06 2.13665525e+06 2.40505150e+06 2.43641000e+06
2.37795700e+06]
[ 4.18520825e+06 2.70071900e+06 2.53829850e+06 2.81433650e+06
2.70602025e+06]
[ 4.49464150e+06 2.98756950e+06 2.94390775e+06 2.91742450e+06
3.08791325e+06]
[ 4.29595600e+06 3.45484525e+06 3.02856725e+06 2.37522150e+06
2.93291875e+06]]]

[[[-4.49195844e+05 -7.77083438e+05 -1.38768862e+06 -1.38334212e+06
-1.52959750e+06]
[ 3.84381616e+03 -4.47577906e+05 3.68553000e+05 -1.08843675e+06
-1.17475375e+06]
[-9.81648926e+03 4.33266328e+04 4.42214406e+05 5.08156055e+04
-3.44537250e+05]
[-2.80629875e+05 2.60619219e+05 3.21178875e+05 3.83826969e+05
1.30072062e+05]
[-7.55292656e+04 3.73847156e+05 3.73405438e+05 3.64018500e+05
4.87516016e+04]]

[[-6.26523100e+06 -7.14752350e+06 -7.44298900e+06 -6.45251850e+06
-5.88915050e+06]
[-7.13857200e+06 -6.41100650e+06 -6.45496950e+06 -6.32590950e+06
-5.79299000e+06]
[-7.69658900e+06 -6.77344350e+06 -5.57806000e+06 -5.31458200e+06
-5.37810800e+06]
[-6.86502100e+06 -6.76391350e+06 -5.29246100e+06 -4.86173650e+06
-4.75874200e+06]
[-6.88654000e+06 -6.60087200e+06 -6.09220250e+06 -4.41331500e+06
-4.64545950e+06]]

[[-9.18106812e+05 -4.78329406e+05 1.73206219e+05 -9.97502578e+04
2.16220044e+03]
[-1.24107862e+06 -2.35839609e+05 3.53121688e+05 -3.15251782e+03
4.98549219e+03]
[-1.15401375e+06 2.42945484e+05 2.54503234e+05 2.25094590e+04
9.54036426e+03]
[-1.46346675e+06 2.91131781e+05 3.63213906e+05 2.49507832e+04
5.31238086e+03]
[-2.12476300e+06 -1.34974612e+06 4.04412812e+05 2.31570254e+04
6.88718457e+03]]

[[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]]

[[-4.34047300e+06 -2.96181725e+06 -2.73728450e+06 -2.93566200e+06
-3.93802600e+06]
[-3.85346275e+06 -4.26613600e+06 -2.74547375e+06 -2.52914875e+06
-3.55543600e+06]
[-3.16147625e+06 -4.72103300e+06 -3.66312175e+06 -1.48799812e+06
-3.55604375e+06]
[-2.41294200e+06 -4.63307350e+06 -3.86152025e+06 -2.97551900e+06
-2.56757725e+06]
[-2.38332200e+06 -4.46809000e+06 -3.99980400e+06 -4.02735425e+06
-3.01030825e+06]]

[[-6.52106050e+06 -4.40498900e+06 -5.00024300e+06 -5.13812600e+06
-3.72370575e+06]
[-5.90379950e+06 -4.09960900e+06 -4.58255600e+06 -5.21934500e+06
-4.20278200e+06]
[-5.43634150e+06 -3.59341375e+06 -2.60658150e+06 -5.11307500e+06
-4.28457100e+06]
[-5.24009300e+06 -3.72630525e+06 -3.35867550e+06 -4.11213100e+06
-3.99290700e+06]
[-4.64898450e+06 -3.68076375e+06 -3.75218675e+06 -3.65572600e+06
-3.54080250e+06]]]

[[[-5.84712031e+04 -2.84318906e+04 1.84151641e+04 -1.02213701e+04
6.58639922e+04]
[-3.29843359e+04 -5.59876758e+04 2.23465938e+04 1.31809172e+05
1.57803938e+05]
[ 6.77605225e+03 -1.65869160e+04 2.62701348e+04 1.36475016e+05
2.55714984e+05]
[ 1.52482219e+05 5.13515781e+04 1.95669344e+05 2.73174656e+05
3.78915219e+05]
[ 1.99063078e+05 3.22968312e+05 3.78808312e+05 3.56359844e+05
5.01188375e+05]]

[[ 8.56032000e+05 2.06287234e+05 2.93098975e+06 4.36367550e+06
4.50479650e+06]
[ 8.22811312e+05 2.10196000e+05 2.93392300e+06 4.30148150e+06
4.64109250e+06]
[ 7.84100188e+05 1.36449641e+05 2.86687975e+06 3.67465350e+06
4.60341900e+06]
[ 8.66368438e+05 1.93266609e+05 2.80164850e+06 3.52035375e+06
4.76735650e+06]
[ 9.63635375e+05 2.48373641e+05 2.80603475e+06 3.47360225e+06
4.35973950e+06]]

[[ 2.67236425e+06 2.48789425e+06 4.12391438e+05 1.71626362e+06
1.67800300e+06]
[ 2.64977550e+06 2.52618650e+06 2.49904891e+05 1.84827575e+06
1.53024638e+06]
[ 2.60319900e+06 2.51487525e+06 1.09993836e+05 1.90208250e+06
1.43907375e+06]
[ 2.57520650e+06 2.49688700e+06 2.80996836e+04 1.81784625e+06
1.99435650e+06]
[ 2.45433750e+06 2.42540775e+06 7.75740938e+04 1.15300938e+06
1.86976725e+06]]

[[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]]

[[ 9.77942812e+04 2.25403675e+06 1.40720775e+06 1.56448975e+06
2.61875200e+06]
[ 1.14564316e+04 2.09822025e+06 1.37892938e+06 1.27304200e+06
2.59267800e+06]
[ 2.40762207e+04 1.91357138e+06 1.26394338e+06 1.18308912e+06
2.47610125e+06]
[ 1.39406953e+05 1.92790712e+06 1.19249425e+06 1.11586250e+06
1.87980012e+06]
[ 1.47640859e+05 1.74034625e+06 1.00478412e+06 8.37003812e+05
5.14684406e+05]]

[[ 3.22218000e+06 3.57122825e+06 2.18848475e+06 4.58014950e+06
5.06407850e+06]
[ 3.11523400e+06 3.62710075e+06 2.08093162e+06 4.35139300e+06
4.80212950e+06]
[ 3.14020575e+06 3.72389175e+06 1.98744550e+06 3.90036775e+06
4.70964200e+06]
[ 3.12091875e+06 3.77045075e+06 1.89577175e+06 3.46204075e+06
4.57214200e+06]
[ 2.89391425e+06 3.62460325e+06 1.76906938e+06 2.77892250e+06
4.46515000e+06]]]]
grad_gpu[conv2_mutated.weight]: [[[[-4.93047734e+04 -2.00046816e+04 -1.62369141e+03 7.59652588e+03
-5.48375039e+04]
[-2.40742041e+03 -2.21753457e+04 -1.43452900e+04 -1.93090645e+04
-1.27709391e+05]
[ 2.43435391e+04 -7.57104834e+03 -9.44760000e+04 -2.01418594e+05
-1.52792047e+05]
[ 7.07376807e+03 1.35068516e+04 -1.40223547e+05 -2.37257938e+05
-2.44709131e+03]
[-6.05375879e+03 -1.56677871e+04 -1.24803984e+05 -1.57144500e+05
1.09178445e+05]]

[[-3.72415094e+05 -3.73191844e+05 -2.26085266e+05 7.64871625e+05
4.97876719e+05]
[-3.47680719e+05 -3.89100500e+05 -3.51337562e+05 5.48388688e+05
8.64762250e+05]
[-2.70533906e+05 -3.60679188e+05 -3.66796125e+05 1.15022586e+05
8.48277250e+05]
[-2.34206016e+05 -3.20029969e+05 -3.39194750e+05 -1.02370867e+05
9.38137000e+05]
[-2.68674750e+05 -3.85438688e+05 -2.50196562e+05 -2.68982000e+05
9.93834250e+05]]

[[ 5.23476016e+04 5.49323750e+04 -1.76759875e+05 1.30762575e+06
1.24978750e+06]
[-2.12730605e+04 6.70920625e+04 -1.06152633e+05 1.45183050e+06
1.10874450e+06]
[-6.74718594e+04 -2.08249062e+04 -1.37789756e+04 1.46574600e+06
6.68377625e+05]
[-5.87181133e+04 -1.46309453e+04 -1.61792156e+05 1.22808225e+06
1.15342688e+06]
[ 7.15725703e+04 1.50467930e+04 -1.45600359e+05 5.29076438e+05
8.43831438e+05]]

[[ 6.55417252e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 6.55417252e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]]

[[-1.50522828e+05 -2.35194391e+05 -1.37334094e+05 -6.06971406e+04
5.99826812e+05]
[-1.92600422e+05 -2.72019844e+05 -2.48360812e+05 -2.65046844e+05
2.52832375e+05]
[-1.68689156e+05 -2.54522750e+05 -1.90627266e+05 -4.63714688e+05
1.61123375e+05]
[-2.40060688e+05 -1.36298641e+05 -1.50286031e+05 -6.30319625e+05
-1.12596191e+04]
[-2.59170984e+05 -1.17558055e+05 -1.12840602e+05 -6.09054438e+05
-1.14436312e+06]]

[[-2.62651000e+05 -1.16864727e+05 -2.15903594e+05 1.72069312e+06
1.94082612e+06]
[-2.53181109e+05 -2.93391906e+05 -1.46332344e+05 1.56495112e+06
1.69054988e+06]
[-2.48328031e+05 -3.12065438e+05 -2.37632844e+05 1.11853675e+06
1.74028250e+06]
[-1.73032984e+05 -2.89477375e+05 -3.02409656e+05 6.62414438e+05
1.57581250e+06]
[-1.86105141e+05 -2.06310688e+05 -2.10907422e+05 -7.55980078e+04
1.25849162e+06]]]

[[[ 2.69577438e+05 2.92178375e+05 1.19407638e+06 1.26323912e+06
9.03335062e+05]
[ 3.85040312e+05 3.34183625e+05 2.97996594e+05 1.23551738e+06
9.29394062e+05]
[ 3.06816562e+05 2.32322766e+05 3.01556188e+05 3.59404125e+05
4.37201250e+05]
[ 6.38243625e+05 1.76048422e+05 2.99430094e+05 4.63545531e+05
2.81540750e+05]
[ 1.25954925e+06 5.14187344e+05 5.08445875e+05 6.36706812e+05
6.04375250e+05]]

[[ 3.86893325e+06 4.05173125e+06 4.45501000e+06 4.21304950e+06
3.39742825e+06]
[ 4.48155050e+06 4.25043750e+06 4.33898750e+06 4.47496950e+06
3.67779125e+06]
[ 4.72280950e+06 4.95814600e+06 4.07699925e+06 4.42552800e+06
3.77526075e+06]
[ 4.83503450e+06 5.12743400e+06 4.12285025e+06 4.53620550e+06
3.67631875e+06]
[ 4.77345650e+06 4.84331350e+06 4.48066900e+06 4.04141925e+06
3.70734300e+06]]

[[ 7.03271750e+05 2.69933469e+05 -4.23029961e+04 -5.50020801e+03
1.76843079e+03]
[ 1.23826362e+06 2.12564078e+05 -4.16835938e+04 1.05213877e+04
-1.23053931e+03]
[ 1.34406150e+06 1.40702234e+05 -1.23120576e+04 5.62885078e+04
2.11702305e+04]
[ 1.40762362e+06 1.92766984e+05 2.50885625e+05 5.62290273e+04
2.75023418e+04]
[ 1.23924212e+06 1.30267900e+06 3.66235125e+05 1.42298531e+05
1.46462766e+05]]

[[-4.24267139e+03 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[-4.24267139e+03 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]]

[[ 2.88902350e+06 2.85713225e+06 1.98817400e+06 2.31171575e+06
2.93669625e+06]
[ 2.88224000e+06 3.37476075e+06 2.37467825e+06 2.53130925e+06
2.85721575e+06]
[ 2.03649662e+06 2.96671000e+06 2.87355000e+06 1.30968850e+06
2.20598125e+06]
[ 1.68180425e+06 2.90120125e+06 3.35518975e+06 2.22286400e+06
1.64376262e+06]
[ 9.88009688e+05 2.37718275e+06 3.19369775e+06 2.03298600e+06
1.51434700e+06]]

[[ 3.93839175e+06 2.78740400e+06 2.82867150e+06 3.36569425e+06
2.41876975e+06]
[ 3.81958800e+06 2.87859925e+06 2.10766125e+06 3.14558600e+06
2.53024800e+06]
[ 4.31836000e+06 3.76848875e+06 1.55136925e+06 3.38057225e+06
2.86755075e+06]
[ 4.11898075e+06 4.54822650e+06 2.07924262e+06 2.66027525e+06
2.71566225e+06]
[ 3.63501550e+06 4.39153000e+06 2.42983650e+06 2.26300375e+06
2.27944725e+06]]]

[[[-1.05050523e+05 -1.31717156e+05 -6.34259938e+05 -9.56963500e+05
-6.74814312e+05]
[-4.52676523e+04 -7.48202656e+04 -1.50089438e+05 -8.61050312e+05
-5.14430156e+05]
[-7.90492031e+04 -2.52166348e+04 -1.00111797e+05 -2.55426844e+05
-1.77222359e+05]
[-3.93505000e+05 -2.51343516e+05 -5.58612461e+04 -9.72137305e+03
-1.15256141e+05]
[-6.20196500e+05 -2.23881906e+05 -1.22978711e+05 -1.30920602e+05
-3.12577375e+05]]

[[-2.55336275e+06 -2.88504050e+06 -3.17217225e+06 -2.82195525e+06
-2.56058850e+06]
[-3.03622100e+06 -3.00843275e+06 -2.97326850e+06 -3.10830650e+06
-2.77870450e+06]
[-3.48695750e+06 -3.44389325e+06 -2.88144150e+06 -2.87904375e+06
-2.74157225e+06]
[-3.89350325e+06 -3.88699800e+06 -2.91640425e+06 -2.78343650e+06
-2.81675875e+06]
[-3.90409825e+06 -3.72165075e+06 -2.94660350e+06 -2.61625950e+06
-2.59234075e+06]]

[[ 4.85538242e+04 -1.86045031e+05 -3.41200281e+05 -1.41402100e+04
7.70520898e+03]
[-9.20572578e+04 1.09843461e+05 -2.94303969e+05 1.23711787e+04
8.47314746e+03]
[-4.05992930e+04 2.74464906e+05 -3.07190812e+05 -9.89759180e+03
-1.53972441e+04]
[-1.71568859e+05 2.72754250e+05 -1.20289148e+05 -1.70354121e+04
-1.57783809e+04]
[-5.17239688e+05 -3.38249094e+05 -3.41111406e+04 -6.68130156e+04
-7.20203984e+04]]

[[ 1.26366481e-01 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 1.26366481e-01 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]]

[[-2.00877562e+06 -1.62695862e+06 -1.55209288e+06 -2.06723475e+06
-1.95982375e+06]
[-1.79910900e+06 -1.77862738e+06 -1.60656750e+06 -1.93451612e+06
-2.18434675e+06]
[-1.47162288e+06 -1.34127000e+06 -1.59201450e+06 -1.24523138e+06
-1.86062175e+06]
[-1.02599194e+06 -1.42478925e+06 -1.62417088e+06 -1.62493725e+06
-1.51670500e+06]
[-1.17623400e+06 -1.50951100e+06 -1.79247938e+06 -1.36786125e+06
-1.04653731e+06]]

[[-2.32106950e+06 -1.82371938e+06 -2.55469225e+06 -2.14721575e+06
-1.33473825e+06]
[-2.09480125e+06 -1.58117188e+06 -2.11944975e+06 -2.26724450e+06
-1.54018650e+06]
[-2.39881150e+06 -1.97004912e+06 -1.61074038e+06 -2.45722875e+06
-1.88083525e+06]
[-2.18387750e+06 -2.45760450e+06 -1.64466275e+06 -1.96643762e+06
-1.69771038e+06]
[-1.98372750e+06 -2.44572025e+06 -1.76979662e+06 -1.49244862e+06
-1.72228388e+06]]]

...

[[[ 5.20250531e+05 6.22928125e+05 3.58193906e+04 4.60490586e+04
-3.99414136e+03]
[ 4.23788906e+05 5.95167062e+05 3.62784961e+04 2.53242637e+04
-2.57375137e+04]
[ 5.38334188e+05 2.15557531e+05 -1.05876318e+04 -1.01525852e+05
-2.78481504e+04]
[ 6.43328250e+05 7.03256562e+05 3.77605625e+05 2.47941688e+05
4.37818906e+05]
[ 1.02748281e+06 1.01651406e+06 7.03210062e+05 7.34159000e+05
7.76076938e+05]]

[[ 5.95240300e+06 5.61964150e+06 4.41002150e+06 3.72228450e+06
3.33303175e+06]
[ 5.86676550e+06 5.70572500e+06 4.64509600e+06 3.95739700e+06
3.57594150e+06]
[ 6.28857200e+06 5.91527550e+06 5.18006800e+06 4.09302800e+06
3.98143650e+06]
[ 6.14971900e+06 5.83138950e+06 5.43428400e+06 4.34911300e+06
4.45576500e+06]
[ 5.75667050e+06 5.80031400e+06 5.27774850e+06 4.86971650e+06
4.73697550e+06]]

[[ 4.63367625e+05 1.77958125e+05 -3.07492051e+04 -7.00024658e+02
-2.24081230e+01]
[ 4.29497469e+05 4.58173844e+05 -2.56045957e+04 -2.67829834e+02
1.97553268e+02]
[ 4.79467562e+05 7.29339312e+05 1.96105117e+04 3.22385664e+04
1.96344551e+04]
[ 8.61398625e+05 8.11065750e+05 -9.00205000e+04 4.09493672e+04
3.24354160e+04]
[ 1.49420462e+06 9.17384375e+05 1.82592312e+05 1.98473297e+05
2.21392969e+05]]

[[ 1.45108016e+05 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 1.45108000e+05 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]]

[[ 3.72800225e+06 2.85056000e+06 2.25826725e+06 2.19885175e+06
2.05793625e+06]
[ 3.81520500e+06 3.29211175e+06 2.47528650e+06 2.01414475e+06
2.12070075e+06]
[ 3.39744325e+06 3.86205425e+06 2.25771975e+06 1.79806525e+06
1.84931000e+06]
[ 3.48766725e+06 4.14789475e+06 2.83664725e+06 2.65714800e+06
2.32061675e+06]
[ 2.64678650e+06 3.41338475e+06 2.59947625e+06 2.62394325e+06
2.04897250e+06]]

[[ 3.16691425e+06 2.29182500e+06 2.74246375e+06 2.57032650e+06
2.37738350e+06]
[ 3.77226475e+06 2.13665650e+06 2.40505200e+06 2.43641100e+06
2.37795725e+06]
[ 4.18520750e+06 2.70072025e+06 2.53829825e+06 2.81433650e+06
2.70602050e+06]
[ 4.49464000e+06 2.98756900e+06 2.94390850e+06 2.91742475e+06
3.08791300e+06]
[ 4.29595500e+06 3.45484550e+06 3.02856675e+06 2.37522175e+06
2.93291875e+06]]]

[[[-4.49196000e+05 -7.77083188e+05 -1.38768850e+06 -1.38334212e+06
-1.52959738e+06]
[ 3.84387451e+03 -4.47577844e+05 3.68553156e+05 -1.08843700e+06
-1.17475388e+06]
[-9.81646875e+03 4.33266836e+04 4.42214344e+05 5.08155586e+04
-3.44537250e+05]
[-2.80630000e+05 2.60619219e+05 3.21178969e+05 3.83827094e+05
1.30072078e+05]
[-7.55293203e+04 3.73847188e+05 3.73405469e+05 3.64018656e+05
4.87516484e+04]]

[[-6.26522700e+06 -7.14752300e+06 -7.44299350e+06 -6.45252250e+06
-5.88915150e+06]
[-7.13857300e+06 -6.41100900e+06 -6.45496600e+06 -6.32591250e+06
-5.79298950e+06]
[-7.69659400e+06 -6.77343800e+06 -5.57806050e+06 -5.31458400e+06
-5.37810650e+06]
[-6.86502350e+06 -6.76391800e+06 -5.29246100e+06 -4.86173750e+06
-4.75874050e+06]
[-6.88654050e+06 -6.60087700e+06 -6.09220550e+06 -4.41331400e+06
-4.64546100e+06]]

[[-9.18106750e+05 -4.78329406e+05 1.73206219e+05 -9.97502734e+04
2.16220068e+03]
[-1.24107812e+06 -2.35839875e+05 3.53121719e+05 -3.15251782e+03
4.98549121e+03]
[-1.15401375e+06 2.42945562e+05 2.54503234e+05 2.25094629e+04
9.54036328e+03]
[-1.46346825e+06 2.91131469e+05 3.63213938e+05 2.49507793e+04
5.31238086e+03]
[-2.12476275e+06 -1.34974625e+06 4.04412750e+05 2.31570273e+04
6.88718457e+03]]

[[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]]

[[-4.34047400e+06 -2.96181825e+06 -2.73728400e+06 -2.93566250e+06
-3.93802550e+06]
[-3.85346275e+06 -4.26613650e+06 -2.74547450e+06 -2.52914925e+06
-3.55543525e+06]
[-3.16147650e+06 -4.72103250e+06 -3.66312250e+06 -1.48799738e+06
-3.55604550e+06]
[-2.41294225e+06 -4.63307400e+06 -3.86152175e+06 -2.97551850e+06
-2.56757775e+06]
[-2.38332250e+06 -4.46808950e+06 -3.99980250e+06 -4.02735725e+06
-3.01030875e+06]]

[[-6.52106100e+06 -4.40498950e+06 -5.00024300e+06 -5.13812700e+06
-3.72370300e+06]
[-5.90380000e+06 -4.09960950e+06 -4.58255600e+06 -5.21934450e+06
-4.20278300e+06]
[-5.43633950e+06 -3.59341425e+06 -2.60658100e+06 -5.11307400e+06
-4.28457000e+06]
[-5.24009500e+06 -3.72630475e+06 -3.35867200e+06 -4.11213425e+06
-3.99290725e+06]
[-4.64898500e+06 -3.68076275e+06 -3.75218225e+06 -3.65572250e+06
-3.54080125e+06]]]

[[[-5.84711953e+04 -2.84319082e+04 1.84151680e+04 -1.02213711e+04
6.58639844e+04]
[-3.29843438e+04 -5.59876992e+04 2.23465938e+04 1.31809188e+05
1.57803938e+05]
[ 6.77605371e+03 -1.65869141e+04 2.62701270e+04 1.36474984e+05
2.55715000e+05]
[ 1.52482266e+05 5.13515664e+04 1.95669312e+05 2.73174594e+05
3.78915125e+05]
[ 1.99063047e+05 3.22968312e+05 3.78808406e+05 3.56359844e+05
5.01188250e+05]]

[[ 8.56031938e+05 2.06286984e+05 2.93098975e+06 4.36367650e+06
4.50479650e+06]
[ 8.22811562e+05 2.10196078e+05 2.93392375e+06 4.30148050e+06
4.64109150e+06]
[ 7.84100188e+05 1.36449516e+05 2.86687950e+06 3.67465350e+06
4.60341850e+06]
[ 8.66368438e+05 1.93266688e+05 2.80164925e+06 3.52035375e+06
4.76735600e+06]
[ 9.63635438e+05 2.48373734e+05 2.80603475e+06 3.47360250e+06
4.35973950e+06]]

[[ 2.67236425e+06 2.48789375e+06 4.12391375e+05 1.71626375e+06
1.67800288e+06]
[ 2.64977475e+06 2.52618600e+06 2.49904922e+05 1.84827562e+06
1.53024662e+06]
[ 2.60319950e+06 2.51487550e+06 1.09993836e+05 1.90208250e+06
1.43907412e+06]
[ 2.57520700e+06 2.49688775e+06 2.80996797e+04 1.81784625e+06
1.99435588e+06]
[ 2.45433725e+06 2.42540750e+06 7.75740703e+04 1.15300888e+06
1.86976688e+06]]

[[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
0.00000000e+00]]

[[ 9.77942031e+04 2.25403700e+06 1.40720812e+06 1.56449038e+06
2.61875250e+06]
[ 1.14563027e+04 2.09822125e+06 1.37892938e+06 1.27304225e+06
2.59267775e+06]
[ 2.40764434e+04 1.91357100e+06 1.26394350e+06 1.18308925e+06
2.47610150e+06]
[ 1.39406812e+05 1.92790700e+06 1.19249425e+06 1.11586262e+06
1.87980050e+06]
[ 1.47640844e+05 1.74034650e+06 1.00478400e+06 8.37003688e+05
5.14684688e+05]]

[[ 3.22218100e+06 3.57122850e+06 2.18848450e+06 4.58015000e+06
5.06408000e+06]
[ 3.11523475e+06 3.62710100e+06 2.08093238e+06 4.35139200e+06
4.80212750e+06]
[ 3.14020600e+06 3.72389200e+06 1.98744612e+06 3.90036725e+06
4.70964000e+06]
[ 3.12091850e+06 3.77045100e+06 1.89577200e+06 3.46204025e+06
4.57214200e+06]
[ 2.89391450e+06 3.62460400e+06 1.76906912e+06 2.77892275e+06
4.46514950e+06]]]]
conv1_mutated.bias: 梯度数据一致
conv1_mutated.weight: 梯度数据不一致, 差值:6.5
grad_cpu[conv1_mutated.weight]: [[[[ 168852.77 292703.1 170261.11 -720956.6 -408108.2 ]
[ 745185.3 319249. 1086475.9 926748.44 -72331.39 ]
[ 874728.5 128694.13 529293.7 631475.4 181684.31 ]
[ 50063.316 -206407.05 -613202. 42018.844 -407522.88 ]
[ -52574.773 -295366.2 -928816.06 -443487.6 -363558.03 ]]

[[-2516005.8 -4645216.5 -4593647.5 -4013469.2 -4365871.5 ]
[-2208759. -3162624.2 -4341300. -4620410.5 -5163825.5 ]
[-2610586. -3329145.2 -5329222. -5117105. -5546823. ]
[-2635945.5 -2655343. -4025384.8 -4356956.5 -4358073.5 ]
[-3107346.2 -2807685.2 -3768914.5 -4015386. -4468246.5 ]]

[[ 868272.5 764133.5 233899.94 -69916.81 672294.25 ]
[ 1219145.9 1106713.6 1232156.9 863448.8 1034688.1 ]
[ 1201847.8 989381.9 893880.4 1322997. 1381965.5 ]
[ 2139331.5 667783.75 421838.78 1176228.2 2079849.9 ]
[ 2525646. 696495.9 -142261.28 894056.9 2495820.8 ]]

[[ 112190.086 112190.08 0. 0. 0. ]
[ 112190.08 112190.086 0. 0. 0. ]
[ 0. 0. 0. 0. 0. ]
[ 0. 0. 0. 0. 0. ]
[ 0. 0. 0. 0. 0. ]]

[[ 6520251. 5139144.5 5132530. 4131283.5 1840510. ]
[ 7220390. 5583667.5 5006015.5 5282349.5 1990606.2 ]
[ 7797469.5 5600566.5 4719616.5 4520076.5 2386464.8 ]
[ 7443312. 6038686. 5765336. 5182943.5 2478079.8 ]
[ 6553215.5 6359214. 6065333. 4619937. 1756663.6 ]]

[[ -584923.5 -1430988.6 -1186714.2 -1241283. -2743481. ]
[ 245206.77 183358.03 -829450.8 -445376.2 -2089538.8 ]
[ 1592576.4 2236018.8 438159.72 130576.37 543495.9 ]
[ 929307.6 1847269.9 1249681.8 1191646.6 -53711.152]
[ 1342443.4 2388655.2 2063002.1 1719627.8 1104373.9 ]]]]
grad_gpu[conv1_mutated.weight]: [[[[ 168853.19 292702.78 170260.92 -720956.8 -408107.78 ]
[ 745186.5 319249.16 1086475.9 926749.06 -72331.34 ]
[ 874726.94 128694.19 529293. 631474.75 181683.8 ]
[ 50063.34 -206406.6 -613201.5 42018.773 -407522.84 ]
[ -52574.71 -295366.3 -928816.8 -443487.25 -363558.53 ]]

[[-2516006.2 -4645215.5 -4593649. -4013469. -4365873. ]
[-2208758.5 -3162625.2 -4341297. -4620410.5 -5163825.5 ]
[-2610586. -3329146.2 -5329221. -5117106.5 -5546818. ]
[-2635945. -2655343. -4025384. -4356955.5 -4358072. ]
[-3107345.8 -2807686.5 -3768914. -4015390. -4468248. ]]

[[ 868272.4 764132.6 233900.9 -69916.48 672295.4 ]
[ 1219144.5 1106713.2 1232156.8 863448.4 1034688.06 ]
[ 1201848.8 989382.6 893881.5 1322996.9 1381965.5 ]
[ 2139332.8 667783.25 421839.8 1176228.1 2079853.6 ]
[ 2525646. 696496.4 -142261.17 894057.06 2495820. ]]

[[ 112190.086 112190.086 0. 0. 0. ]
[ 112190.086 112190.086 0. 0. 0. ]
[ 0. 0. 0. 0. 0. ]
[ 0. 0. 0. 0. 0. ]
[ 0. 0. 0. 0. 0. ]]

[[ 6520250. 5139144. 5132531. 4131284. 1840511.4 ]
[ 7220393.5 5583668. 5006016. 5282348.5 1990604.8 ]
[ 7797463. 5600565.5 4719617. 4520075. 2386465.2 ]
[ 7443312.5 6038684. 5765336. 5182945. 2478081. ]
[ 6553219.5 6359214.5 6065334. 4619936. 1756663. ]]

[[ -584924.06 -1430985. -1186712.4 -1241281.6 -2743481.2 ]
[ 245207.83 183357.06 -829448.06 -445374.97 -2089537.4 ]
[ 1592576.4 2236019. 438159.4 130575.17 543496.6 ]
[ 929309.4 1847269.8 1249682.4 1191648.4 -53710.863]
[ 1342444.2 2388656.8 2063002.4 1719627.8 1104373.1 ]]]]
tail_fc.bias: 梯度数据一致
linear1_mutated.bias: 梯度数据一致
linear1_mutated.weight: 梯度数据一致
tail_fc.weight: 梯度数据一致

from paddle.

从你的输出结果看不出来啥，是没有把梯度之类的放上去么

diff.txt
我把梯度差异存成了txt

from paddle.

建议能否把代码再进行进一步简化，定位到具体是哪个API存在这个问题。现在看你的结果，就是conv1和conv2这个地方梯度不同是么？如果是的话，我再麻烦相关同学看下。

from paddle.

建议能否把代码再进行进一步简化，定位到具体是哪个API存在这个问题。现在看你的结果，就是conv1和conv2这个地方梯度不同是么？如果是的话，我再麻烦相关同学看下。

我们觉得虽然表面上是conv的问题，但是每个出问题的模型都涉及到了paddle.reciprocal，可能是在倒数这里出现了问题。下面是一个出错的例子
AlexNet-19-112_paddle_cpu.txt
AlexNet-19-112_paddle_gpu.txt
diff.txt

from paddle.

如何怀疑这个API，你那边能否提供最小复现例子？我这边简单测了下，看着前向略有点diff，梯度能对齐。

from paddle.

如何怀疑这个API，你那边能否提供最小复现例子？我这边简单测了下，看着前向略有点diff，梯度能对齐。

https://github.com/LouisChenB/Sample.git 运行test.py可以读取我们之前运行的结果，将会在AlexNet-19-112中生成diff.txt，展示梯度不一致的地方。如果希望复现错误的话，在AlexNet-19-112/case/paddle_gpu/AlexNet-19-112_paddle_gpu.py和AlexNet-19-112/case/paddle_cpu/AlexNet-19-112_paddle_cpu.py中分别添加如下代码即可

input = np.load("../input.npz")
train(input["inp"], input["label"])

from paddle.