Giter VIP home page Giter VIP logo

Comments (9)

avik-pal avatar avik-pal commented on June 2, 2024 1

๐Ÿ˜“ this is the third order requiring a forward over an HVP or a Reverse over Taylor Mode (TaylorDiff.jl) to be remotely efficient.

Firstly I would rewrite the function to use some sort of a custom hessian because you just need the diagonals so Jacobian over Jacobian is a bad idea (performance wise). There was some AAAI paper (I think) which showed how to do diagonal of 2nd order very fast, but I can;t seem to locate the paper rn.

That said, it would still require a custom rrule where you use forwarddiff over HVP. I don't think anyone is actively working on 3rd order (including me), so all I can do is point you to the code which might be helpful

Lux.jl/ext/LuxZygoteExt.jl

Lines 42 to 63 in afd8555

function CRC.rrule(cfg::CRC.RuleConfig{>:CRC.HasReverseMode},
::typeof(__internal_gradient_capture), f::F, x::AbstractArray, ps) where {F}
if !Lux._is_extension_loaded(Val(:ForwardDiff)) || DISABLE_AUTOMATIC_NESTED_AD_SWITCH
if !DISABLE_AUTOMATIC_NESTED_AD_SWITCH
@warn "Load ForwardDiff.jl for better nested AD handling." maxlog=1
end
# Use the AD itself for whatever reason
y, pb_f = CRC.rrule_via_ad(cfg, Zygote.gradient, f, x, ps)
return (first(y),), pb_f
end
y = __internal_gradient_capture(f, x, ps)
โˆ‡internal_gradient_capture = @closure ฮ” -> begin
(ฮ” isa CRC.NoTangent || ฮ” isa CRC.ZeroTangent) &&
return ntuple(Returns(CRC.NoTangent()), 4)
ฮ”_ = reshape(CRC.unthunk(only(ฮ”)), size(x))
โˆ‚x, โˆ‚ps = Lux.__forwarddiff_jvp(
@closure((x, ps)->Zygote.gradient(f, x, ps)), x, ฮ”_, ps)
return CRC.NoTangent(), CRC.NoTangent(), โˆ‚x, โˆ‚ps
end
return y, โˆ‡internal_gradient_capture
end

from lux.jl.

avik-pal avatar avik-pal commented on June 2, 2024 1

No worries, I do think it would be a nice to have feature, if we can get a general implementation for this up and running.

The ๐Ÿ˜“ was for Zygote giving bad error messages. For nested reverse, Zygote becomes type unstable (beyond the simple cases) and then it throws that undefined reference error which can't be parsed by anyone who doesn't already know what is going wrong (i.e. pretty much the opposite of what an error message should do)

from lux.jl.

tansongchen avatar tansongchen commented on June 2, 2024 1

Could you write the mathematical expression for the derivative you want to get? When I hear keywords like "diagonal", "forward over reverse" stuff, I have some confidence to reformulate it to make use of TaylorDiff ๐Ÿค”

from lux.jl.

avik-pal avatar avik-pal commented on June 2, 2024 1

@tansongchen do you think we can rewrite this in-terms of TaylorDiff?

from lux.jl.

tansongchen avatar tansongchen commented on June 2, 2024 1

It would be easy to do once I support chunking (surely will be doing that in a month or two). The model can then be evaluated in one pass of TaylorDiff, so this will be a standard Zygote-over-TaylorDiff overlay which is well understood

from lux.jl.

aksuhton avatar aksuhton commented on June 2, 2024

Stacktrace[38:end]

 [38] _pullback(ctx::Zygote.Context{false}, f::Zygote.Pullback{Tuple{typeof(jacobian), StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}, Matrix{Float32}}, Tuple{Zygote.ZBack{LuxZygoteExt.var"#13#17"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}, Matrix{Float32}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, Tuple{Matrix{Float32}}}}, Zygote.var"#2180#back#303"{Zygote.var"#back#302"{:ps, Zygote.Context{false}, StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}}}}}, args::Tuple{Diagonal{Float32, Vector{Float32}}})
    @ Zygote ~/.julia/packages/Zygote/jxHJc/src/compiler/interface2.jl:0
 [39] Pullback
    @ ./none:0 [inlined]
 [40] _pullback(ctx::Zygote.Context{false}, f::Zygote.Pullback{Tuple{var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}, Matrix{Float32}}, Tuple{Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2763#back#609"{Zygote.var"#603#607"{Vector{Float32}, Tuple{Tuple{Int64, Int64}}}}, Zygote.Pullback{Tuple{typeof(jacobian), StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}, Matrix{Float32}}, Tuple{Zygote.ZBack{LuxZygoteExt.var"#13#17"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}, Matrix{Float32}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, Tuple{Matrix{Float32}}}}, Zygote.var"#2180#back#303"{Zygote.var"#back#302"{:ps, Zygote.Context{false}, StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}}}}}, Zygote.var"#2180#back#303"{Zygote.var"#back#302"{:potential, Zygote.Context{false}, var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}, StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, Zygote.ZBack{ChainRules.var"#diag_pullback#2059"}, Zygote.Pullback{Tuple{typeof(only), Tuple{Matrix{Float32}}}, Tuple{Zygote.var"#2029#back#213"{Zygote.var"#back#211"{1, 1, Zygote.Context{false}, Matrix{Float32}}}}}}}, args::Matrix{Float32})
    @ Zygote ~/.julia/packages/Zygote/jxHJc/src/compiler/interface2.jl:0
 [41] #291
    @ ~/.julia/packages/Zygote/jxHJc/src/lib/lib.jl:206 [inlined]
 [42] _pullback(ctx::Zygote.Context{false}, f::Zygote.var"#291#292"{Tuple{Tuple{Nothing}}, Zygote.Pullback{Tuple{var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}, Matrix{Float32}}, Tuple{Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2763#back#609"{Zygote.var"#603#607"{Vector{Float32}, Tuple{Tuple{Int64, Int64}}}}, Zygote.Pullback{Tuple{typeof(jacobian), StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}, Matrix{Float32}}, Tuple{Zygote.ZBack{LuxZygoteExt.var"#13#17"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}, Matrix{Float32}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, Tuple{Matrix{Float32}}}}, Zygote.var"#2180#back#303"{Zygote.var"#back#302"{:ps, Zygote.Context{false}, StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}}}}}, Zygote.var"#2180#back#303"{Zygote.var"#back#302"{:potential, Zygote.Context{false}, var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}, StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, Zygote.ZBack{ChainRules.var"#diag_pullback#2059"}, Zygote.Pullback{Tuple{typeof(only), Tuple{Matrix{Float32}}}, Tuple{Zygote.var"#2029#back#213"{Zygote.var"#back#211"{1, 1, Zygote.Context{false}, Matrix{Float32}}}}}}}}, args::Matrix{Float32})
    @ Zygote ~/.julia/packages/Zygote/jxHJc/src/compiler/interface2.jl:0
 [43] #2169#back
    @ ~/.julia/packages/ZygoteRules/M4xmc/src/adjoint.jl:72 [inlined]
 [44] _pullback(ctx::Zygote.Context{false}, f::Zygote.var"#2169#back#293"{Zygote.var"#291#292"{Tuple{Tuple{Nothing}}, Zygote.Pullback{Tuple{var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}, Matrix{Float32}}, Tuple{Zygote.ZBack{ChainRules.var"#size_pullback#917"}, Zygote.var"#2763#back#609"{Zygote.var"#603#607"{Vector{Float32}, Tuple{Tuple{Int64, Int64}}}}, Zygote.Pullback{Tuple{typeof(jacobian), StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}, Matrix{Float32}}, Tuple{Zygote.ZBack{LuxZygoteExt.var"#13#17"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}, Matrix{Float32}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, Tuple{Matrix{Float32}}}}, Zygote.var"#2180#back#303"{Zygote.var"#back#302"{:ps, Zygote.Context{false}, StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}}}}}, Zygote.var"#2180#back#303"{Zygote.var"#back#302"{:potential, Zygote.Context{false}, var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}, StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, Zygote.ZBack{ChainRules.var"#diag_pullback#2059"}, Zygote.Pullback{Tuple{typeof(only), Tuple{Matrix{Float32}}}, Tuple{Zygote.var"#2029#back#213"{Zygote.var"#back#211"{1, 1, Zygote.Context{false}, Matrix{Float32}}}}}}}}}, args::Matrix{Float32})
    @ Zygote ~/.julia/packages/Zygote/jxHJc/src/compiler/interface2.jl:0
 [45] Pullback
    @ ./operators.jl:1045 [inlined]
 [46] _pullback(ctx::Zygote.Context{false}, f::Zygote.Pullback{Tuple{typeof(Base.call_composed), Tuple{var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, Tuple{Matrix{Float32}}, @Kwargs{}}, Any}, args::Matrix{Float32})
    @ Zygote ~/.julia/packages/Zygote/jxHJc/src/compiler/interface2.jl:0
 [47] Pullback
    @ ./operators.jl:1044 [inlined]
 [48] Pullback
    @ ./operators.jl:1041 [inlined]
 [49] _pullback(ctx::Zygote.Context{false}, f::Zygote.Pullback{Tuple{Base.var"##_#103", @Kwargs{}, ComposedFunction{typeof(Zygote._jvec), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, Matrix{Float32}}, Tuple{Zygote.Pullback{Tuple{typeof(Base.call_composed), Tuple{typeof(Zygote._jvec), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, Tuple{Matrix{Float32}}, @Kwargs{}}, Tuple{Zygote.Pullback{Tuple{typeof(Zygote._jvec), Matrix{Float32}}, Tuple{Zygote.Pullback{Tuple{typeof(vec), Matrix{Float32}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#749"}, Zygote.var"#2763#back#609"{Zygote.var"#603#607"{Matrix{Float32}, Tuple{Int64}}}}}}}, Zygote.Pullback{Tuple{typeof(Base.call_composed), Tuple{var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, Tuple{Matrix{Float32}}, @Kwargs{}}, Any}, Zygote.var"#2141#back#281"{Zygote.var"#277#280"}, Zygote.var"#2029#back#213"{Zygote.var"#back#211"{2, 1, Zygote.Context{false}, typeof(Zygote._jvec)}}}}, Zygote.Pullback{Tuple{typeof(Base.unwrap_composed), ComposedFunction{typeof(Zygote._jvec), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}}, Tuple{Zygote.var"#2169#back#293"{Zygote.var"#291#292"{Tuple{Tuple{Nothing}, Tuple{Nothing}}, Zygote.var"#2013#back#204"{typeof(identity)}}}, Zygote.Pullback{Tuple{typeof(Base.unwrap_composed), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, Tuple{Zygote.Pullback{Tuple{typeof(Base.maybeconstructor), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, Tuple{}}, Zygote.var"#2013#back#204"{typeof(identity)}}}, Zygote.var"#2180#back#303"{Zygote.var"#back#302"{:outer, Zygote.Context{false}, ComposedFunction{typeof(Zygote._jvec), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, typeof(Zygote._jvec)}}, Zygote.var"#2180#back#303"{Zygote.var"#back#302"{:inner, Zygote.Context{false}, ComposedFunction{typeof(Zygote._jvec), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}}, Zygote.Pullback{Tuple{typeof(Base.unwrap_composed), typeof(Zygote._jvec)}, Tuple{Zygote.Pullback{Tuple{typeof(Base.maybeconstructor), typeof(Zygote._jvec)}, Tuple{}}, Zygote.var"#2013#back#204"{typeof(identity)}}}}}}}, args::Vector{Float32})
    @ Zygote ~/.julia/packages/Zygote/jxHJc/src/compiler/interface2.jl:0
 [50] #291
    @ ~/.julia/packages/Zygote/jxHJc/src/lib/lib.jl:206 [inlined]
 [51] _pullback(ctx::Zygote.Context{false}, f::Zygote.var"#291#292"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.Pullback{Tuple{Base.var"##_#103", @Kwargs{}, ComposedFunction{typeof(Zygote._jvec), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, Matrix{Float32}}, Tuple{Zygote.Pullback{Tuple{typeof(Base.call_composed), Tuple{typeof(Zygote._jvec), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, Tuple{Matrix{Float32}}, @Kwargs{}}, Tuple{Zygote.Pullback{Tuple{typeof(Zygote._jvec), Matrix{Float32}}, Tuple{Zygote.Pullback{Tuple{typeof(vec), Matrix{Float32}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#749"}, Zygote.var"#2763#back#609"{Zygote.var"#603#607"{Matrix{Float32}, Tuple{Int64}}}}}}}, Zygote.Pullback{Tuple{typeof(Base.call_composed), Tuple{var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, Tuple{Matrix{Float32}}, @Kwargs{}}, Any}, Zygote.var"#2141#back#281"{Zygote.var"#277#280"}, Zygote.var"#2029#back#213"{Zygote.var"#back#211"{2, 1, Zygote.Context{false}, typeof(Zygote._jvec)}}}}, Zygote.Pullback{Tuple{typeof(Base.unwrap_composed), ComposedFunction{typeof(Zygote._jvec), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}}, Tuple{Zygote.var"#2169#back#293"{Zygote.var"#291#292"{Tuple{Tuple{Nothing}, Tuple{Nothing}}, Zygote.var"#2013#back#204"{typeof(identity)}}}, Zygote.Pullback{Tuple{typeof(Base.unwrap_composed), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, Tuple{Zygote.Pullback{Tuple{typeof(Base.maybeconstructor), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, Tuple{}}, Zygote.var"#2013#back#204"{typeof(identity)}}}, Zygote.var"#2180#back#303"{Zygote.var"#back#302"{:outer, Zygote.Context{false}, ComposedFunction{typeof(Zygote._jvec), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, typeof(Zygote._jvec)}}, Zygote.var"#2180#back#303"{Zygote.var"#back#302"{:inner, Zygote.Context{false}, ComposedFunction{typeof(Zygote._jvec), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}}, Zygote.Pullback{Tuple{typeof(Base.unwrap_composed), typeof(Zygote._jvec)}, Tuple{Zygote.Pullback{Tuple{typeof(Base.maybeconstructor), typeof(Zygote._jvec)}, Tuple{}}, Zygote.var"#2013#back#204"{typeof(identity)}}}}}}}}, args::Vector{Float32})
    @ Zygote ~/.julia/packages/Zygote/jxHJc/src/compiler/interface2.jl:0
 [52] #2169#back
    @ ~/.julia/packages/ZygoteRules/M4xmc/src/adjoint.jl:72 [inlined]
 [53] _pullback(ctx::Zygote.Context{false}, f::Zygote.var"#2169#back#293"{Zygote.var"#291#292"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.Pullback{Tuple{Base.var"##_#103", @Kwargs{}, ComposedFunction{typeof(Zygote._jvec), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, Matrix{Float32}}, Tuple{Zygote.Pullback{Tuple{typeof(Base.call_composed), Tuple{typeof(Zygote._jvec), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, Tuple{Matrix{Float32}}, @Kwargs{}}, Tuple{Zygote.Pullback{Tuple{typeof(Zygote._jvec), Matrix{Float32}}, Tuple{Zygote.Pullback{Tuple{typeof(vec), Matrix{Float32}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#749"}, Zygote.var"#2763#back#609"{Zygote.var"#603#607"{Matrix{Float32}, Tuple{Int64}}}}}}}, Zygote.Pullback{Tuple{typeof(Base.call_composed), Tuple{var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, Tuple{Matrix{Float32}}, @Kwargs{}}, Any}, Zygote.var"#2141#back#281"{Zygote.var"#277#280"}, Zygote.var"#2029#back#213"{Zygote.var"#back#211"{2, 1, Zygote.Context{false}, typeof(Zygote._jvec)}}}}, Zygote.Pullback{Tuple{typeof(Base.unwrap_composed), ComposedFunction{typeof(Zygote._jvec), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}}, Tuple{Zygote.var"#2169#back#293"{Zygote.var"#291#292"{Tuple{Tuple{Nothing}, Tuple{Nothing}}, Zygote.var"#2013#back#204"{typeof(identity)}}}, Zygote.Pullback{Tuple{typeof(Base.unwrap_composed), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, Tuple{Zygote.Pullback{Tuple{typeof(Base.maybeconstructor), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, Tuple{}}, Zygote.var"#2013#back#204"{typeof(identity)}}}, Zygote.var"#2180#back#303"{Zygote.var"#back#302"{:outer, Zygote.Context{false}, ComposedFunction{typeof(Zygote._jvec), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, typeof(Zygote._jvec)}}, Zygote.var"#2180#back#303"{Zygote.var"#back#302"{:inner, Zygote.Context{false}, ComposedFunction{typeof(Zygote._jvec), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}}, Zygote.Pullback{Tuple{typeof(Base.unwrap_composed), typeof(Zygote._jvec)}, Tuple{Zygote.Pullback{Tuple{typeof(Base.maybeconstructor), typeof(Zygote._jvec)}, Tuple{}}, Zygote.var"#2013#back#204"{typeof(identity)}}}}}}}}}, args::Vector{Float32})
    @ Zygote ~/.julia/packages/Zygote/jxHJc/src/compiler/interface2.jl:0
 [54] Pullback
    @ ./operators.jl:1041 [inlined]
 [55] _pullback(ctx::Zygote.Context{false}, f::Zygote.Pullback{Tuple{ComposedFunction{typeof(Zygote._jvec), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, Matrix{Float32}}, Tuple{Zygote.var"#2366#back#419"{Zygote.var"#pairs_namedtuple_pullback#418"{(), @NamedTuple{}}}, Zygote.Pullback{Tuple{Type{NamedTuple}}, Tuple{}}, Zygote.var"#2169#back#293"{Zygote.var"#291#292"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.Pullback{Tuple{Base.var"##_#103", @Kwargs{}, ComposedFunction{typeof(Zygote._jvec), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, Matrix{Float32}}, Tuple{Zygote.Pullback{Tuple{typeof(Base.call_composed), Tuple{typeof(Zygote._jvec), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, Tuple{Matrix{Float32}}, @Kwargs{}}, Tuple{Zygote.Pullback{Tuple{typeof(Zygote._jvec), Matrix{Float32}}, Tuple{Zygote.Pullback{Tuple{typeof(vec), Matrix{Float32}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#749"}, Zygote.var"#2763#back#609"{Zygote.var"#603#607"{Matrix{Float32}, Tuple{Int64}}}}}}}, Zygote.Pullback{Tuple{typeof(Base.call_composed), Tuple{var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, Tuple{Matrix{Float32}}, @Kwargs{}}, Any}, Zygote.var"#2141#back#281"{Zygote.var"#277#280"}, Zygote.var"#2029#back#213"{Zygote.var"#back#211"{2, 1, Zygote.Context{false}, typeof(Zygote._jvec)}}}}, Zygote.Pullback{Tuple{typeof(Base.unwrap_composed), ComposedFunction{typeof(Zygote._jvec), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}}, Tuple{Zygote.var"#2169#back#293"{Zygote.var"#291#292"{Tuple{Tuple{Nothing}, Tuple{Nothing}}, Zygote.var"#2013#back#204"{typeof(identity)}}}, Zygote.Pullback{Tuple{typeof(Base.unwrap_composed), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, Tuple{Zygote.Pullback{Tuple{typeof(Base.maybeconstructor), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, Tuple{}}, Zygote.var"#2013#back#204"{typeof(identity)}}}, Zygote.var"#2180#back#303"{Zygote.var"#back#302"{:outer, Zygote.Context{false}, ComposedFunction{typeof(Zygote._jvec), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, typeof(Zygote._jvec)}}, Zygote.var"#2180#back#303"{Zygote.var"#back#302"{:inner, Zygote.Context{false}, ComposedFunction{typeof(Zygote._jvec), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}}, Zygote.Pullback{Tuple{typeof(Base.unwrap_composed), typeof(Zygote._jvec)}, Tuple{Zygote.Pullback{Tuple{typeof(Base.maybeconstructor), typeof(Zygote._jvec)}, Tuple{}}, Zygote.var"#2013#back#204"{typeof(identity)}}}}}}}}}, Zygote.var"#2013#back#204"{typeof(identity)}}}, args::Vector{Float32})
    @ Zygote ~/.julia/packages/Zygote/jxHJc/src/compiler/interface2.jl:0
 [56] #75
    @ ~/.julia/packages/Zygote/jxHJc/src/compiler/interface.jl:91 [inlined]
 [57] _pullback(ctx::Zygote.Context{false}, f::Zygote.var"#75#76"{Zygote.Pullback{Tuple{ComposedFunction{typeof(Zygote._jvec), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, Matrix{Float32}}, Tuple{Zygote.var"#2366#back#419"{Zygote.var"#pairs_namedtuple_pullback#418"{(), @NamedTuple{}}}, Zygote.Pullback{Tuple{Type{NamedTuple}}, Tuple{}}, Zygote.var"#2169#back#293"{Zygote.var"#291#292"{Tuple{Tuple{Nothing, Nothing}, Tuple{Nothing}}, Zygote.Pullback{Tuple{Base.var"##_#103", @Kwargs{}, ComposedFunction{typeof(Zygote._jvec), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, Matrix{Float32}}, Tuple{Zygote.Pullback{Tuple{typeof(Base.call_composed), Tuple{typeof(Zygote._jvec), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, Tuple{Matrix{Float32}}, @Kwargs{}}, Tuple{Zygote.Pullback{Tuple{typeof(Zygote._jvec), Matrix{Float32}}, Tuple{Zygote.Pullback{Tuple{typeof(vec), Matrix{Float32}}, Tuple{Zygote.ZBack{ChainRules.var"#length_pullback#749"}, Zygote.var"#2763#back#609"{Zygote.var"#603#607"{Matrix{Float32}, Tuple{Int64}}}}}}}, Zygote.Pullback{Tuple{typeof(Base.call_composed), Tuple{var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, Tuple{Matrix{Float32}}, @Kwargs{}}, Any}, Zygote.var"#2141#back#281"{Zygote.var"#277#280"}, Zygote.var"#2029#back#213"{Zygote.var"#back#211"{2, 1, Zygote.Context{false}, typeof(Zygote._jvec)}}}}, Zygote.Pullback{Tuple{typeof(Base.unwrap_composed), ComposedFunction{typeof(Zygote._jvec), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}}, Tuple{Zygote.var"#2169#back#293"{Zygote.var"#291#292"{Tuple{Tuple{Nothing}, Tuple{Nothing}}, Zygote.var"#2013#back#204"{typeof(identity)}}}, Zygote.Pullback{Tuple{typeof(Base.unwrap_composed), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, Tuple{Zygote.Pullback{Tuple{typeof(Base.maybeconstructor), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, Tuple{}}, Zygote.var"#2013#back#204"{typeof(identity)}}}, Zygote.var"#2180#back#303"{Zygote.var"#back#302"{:outer, Zygote.Context{false}, ComposedFunction{typeof(Zygote._jvec), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, typeof(Zygote._jvec)}}, Zygote.var"#2180#back#303"{Zygote.var"#back#302"{:inner, Zygote.Context{false}, ComposedFunction{typeof(Zygote._jvec), var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}, var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}}}, Zygote.Pullback{Tuple{typeof(Base.unwrap_composed), typeof(Zygote._jvec)}, Tuple{Zygote.Pullback{Tuple{typeof(Base.maybeconstructor), typeof(Zygote._jvec)}, Tuple{}}, Zygote.var"#2013#back#204"{typeof(identity)}}}}}}}}}, Zygote.var"#2013#back#204"{typeof(identity)}}}}, args::Vector{Float32})
    @ Zygote ~/.julia/packages/Zygote/jxHJc/src/compiler/interface2.jl:0
 [58] withjacobian
    @ ~/.julia/packages/Zygote/jxHJc/src/lib/grad.jl:150 [inlined]
 [59] _pullback(::Zygote.Context{false}, ::typeof(withjacobian), ::var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}, ::Matrix{Float32})
    @ Zygote ~/.julia/packages/Zygote/jxHJc/src/compiler/interface2.jl:0
 [60] _apply(::Function, ::Vararg{Any})
    @ Core ./boot.jl:838
 [61] adjoint
    @ ~/.julia/packages/Zygote/jxHJc/src/lib/lib.jl:203 [inlined]
 [62] _pullback
    @ ~/.julia/packages/ZygoteRules/M4xmc/src/adjoint.jl:67 [inlined]
 [63] jacobian
    @ ~/.julia/packages/Zygote/jxHJc/src/lib/grad.jl:128 [inlined]
 [64] _pullback(::Zygote.Context{false}, ::typeof(jacobian), ::var"#jac_pot#45"{StatefulLuxLayer{true, Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}, @NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}, @NamedTuple{}}}, ::Matrix{Float32})
    @ Zygote ~/.julia/packages/Zygote/jxHJc/src/compiler/interface2.jl:0
 [65] #43
    @ ./none:0 [inlined]
 [66] _pullback(::Zygote.Context{false}, ::var"#43#44", ::@NamedTuple{potential::Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}}, ::Matrix{Float32}, ::@NamedTuple{potential::@NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}}, ::@NamedTuple{potential::@NamedTuple{}})
    @ Zygote ~/.julia/packages/Zygote/jxHJc/src/compiler/interface2.jl:0
 [67] CompactLuxLayer
    @ ~/.julia/packages/Lux/ANzxX/src/helpers/compact.jl:422 [inlined]
 [68] _pullback(::Zygote.Context{false}, ::CompactLuxLayer{nothing, var"#43#44", @NamedTuple{potential::Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}}, @NamedTuple{potential::Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}}, Lux.ValueStorage{@NamedTuple{}, @NamedTuple{}}, Tuple{Tuple{}, Tuple{}}}, ::Matrix{Float32}, ::@NamedTuple{potential::@NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}}, ::@NamedTuple{potential::@NamedTuple{}})
    @ Zygote ~/.julia/packages/Zygote/jxHJc/src/compiler/interface2.jl:0
 [69] _apply(::Function, ::Vararg{Any})
    @ Core ./boot.jl:838
 [70] adjoint
    @ ~/.julia/packages/Zygote/jxHJc/src/lib/lib.jl:203 [inlined]
 [71] _pullback
    @ ~/.julia/packages/ZygoteRules/M4xmc/src/adjoint.jl:67 [inlined]
 [72] call_composed
    @ ./operators.jl:1045 [inlined]
 [73] call_composed (repeats 2 times)
    @ ./operators.jl:1044 [inlined]
 [74] #_#103
    @ ./operators.jl:1041 [inlined]
 [75] _pullback(::Zygote.Context{false}, ::Base.var"##_#103", ::@Kwargs{}, ::ComposedFunction{ComposedFunction{Base.Fix1{typeof(sum), typeof(abs2)}, typeof(first)}, CompactLuxLayer{nothing, var"#43#44", @NamedTuple{potential::Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}}, @NamedTuple{potential::Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}}, Lux.ValueStorage{@NamedTuple{}, @NamedTuple{}}, Tuple{Tuple{}, Tuple{}}}}, ::Matrix{Float32}, ::@NamedTuple{potential::@NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}}, ::@NamedTuple{potential::@NamedTuple{}})
    @ Zygote ~/.julia/packages/Zygote/jxHJc/src/compiler/interface2.jl:0
 [76] _apply(::Function, ::Vararg{Any})
    @ Core ./boot.jl:838
 [77] adjoint
    @ ~/.julia/packages/Zygote/jxHJc/src/lib/lib.jl:203 [inlined]
 [78] _pullback
    @ ~/.julia/packages/ZygoteRules/M4xmc/src/adjoint.jl:67 [inlined]
 [79] ComposedFunction
    @ ./operators.jl:1041 [inlined]
 [80] _pullback(::Zygote.Context{false}, ::ComposedFunction{ComposedFunction{Base.Fix1{typeof(sum), typeof(abs2)}, typeof(first)}, CompactLuxLayer{nothing, var"#43#44", @NamedTuple{potential::Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}}, @NamedTuple{potential::Dense{true, typeof(gelu), typeof(glorot_uniform), typeof(zeros32)}}, Lux.ValueStorage{@NamedTuple{}, @NamedTuple{}}, Tuple{Tuple{}, Tuple{}}}}, ::Matrix{Float32}, ::@NamedTuple{potential::@NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}}, ::@NamedTuple{potential::@NamedTuple{}})
    @ Zygote ~/.julia/packages/Zygote/jxHJc/src/compiler/interface2.jl:0
 [81] pullback(::Function, ::Zygote.Context{false}, ::Matrix{Float32}, ::Vararg{Any})
    @ Zygote ~/.julia/packages/Zygote/jxHJc/src/compiler/interface.jl:90
 [82] pullback(::Function, ::Matrix{Float32}, ::@NamedTuple{potential::@NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}}, ::Vararg{Any})
    @ Zygote ~/.julia/packages/Zygote/jxHJc/src/compiler/interface.jl:88
 [83] gradient(::Function, ::Matrix{Float32}, ::@NamedTuple{potential::@NamedTuple{weight::Matrix{Float32}, bias::Matrix{Float32}}}, ::Vararg{Any})
    @ Zygote ~/.julia/packages/Zygote/jxHJc/src/compiler/interface.jl:147

from lux.jl.

avik-pal avatar avik-pal commented on June 2, 2024

Actually @tansongchen might know how to do these kind of 3rd order differentiation efficiently.

from lux.jl.

aksuhton avatar aksuhton commented on June 2, 2024

Thank you @avik-pal, sorry for the ๐Ÿ˜“.

from lux.jl.

aksuhton avatar aksuhton commented on June 2, 2024

Hey @tansongchen, thanks for joining us! Absolutely. I'm trying to learn a mapping between two time-sequences

$\varepsilon_t \rightarrow \sigma_t$.

Now, thermodynamic arguments suggest that

$\sigma_t = \frac{\partial F_t}{\partial \varepsilon_t}$

for some function $F_t$ that depends on the history of $\varepsilon$ up to time $t$. The recent nested AD updates let me seek $F_t$ here as a temporal convolutional neural network that takes as input the time-sequence $\varepsilon_t$. Yay! However, this still misses some physical structure.

Ideally, I'd seek $F_t$ via

$\partial_t \sigma_t = \frac{\partial^2 F_t}{\partial \varepsilon_t^2}\partial_t \varepsilon_t$

as this makes it manifest that the stress $\sigma$ remains fixed whenever the strain $\varepsilon$ remains fixed.

There's (at least) two difficulties here:

  1. I want to take two derivatives of the network output with respect to its inputs before taking another derivative for the purpose of optimizing the network parameters.
  2. The neural network output is not a number, rather I'd like to output a sequence

$F = ({F_1[\varepsilon], F_2[\varepsilon], ..., F_T[\varepsilon]})$

and then collect the derivatives

$\partial^2F = ({\partial^2_{\varepsilon_1}F_1[\varepsilon], \partial^2_{\varepsilon_2}F_2[\varepsilon], ..., \partial^2_{\varepsilon_T}F_T[\varepsilon]})$.

Please note my shorthand for "functional" dependence

$F_i[\varepsilon] = F_i(\varepsilon_T, \varepsilon_{T-1}, ..., \varepsilon_1)$.

Similar structure comes up in Equation (22a) of the following paper, though they use a set of internal variables $\zeta_t$ to capture dynamics that I'd rather implicitly handle through a $t$ subscript on $F$:

https://arxiv.org/abs/2005.12183

from lux.jl.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.