Make Constant-folding a generic greedy Optimizer about theano HOT 5 CLOSED

On Fri, Oct 28, 2011 at 11:53 AM, James Bergstra
[email protected]
wrote:

It's nice to have constant folding as a local optimizer so that it can be done during stabilization and specialization.

For better performance - and to permit lowering the max_use_ratio back to a sane level, it would also be good to do Constant folding "passes" similar to what the MergeOptimizer does.  Since constant folding is a kind of bottom-up transformation, a single pass could fold entire subgraphs all at once. This would be faster, because it would avoid running all of the equilibrium optimizers between every constant-folding substitution!

I would suggest making a single pass of the ConstantFoldingOptimizer, after stabilization, but before specialization. We don't do any constant folding before specialization right?

We do it... It is not trivial to move them after stabilize as this
disable some optimization done in the Canonicalize...

from theano.

On Fri, Oct 28, 2011 at 11:57 AM, nouiz
[email protected]
wrote:

I would suggest making a single pass of the ConstantFoldingOptimizer, after stabilization, but before specialization. We don't do any constant folding before specialization right?

We do it... It is not trivial to move them after stabilize as this
disable some optimization done in the Canonicalize...

Ok, then maybe a preliminary greedy constant-folding pass could be
done before canonicalize? Maybe we can even recognize safe vs. unsafe
folding operations before canonicalize?

from theano.

I have no idea how to recognize safe vs unsafe folding operation. The real fix is to "port" the Canonicalizer optimizer to after the constant folding phase. But we must change the Canonicalizer to change the graph only when it do some optimization at the same time...

What about just rerunning the full Canonicalizer instead? In summary what about this:

1. current canonicalization phase without constant folding
2. current stabilize phase without constant folding
3. constant folding phase
4. new Canonizer class phase or the current canonicalization phase WITH constant folding
5. we continue....

There is also the blas opt some where in there.

from theano.

We now have topo_constant_folding running at a couple of places, which is simply defined as in2out(constant_folding).

from theano.

from theano.