Alternative to Rademacher distribution about adahessian HOT 2 CLOSED

Hi Nestor,

Yes you can use Gaussian random numbers as well. However, Rademacher random numbers have better convergence. For details please see the following paper:

Ubaru S, Chen J, Saad Y. Fast Estimation of tr(f(A)) via Stochastic Lanczos Quadrature. SIAM Journal on Matrix Analysis and Applications. 2017;38(4):1075-99.

Best,
-Amir

from adahessian.

As I understand it, Theorem 4.3 in the paper gives a better bound for the Rademacher distribution (compared to the Gaussian distribution) but it is a lower bound on the number of samples required to be precise and not a proof that it will perform better in general (as those are not tight bounds). The following paper shows (using both bounds and experiments) that the best estimator varies as a function of the properties of the matrix studied:

Improved bounds on sample size for implicit matrix trace estimators

Hence my question, an empirical study might reveal a better estimator for the type of matrix produced when looking at the hessian of neural networks (however I do not expect another estimator to yield huge benefits).

from adahessian.