• README.md
• DESCRIPTION
• ?mistnet2
• top of github page
• mistnet
• examples with mistnet()

For example, starttests is currently ignored

Right now, some functions take x (maybe just dldx and density?) and others take y. This should be as consistent as possible.

Basically just sample from z's prior and then feedforward, I think.

Then, once MCMC sampling for z exists (#21), we could condition on partial observations too.

samples from prior? Something narrower?

So the package can fit models like the original mistnet package or fine-tune model estimates that were based on point estimates from BFGS

To keep this from being machine-dependent, I could say that the total computation for feedforward or for backprop shouldn't take more than x% longer than the underlying matrix multiplication.

This would help ensure that I'm aware of any situations where I'm adding a lot of overhead to the computation.

and replace toy.R

Probably with pseudolikelihood approximation?

• adjustable priors
• adjustable error_distributions

• priors should probably be weight_priors or something, since priors can go elsewhere
• error_distributions should probably just be distributions, since they can also refer to priors
• networks should probably be mistnets, in case I want other kinds of networks (e.g. with other kinds of latent variables)
• par_skeleton might be better as adjustable_parameters or just parameters, given the way it's used.
• make_gamlss_distribution should probably just be make_distribution

...

Probably

e.g. these repeated observations should have the same Z values in this column

I shouldn't need to do log_density(net, par = unlist(net$par_list)).

log_density(net) and log_density(net, par = net$par_list) are unambiguous and create less room for error.

It would be great to have semantics for adding log distributions to one another (i.e. setting two or more independent objective functions for the same parameters).

Some families have a sigma or tau term for scale or shape. Others have special terms like bd for binomial denominators. Sometimes, these will be adjustable parameters and will need to be handed to optimx. Other times, they won't be. How to structure this so it always makes sense?

Right now, passing a value to REPORT or kkt overwrites maximize and starttests.

Also, REPORT seems to get ignored (or maybe isn't even passed where it needs to go)

net = mistnet_fit(net, itnmax = 7000, 
                  control = list(maximize = TRUE, starttests = FALSE, 
                                 kkt = FALSE, REPORT = 1))

This sort of detail doesn't need to be taken care of for this milestone, so let's close #10 and move the long-term wish list here.

e.g. "these columns are Gaussian, these are Poisson, these are Bernoulli"

Sampling interferes with second-order batch methods like L-BFGS-B, but dropout is probably important.

Sida Wang and Christoper Manning's "Fast Dropout Training" might be optimal. In the mean time, the variant of an L2 penalty they suggest in between Eqn 9 and Eqn 10 is easy: instead of just having the penalty for weight ij be proportional to its square, multiply that square by c, which is basically the variance of the input variable multiplied by that weight.

Also, batch normalization might solve some of the same problems?

e.g. each column of the error distribution or each row of a prior distribution should be able to have its own sigma, and those sigmas should be easily adjustable.

Penalties could arise in a number of different places:

• Penalties on weights (and maybe even biases)
• Penalties on the latent variables
• Penalties on the outputs

How should I structure and manage them all?

like in the original mistnet package. Less chance of things getting out of sync.

Simpler than mcmc; the sampling for the predict function (#20) will get me 90% of the way there.

• Zero or one column in x
• Zero or one column in z