Pause and restart about smm.jl HOT 12 CLOSED

no, but that would be a good feature. you can save and load an AlgoBGP object from disk already. so the pieces are all there, just need to connect them somehow. sounds like a pull request to me? :-)

from smm.jl.

Can we convert this "issue" into a milestone?
For an unknown reason, I can't just follow this tutorial: https://help.github.com/articles/creating-and-editing-milestones-for-issues-and-pull-requests
At step 4, the option "New Milestone" is not there.

from smm.jl.

from smm.jl.

What I mentioned is not an issue per se. It's more of a new feature for next versions. Maybe milestones are a convenient device for listing such extensions? (https://www.youtube.com/watch?v=ukYSRu4k0gs)

from smm.jl.

i see. that's fine as an issue I think. :-)

from smm.jl.

Since JLD2 supports saving functions, it seems to be the way to go for me.
If you agree, I can prepare a PR.
I also see that in order to save every x iterations, the user needs to include "save_frequency" and "filename" when setting up the opts dictionary. Is that right? If yes, it should probably appear in the README or docs.
Cheers,
Julien

from smm.jl.

yes, that's a good idea. JLD2 can save any julia object. it can save the entire MAlgoBGP object for example at it's current state. I am not sure how efficient that is though, i.e. how much time each save takes. the idea with the save_frequency was to just add the last save_frequency rows of data to a dataframe (in HDF5 format, but still). I guess we should just try with saving the entire thing every save_frequency steps as a .jld2 file and see how much time that takes. and yes, that should be better documented - like everything else really (given there is no docs whatsoever, that is hardly surprising ;-) ). thanks for this!

from smm.jl.

It appears to be more complicated than initially planned. Simply saving the entire MAlgoBGP does not work, as JLD2 does not seem to fully comprehend the structure of this object. When loading the same object, some information is lost regarding the chains.

I have made some experiments and it seems that it is necessary to save each chain individually. But then it would imply a large number of files when working with several chains.

Any thought on this matter?

from smm.jl.

from smm.jl.

I am weighing the pros and the cons of using a dataframe for chains.
On the one hand, it would avoid the issue of having several files when working with several chains, which is great. But on the other hand, the chains attached to a MAlgoBGP would have to be re-created using the dataframe. So that means creating a new function to do that. I don't know how much time it would take.

Regarding what to be saved, MAlgoBGP contains 6 fields:

• m
• opts
• i
• chains
• anim
• dist_func

My gut feeling is that only chains deserved a special treatment (and maybe anim, but I am not even sure that animation are still currently supported). So it would be perfectly fine to store the "easy ones" in a single JLD2 file:

@JLD2.save "filename" MAlgoBGP.m MAlgoBGP.opts MAlgoBGP.i MAlgoBGP.dist_func

Then something would have to be done for MAlgoBGP.chains. For the moment I have a patch that creates one file for each chain. Yet, if you think that using a dataframe is the way to go, I will have a look and see to what extent this is feasible.

from smm.jl.

I have news on this side. The issue was not related to JLD2, but it was a subtle one about the type BGPChain:

type BGPChain <: AbstractChain
    evals     :: Array{Eval}
    best_id   :: Vector{Int}   # index of best eval.value so far
    best_val  :: Vector{Float64}   # best eval.value so far
    curr_val  :: Vector{Float64}   # current value
    probs_acc :: Vector{Float64}    # vector of probabilities with which to accept
    #mprob     :: MProb
    id        :: Int64
    iter      :: Int64
    accepted  :: Array{Bool}
    accept_rate :: Float64
    acc_tuner :: Float64
    exchanged :: Array{Int}
    m         :: MProb
    sigma     :: PDiagMat{Float64}
    sigma_update_steps :: Int64   # update sampling vars every sigma_update_steps iterations
    sigma_adjust_by :: Float64   # adjust sampling vars by sigma_adjust_by percent up or down
    smpl_iters :: Int64   # max number of trials to get a new parameter from MvNormal that lies within support
    maxdist  :: Float64 

The mprob that I have commented out should not be there. There is already m. Given that mprob was never assigned any value, trying to load a BGPChain with JLD2 would return an error ("undefined reference").

Without this extra mprob, saving and loading the entire MAlgoBGP does work like a charm.

from smm.jl.

This is no longer an issue, since the PR has been merged. Thanks.

from smm.jl.