Further granularity for JLD about fileio.jl HOT 11 CLOSED

I think you're looking for a custom serializer: https://github.com/JuliaIO/JLD.jl/blob/master/doc/jld.md#custom-serialization. You can define JLD.readas and JLD.writeas methods for your types in your packages. (That makes JLD a dependency, of course.)

Since I don't think this is a FileIO issue, I'm closing this (but if you disagree, I can reopen).

from fileio.jl.

Thanks,

This is more/less what I was looking for. I had missed that part in the docs... There still might be a small relation to FileIO: In the documented example, an obj2 = load(filename) will still result in an object of type MyVectors5Serializer, so the FileIO magic goes as far as running JLD.jldopen() I suppose, but not the associated JLD.readas(). At least, this is what happened for my type. Is this right?

Would there be a way of cascading the FileIO magic to readas?

from fileio.jl.

There still might be a small relation to FileIO: In the documented example, an obj2 = load(filename) will still result in an object of type MyVectors5Serializer

It shouldn't, if the package that defines the readas method has been loaded. If you can observe this behavior in a reproducible example, then it's a bug that should be filed to JLD.jl.

from fileio.jl.

Relevant to the previous point: see also the part in the docs about addrequire.

from fileio.jl.

Yes, thanks, I did have the addrequire in the jldopen(filename, "w") do ... end in my test. But perhaps the test didn't result in the right result because of module-reloading issues. I'll try to do a clean test now

from fileio.jl.

Ah---I found the reason why it just does not exactly what I want, and why I was looking for a FileIO solution.

In the documented example, things work if I say something like

obj4 = FileIO.load(filename, "somedata")

but what I was looking for was a way to be able to say

obj3 = FileIO.load(filename)

which would need additional magic that files that require MyModule and have as a root dataset "somedata" (a better name would be MyVector5 in such a case).

The parallel with FileIO would be that most standard formats have "unnamed" objects as well, e.g., :WAV just has the audio. I am looking for a way to save objects from my types in a JLD container, namelessly.

from fileio.jl.

You'd probably need to define a new file extension, and most likely give it a special magic #.

from fileio.jl.

Is there a way to add additional magic after string(magic_base, f.version) in JLD, so that the file will be both :JLD and the special :MyVector5 file type? I assume I can always re-write the header by hand.

This is assuming that FileIO will go for the longer magic match.

from fileio.jl.

I don't think that FileIO can (in the central repository) reasonably support "custom JLD content" as separate file types; after all, there are an infinite number of possible combinations of variable names, types, etc., and so you might define 3 different ones, I might define 5 different ones, etc. Only if there is some standard in widespread use would we consider such a thing.

However, of course you can add this locally. But perhaps an easier option would be to write a function called myload which checks to see if it's a JLD file and, if so, asks whether the content is one of your "standard" formats. If not, myload can call load.

from fileio.jl.

Yes, that makes sense.

It is funny, though, that I tend to save objects (GaussianMixtures.GMM, IVectors.IExtractor) in a JLD under their type name. Another interface that appears to make sense is load(file, ::Type), because in the code that calls load you usually know what the resulting type should be. This makes it a little more explicit.

from fileio.jl.

Right, but other people might save 20 different objects of the same type to a single file, using a different pathname for each.

from fileio.jl.