Backwards compatibility? (new structure definitions but old json serialization) about json HOT 17 OPEN

So, the simplest thing to do is use optionals in the new version of the structure. That works if that new version simply extends the previous one. But if some types are changed or some fields are removed, that won't work.

The next option is using a variant. We support any type with the interface similar to that of std::variant (I recommend using boost::variant2::variant). E.g. you parse into variant<mystruct_v1_1, mystruct_v1_0>. And then you check index, and if it corresponds to the old version, you construct the new struct from the old one.

Finally, if the struct is not the root of the target, then you can write a tag_invoke overload. It can also use the variant trick above in its implementation.

Update:
Although, do remember that if you invoke value_to< variant<mystruct_v1_1, mystruct_v1_0> > from inside tag_invoke for mystruct_v1_1, you'll get an infinite recursion. This can be avoided with using a third type (e.g. mystruct) or using contextual conversions.

from json.

Thanks for the suggestions. Currently using a tag_invoke overload as a user-provided conversion, as in example here

But this seems to not correctly handle the std::variant. Is it anything specific missing in this example to handle the variant? I can see std::variant working correctly without the user-provided conversion linked above. I wonder if there is something with the context that needs to passed?

from json.

That example doesn't deal with variant, so I'm not sure what you mean. Do you have a variant as a member of a described struct?

from json.

Yes I have a described struct containing a variant. What is needed to handle that? Is there an example you can point me to?

from json.

It should work: https://godbolt.org/z/1YzKs4s4e

from json.

It does not work for me if I use std::variant and a tag_invoke overload.

https://godbolt.org/z/Gc7Tq9PzK

Passing the contextual tag there causes it to not hold the alternative B when doing value_to on the json string.

from json.

The problem with your approach is that your tag_invoke download never fails. Inside value_to<C> you invoke value_to< variant<A, B> >, it calls value_to<A>, it tries to convert to A, fails, the failure is ignored and a default-constructed A is returned.

You should either throw an exception (since you're using a throwing overload) or use a non-throwing overload and return a result with set error_code.

Also, I've noticed that you are using contexts incorrectly. When you are using a two context overload of tag_invoke your context is supposed to be used in the function which is defined, but you should pass the full context to inner calls of value_to. That is, unless you explicitly don't want that for some reason.

BTW, why do you need that custom overload? Currently it doesn't do anything that the library isn't doing already. And the library actually does more, e.g. it has special handling for optionals.

from json.

The custom overload is there to handle backward compatibility in a generic way. As new parameters are added I want to decode them as default if they are missing in the json serialization. Consider the following use case

struct C {
    std::variant<A, B> variant;
    double new_param{14.1}; // new parameter added
};
BOOST_DESCRIBE_STRUCT(C, (), (variant, new_param))

// json string generated by old describe, need backward compatibility
std::string_view json = R"foo({"variant":{"d":3.14E0}})foo";

Is there a way to distinguish between boost json trying the different variants versus the case where a new field has been added? Can I look at boost::json::error to tell what happened?

from json.

I see. You are writing a templated overload seemingly intended for all described classes, but in fact you only need it for C. So, make it just for C. That way you don't need a context. Also, if it's an overload for a specific type, you can easily implement it using e.g.

C c;
c.a = value_to<A>( obj.at("a") );
c.b = value_to<B>( obj.at("b") );

But if you don't want to do that, you can take the implementation for described classes from the library and slightly change it. I actuall did it here: https://godbolt.org/z/frbhPxqe5

The main interesting point is usage of descriptor_by_pointer from Boost.Describe. With that component I define a special trait that judges wether a particular member of C is required (default) or not.

template<>
struct is_required<
    boost::describe::descriptor_by_pointer<C_Ds, &C::new_param> >
    : std::false_type
{};

And then inside the overload I use it like this:

auto const found = obj->find(D.name);
if( found == obj->end() )
{
    if constexpr( !is_required< decltype(D) >::value ) // ignore missing new fields
        return;
    throw std::runtime_error("missing field");
}

from json.

I would probably want to make is_required default to false since I have many newly added fields. But since an std::variant may be a subset of another variant it is not clear to me how to make boost continue trying.

struct A {
    int i;
};
BOOST_DESCRIBE_STRUCT(A, (), (i))

struct B {
     int i;
    double d;
};
BOOST_DESCRIBE_STRUCT(B, (), (i,d))

struct C {
    std::variant<A, B> variant;
    double new_param = 14.1; // new parameter added
};

from json.

Oh, so you want to support these added fields in other structs too? Then, you do need to make the overload a template.

from json.

Backwards support means I need to support added and removed fields in data structures. I can see boost 1.83 does support std::optional to handle added fields (using is_optional). That is good, something I could try to use. But next problem would then be deprecated fields. Currently the boost implementation returns error::size_mismatch is a field has been deprecated. Any suggestion on how to handle deprecated fields?

from json.

Ok, so you have several structs with a few members which were added, and a few which were removed. In addition, those structs are nested inside each other, right?

To be honest, in this case I'd write custom tag_invoke overloads for each struct. The complexity of trying to handle it in a generic way is probably not worth it.

BTW, this issue accidentally helped me to catch a bug: #999.

from json.