Deserializing Vec fails if there's something in between about serde-xml-rs HOT 31 CLOSED

This is really hard to do and if implemented, completely messes up how serde operates and requires heap allocations.

Additionally to considering solutions on the serde-end, please consider talking to the provider of your data whether they can produce a format that doesn't require a lot of state during deserialization.

It actually means your approach is unusable for XML, because OP gave absolutely valid XML and any valid parser should parse it. If this design doesn't work, parser should be redesigned. It's not some "extra feature", it's basic functionality.

from serde-xml-rs.

fyi, I am happy that this wasnt solved anyhow - thanks to this error I realised that I have dependency on the order of xml children, and if this would be /working/ I would spent hours trying to find the rootcause! so the enum solution here is actually what I really wanted, thanks!

from serde-xml-rs.

This is really hard to do and if implemented, completely messes up how serde operates and requires heap allocations.

Additionally to considering solutions on the serde-end, please consider talking to the provider of your data whether they can produce a format that doesn't require a lot of state during deserialization.

from serde-xml-rs.

You can use enums for this:

#[macro_use] extern crate serde_derive;
extern crate serde_xml_rs;

#[derive(Debug, Deserialize)]
struct Root {
    #[serde(rename = "$value")]
    items: Vec<Elems>,
}

#[derive(Debug, Deserialize)]
#[serde(rename_all = "lowercase")]
enum Elems {
    Foo(String),
    Bar(String)
}


fn main() {

    let input: &[u8] = b"<root>
        <foo>abc</foo>
        <foo>def</foo>

        <bar>lmn</bar>
        <bar>opq</bar>

        <foo>ghi</foo>
        </root>";

    let res: Root = match serde_xml_rs::deserialize(input) {
        Ok(r) => r,
        Err(e) => panic!("{:?}", e),
    };

    println!("{:?}", res);
}

from serde-xml-rs.

The best workaround I found at the moment is to use an enum with #[serde(other) and #[serde(deserialize_with)] to deserialize unknown tags:

use serde::{Deserialize, Deserializer};

#[derive(Deserialize, Debug)]
#[serde(rename_all = "lowercase")]
enum Item {
    Foo(String),
    Bar(String),
    #[serde(other, deserialize_with = "deserialize_ignore_any")]
    Other,
}

fn deserialize_ignore_any<'de, D: Deserializer<'de>>(deserializer: D) -> Result<(), D::Error> {
    serde::de::IgnoredAny::deserialize(deserializer)?;
    Ok(())
}

#[derive(Deserialize, Debug)]
struct Root{
    #[serde(rename="$value")]
    items: Vec<Item>
}

fn main() {
    let xml = r#"<root> <foo>a</foo> <bar>b</bar> <foo>c</foo>  <unknown>d</unknown> </root>"#;
    let v: Root = serde_xml_rs::from_str(xml).unwrap();
    println!("{:?}", v); // prints: Root { items: [Foo("a"), Bar("b"), Foo("c"), Other] }
}

from serde-xml-rs.

Just as a heads up, I've started work on a potential solution to this issue. I hope to have a PR in the coming weeks.

from serde-xml-rs.

Changing the output from the original program probably isn't going to happen, it's old and the order represents something from the underlying data.

For now I'm using a separate processing pass to group children together, I guess that's the best option for now. Thanks!

from serde-xml-rs.

@sapessi, actually the only problem is maybe the error message.

The solution is to modify the type definitions to the following.

#[derive(Debug, Serialize, Deserialize, PartialEq)]
#[serde(rename = "operation")]
pub struct OperationBinding {
    name: String,
    operation: SoapOperation,
    #[serde(rename = "input")]
    input: SoapDataBindings,
    #[serde(rename = "output")]
    output: SoapDataBindings,
}

#[derive(Debug, Serialize, Deserialize, PartialEq)]
#[serde(rename = "operation")]
pub struct SoapOperation {
    #[serde(rename = "soapAction")]
    soap_action: String,
}

#[derive(Debug, Serialize, Deserialize, PartialEq)]
pub struct SoapDataBindings {
    #[serde(rename = "$value")]
    bindings: Vec<SoapDataBinding>
}

#[derive(Debug, Serialize, Deserialize, PartialEq)]
#[serde(rename_all = "lowercase")]
pub enum SoapDataBinding {
    Body(BodyBinding),
    Header(HeaderBinding),
}

#[derive(Debug, Serialize, Deserialize, PartialEq)]
#[serde(rename = "body")]
pub struct BodyBinding {
    parts: String,
    #[serde(rename = "use")]
    body_use: String,
}

#[derive(Debug, Serialize, Deserialize, PartialEq)]
#[serde(rename = "header")]
pub struct HeaderBinding {
    message: String,
    part: String,
    #[serde(rename = "use")]
    header_use: String,
}

from serde-xml-rs.

@tobz1000 https://github.com/ralpha/serde_deserializer_best_effort I created this some time ago, it works but is not clean by any means. But maybe it helps.
If you actually solve this problem that would be wonderful! 😃

from serde-xml-rs.

Thank you @lovasoa.

IMHO, this isn't a workaround but a solution.

from serde-xml-rs.

I opened an issue for this on serde's end, but they seem to point to this crate to handle it.

from serde-xml-rs.

@e-oz I've provided working code as above. It's not an issue with this crate, just how serde works. You can always implement a custom deserializer as well.

from serde-xml-rs.

@cetra3 I wasn't answering to author of this crate. It's issue with serde design. And we all can implement something, but only if basic functionality is covered we can call it "xml deserializer".

from serde-xml-rs.

@e-oz You can use serde to deserialize data, you just need to instruct it how to construct your struct. Examples are given for map and struct.

But if you are using the serde_derive crate, then you will need to conform to what shape it expects the data in, but it will deserialize. If you tried the same thing with any other magical deserialization library (i.e, jackson) you'd have the same problem.

from serde-xml-rs.

@cetra3 thank you for trying to help me.

from serde-xml-rs.

Just to say this bit me as well. For reference for those reading this up in future here's a tldr:

• There were discussions for customizing duplicate handling (serde-rs/serde#690) but for some reason this was de-scoped from the project and no workaround is available atm (to my knowledge)
• Because serde-xml-rs does not load the entire document in memory (it reads it as a stream), it's unable to read forward without skipping over the items in between (so without a re-write + an agreement on reducing performance to support this it won't be happening anytime soon/ever)

The #[serde(rename = "$value")] + Enum approach doesn't work for me either as it expects to handle every child element it encounters and requires putting every possible tag into the enum (any attempt to declare it in the parent struct instead is ignored and triggers an error "unknown variant xxxx")

• OK I could put every possible element into the enum but it will get annoying going through a list of enums just to a get a singleton child element that should have its own field

The only real option atm is to either hack up your own solution or to modify your input xml.

• (FYI this isn't just a problem with this serde deserializer, serde-json arguably has a worse problem with this, although repeated keys are much rarer!)

from serde-xml-rs.

Because I came across this issue multiple times now, I have tried to solve it.
I first tried using the Enum example above, but this had other limitation that I could not work around and it made the code look terrible.
So I implemented my own deserializer. And got it working! 😃
I also tried implementing a derive macro. And after a lot of trail and error I got it working.
The result is not pretty, but you can find it here:
https://github.com/ralpha/serde_deserializer_best_effort
(it will probably not solve your problem, but might help you)

I included both the manual implementation and the derive trait.
The code could be so much nicer if it was included in the serde crate.
But I was not very familiar with writing macro's before this.
(A good thing that compilers course from a few year ago came in handy 😉 )
I don't know if this behavior could be implement in this or the serde crate/repo.

TL;DR: Solved it, but with its limitation: https://github.com/ralpha/serde_deserializer_best_effort

from serde-xml-rs.

@Jonesey13 and @ralpha, thank you for these contributions to the discussion.

Please, would you mind contributing example schemas or documents?

from serde-xml-rs.

@punkstarman Could you be a bit more specific? What part would you like me to clarify?
I looked even more into this and see that the problem is pity hard-coded into the serde crate.
So I think it should be changed there. (with possible changes here afterwords depending on how this is solved).

I put a longer and more detailed explanation of how this could be solved here: serde-rs/serde#1725 (comment)

from serde-xml-rs.

Could you be a bit more specific? What part would you like me to clarify?

@ralpha, the examples you and others have provided are all minimalistic unless I have overlooked something. This is great for reproducing the problem and checking whether it has been addressed or not, but it makes it difficult to convince that the problem is valid.

My difficulty currently is that I cannot find a compelling real-world use case where deserializing as an enum is unwarranted.

One case mentioned here is that when there is a mixture of singleton and repeatable children. In the few XML schemas I know that allow this, the singleton elements must appear first. Other schemas avoid mixing singleton and repeated elements as siblings by putting the latter into a collection element.

On the other hand, I feel that the order is important when repeated elements can be interleaved as siblings. It seems odd to me to want to deserialize into segregated arrays or vectors and lose the order.

For instance in HTML order is important,

<body>
  <p>...</p>
  <ul>...</ul>
  <p>...</p>
  <ul>...</ul>
</body>

Would not be serialized as

Body {
  ps: [ ..., ... ]
  uls: [ ..., ... ]
}

The fact that I cannot find a case doesn't there isn't one, so if anyone can provide one, I would be grateful.

I looked even more into this and see that the problem is [pretty] hard-coded into the serde crate. So I think it should be changed there.

I agree

I think one or several compelling use cases would go a long way in motivating the change.

from serde-xml-rs.

Okay, I'm deserializing xml data that was created by an other program that is outside my control, so I can not change the data or structure it gives me. (and there are multiple versions of the software that give slightly different exports)

The program that deserializes the data (that I'm creating) is going to be used by clients that have there own export from the program. They can input there xml file(s) and it is deserialized right there and the data is used by the program.

The data consist of a bunch of lists with objects that with a lot of optional tags. So I have many structs with about 30-40 tags in them, with sub struct nested in them. So changing this to an Enum make is much much harder to manage the code. The order of the items in the objects is not important to me, just the data. So it is not like the html you put above.

The program that creates the xml (NOT my program) that I'm using probably has some code like this:

for x in list {
    if x.type == "type1" {
        println!("<type1>...<a>1</a><b>2</b>...With a lot of data in here with nested tags...</type1>");
    } else {
        println!("<type2>...<c>3</c><d>4</d>...With a lot of data in here with nested tags...</type2>");
    }
}

This (I think) is a common thing that happens in a lot of programs and this creates interleaved tags. Which is valid xml so they do this. And there is no way for me to know this up front if it does this. (code is closed source, so can not inspect it) So I just test it out on about 20+ different exports and see where it gives me errors.

But this means I have to rewrite my whole codebase just because there is/could be 1 interleaved tag in the whole xml.
The files I'm loading could be as large as 700 MB if not larger, so it could take a while to find 1 very infrequent tag that just happens to be interleaved with a probability of 1/2 000 000 or so. This will then throw an error and stop all the parsing of the whole file.

Summary
The enum solution might work in some cases but when you have a struct with 40 optional fields this quickly gets complicated. And when I want to use the data I have to write massive match blocks to capture all optional fields.

That is why if this problem is fixed I will make to SO much easier to actually use it. Writing that custom deserializer was less work (even though it took me quite a while) then changing the program to use enums.

So I think there are very valid use-cases for this.
And you only find out you have this problem when you have already created a large part of your code. Which is not fun in any way.
If this problem is solved in Serde it will also solve problems with json and others.

Here are reported issues I found where people has this problem (not trying to count people double):
serde-rs/serde#1725 (2 people)
tafia/quick-xml#177 (1 new person)
#55 (this issue, 4 people)
serde-rs/serde#1113
serde-rs/serde#690
faebser/beautiful-wallpaper-every-day#1
#5
serde-rs/serde#1661
... And there are probably many, many more that did not report the issue or just read it...
( In some of these cases Enums solved the problem or the problem could be fixed with other options like serde_with )

from serde-xml-rs.

@punkstarman The use case I was working against was for parsing SVG files. For example when generating SVG in inkscape it's quite common to output interspaced <rect> and <path> elements.

Your point about the ordering of the elements is a very good one and tbh I can't think of a good counterpoint (as it's also important in the SVG case like it is with HTML; technically you could argue layers should be used for the draw order but it's a bit of a moot point as it's part of the SVG spec).

However I do agree with @ralpha that having to include all possible tag in the enum is annoying to work with at best when you can't choose to just ignore them.

from serde-xml-rs.

Well seems like it is not getting fixed in Serde: serde-rs/serde#1725 (comment)
And as it is currently hard coded in serder_derive It is probably not going to be fixed there unless serde(_derive) is forked and changed for XML. Which sadly splits the ecosystem a bit.
But maybe only the derive traits could be changed and included in this crate?
@punkstarman What do you think? What should happen, if anything at all?

from serde-xml-rs.

@ralpha

Well seems like it is not getting fixed in Serde: serde-rs/serde#1725 (comment)
And as it is currently hard coded in serder_derive It is probably not going to be fixed there unless serde(_derive) is forked and changed for XML. Which sadly splits the ecosystem a bit.

Someone already did this, see https://github.com/media-io/yaserde.

But maybe only the derive traits could be changed and included in this crate?

I don't believe that this is technically possible.

I think that packaging a custom deserializer in its own crate would be the way to go.

@punkstarman What do you think? What should happen, if anything at all?

With regards to this problem, I don't see how anything can be done in the serde-xml-rs crate. The serde and serde_derive crates paint us into a corner and were never really designed with XML in mind.

from serde-xml-rs.

I'm trying to use enums to parse a WSDL definition that has body/headers in the wrong order (example below). It seems that the parser fails to successfully close to element or "peek" at the next one. Have you encountered this before?

XML sample

<wsdl:operation name="ResolveNames">
  <soap:operation soapAction="http://schemas.microsoft.com/exchange/services/2006/messages/ResolveNames"/>
  <wsdl:input>
    <soap:body parts="request" use="literal"/>
    <soap:header message="tns:ResolveNamesSoapIn" part="Impersonation" use="literal"/>
    <soap:header message="tns:ResolveNamesSoapIn" part="MailboxCulture" use="literal"/>
    <soap:header message="tns:ResolveNamesSoapIn" part="RequestVersion" use="literal"/>
  </wsdl:input>
  <wsdl:output>
    <soap:body parts="ResolveNamesResult" use="literal"/>
    <soap:header message="tns:ResolveNamesSoapOut" part="ServerVersion" use="literal"/>
  </wsdl:output>
</wsdl:operation>

Rust structs

#[derive(Debug, Serialize, Deserialize, PartialEq)]
#[serde(rename = "wsdl:operation")]
pub struct OperationBinding {
    name: String,
    operation: SoapOperation,
    #[serde(rename = "input")]
    input: Vec<SoapDataBinding>,
    #[serde(rename = "output")]
    output: Vec<SoapDataBinding>,
}

#[derive(Debug, Serialize, Deserialize, PartialEq)]
#[serde(rename = "soap:operation")]
pub struct SoapOperation {
    #[serde(rename = "soapAction")]
    soap_action: String,
}

#[derive(Debug, Serialize, Deserialize, PartialEq)]
#[serde(rename_all = "lowercase")]
pub enum SoapDataBinding {
    Body(BodyBinding),
    Header(HeaderBinding),
}

#[derive(Debug, Serialize, Deserialize, PartialEq)]
#[serde(rename = "soap:body")]
pub struct BodyBinding {
    parts: String,
    #[serde(rename = "use")]
    body_use: String,
}

#[derive(Debug, Serialize, Deserialize, PartialEq)]
#[serde(rename = "soap:header")]
pub struct HeaderBinding {
    message: String,
    part: String,
    #[serde(rename = "use")]
    header_use: String,
}

Log output

2020-05-22 08:30:00,076 DEBUG [serde_xml_rs::de] Peeked EndElement({http://schemas.xmlsoap.org/wsdl/soap/}soap:operation)
2020-05-22 08:30:00,076 DEBUG [serde_xml_rs::de] Fetched EndElement({http://schemas.xmlsoap.org/wsdl/soap/}soap:operation)
2020-05-22 08:30:00,076 DEBUG [serde_xml_rs::de] Peeked StartElement({http://schemas.xmlsoap.org/wsdl/}wsdl:input, {"": "", "s": "http://www.w3.org/2001/XMLSchema", "soap": "http://schemas.xmlsoap.org/wsdl/soap/", "t": "http://schemas.microsoft.com/exchange/services/2006/types", "tns": "http://schemas.microsoft.com/exchange/services/2006/messages", "wsdl": "http://schemas.xmlsoap.org/wsdl/", "xml": "http://www.w3.org/XML/1998/namespace", "xmlns": "http://www.w3.org/2000/xmlns/"})
2020-05-22 08:30:00,076 DEBUG [serde_xml_rs::de] Peeked StartElement({http://schemas.xmlsoap.org/wsdl/}wsdl:input, {"": "", "s": "http://www.w3.org/2001/XMLSchema", "soap": "http://schemas.xmlsoap.org/wsdl/soap/", "t": "http://schemas.microsoft.com/exchange/services/2006/types", "tns": "http://schemas.microsoft.com/exchange/services/2006/messages", "wsdl": "http://schemas.xmlsoap.org/wsdl/", "xml": "http://www.w3.org/XML/1998/namespace", "xmlns": "http://www.w3.org/2000/xmlns/"})
2020-05-22 08:30:00,076 DEBUG [serde_xml_rs::de] Peeked StartElement({http://schemas.xmlsoap.org/wsdl/}wsdl:input, {"": "", "s": "http://www.w3.org/2001/XMLSchema", "soap": "http://schemas.xmlsoap.org/wsdl/soap/", "t": "http://schemas.microsoft.com/exchange/services/2006/types", "tns": "http://schemas.microsoft.com/exchange/services/2006/messages", "wsdl": "http://schemas.xmlsoap.org/wsdl/", "xml": "http://www.w3.org/XML/1998/namespace", "xmlns": "http://www.w3.org/2000/xmlns/"})
2020-05-22 08:30:00,076 DEBUG [serde_xml_rs::de] Peeked StartElement({http://schemas.xmlsoap.org/wsdl/}wsdl:input, {"": "", "s": "http://www.w3.org/2001/XMLSchema", "soap": "http://schemas.xmlsoap.org/wsdl/soap/", "t": "http://schemas.microsoft.com/exchange/services/2006/types", "tns": "http://schemas.microsoft.com/exchange/services/2006/messages", "wsdl": "http://schemas.xmlsoap.org/wsdl/", "xml": "http://www.w3.org/XML/1998/namespace", "xmlns": "http://www.w3.org/2000/xmlns/"})
2020-05-22 08:30:00,076 DEBUG [serde_xml_rs::de] Fetched StartElement({http://schemas.xmlsoap.org/wsdl/}wsdl:input, {"": "", "s": "http://www.w3.org/2001/XMLSchema", "soap": "http://schemas.xmlsoap.org/wsdl/soap/", "t": "http://schemas.microsoft.com/exchange/services/2006/types", "tns": "http://schemas.microsoft.com/exchange/services/2006/messages", "wsdl": "http://schemas.xmlsoap.org/wsdl/", "xml": "http://www.w3.org/XML/1998/namespace", "xmlns": "http://www.w3.org/2000/xmlns/"})
2020-05-22 08:30:00,076 DEBUG [serde_xml_rs::de] Peeked StartElement({http://schemas.xmlsoap.org/wsdl/soap/}soap:body, {"": "", "s": "http://www.w3.org/2001/XMLSchema", "soap": "http://schemas.xmlsoap.org/wsdl/soap/", "t": "http://schemas.microsoft.com/exchange/services/2006/types", "tns": "http://schemas.microsoft.com/exchange/services/2006/messages", "wsdl": "http://schemas.xmlsoap.org/wsdl/", "xml": "http://www.w3.org/XML/1998/namespace", "xmlns": "http://www.w3.org/2000/xmlns/"}, [parts -> request, use -> literal])
2020-05-22 08:30:00,076 DEBUG [serde_xml_rs::de] Fetched StartElement({http://schemas.xmlsoap.org/wsdl/soap/}soap:body, {"": "", "s": "http://www.w3.org/2001/XMLSchema", "soap": "http://schemas.xmlsoap.org/wsdl/soap/", "t": "http://schemas.microsoft.com/exchange/services/2006/types", "tns": "http://schemas.microsoft.com/exchange/services/2006/messages", "wsdl": "http://schemas.xmlsoap.org/wsdl/", "xml": "http://www.w3.org/XML/1998/namespace", "xmlns": "http://www.w3.org/2000/xmlns/"}, [parts -> request, use -> literal])
2020-05-22 08:30:00,076 DEBUG [serde_xml_rs::de] Peeked EndElement({http://schemas.xmlsoap.org/wsdl/soap/}soap:body)
2020-05-22 08:30:00,076 DEBUG [serde_xml_rs::de] Fetched EndElement({http://schemas.xmlsoap.org/wsdl/soap/}soap:body)
2020-05-22 08:30:00,077 DEBUG [serde_xml_rs::de] Fetched StartElement({http://schemas.xmlsoap.org/wsdl/soap/}soap:header, {"": "", "s": "http://www.w3.org/2001/XMLSchema", "soap": "http://schemas.xmlsoap.org/wsdl/soap/", "t": "http://schemas.microsoft.com/exchange/services/2006/types", "tns": "http://schemas.microsoft.com/exchange/services/2006/messages", "wsdl": "http://schemas.xmlsoap.org/wsdl/", "xml": "http://www.w3.org/XML/1998/namespace", "xmlns": "http://www.w3.org/2000/xmlns/"}, [message -> tns:ResolveNamesSoapIn, part -> Impersonation, use -> literal])
ERROR : Could not parse WSDL: Expected token XmlEvent::EndElement { name, .. }, found StartElement({http://schemas.xmlsoap.org/wsdl/soap/}soap:header, {"": "", "s": "http://www.w3.org/2001/XMLSchema", "soap": "http://schemas.xmlsoap.org/wsdl/soap/", "t": "http://schemas.microsoft.com/exchange/services/2006/types", "tns": "http://schemas.microsoft.com/exchange/services/2006/messages", "wsdl": "http://schemas.xmlsoap.org/wsdl/", "xml": "http://www.w3.org/XML/1998/namespace", "xmlns": "http://www.w3.org/2000/xmlns/"}, [message -> tns:ResolveNamesSoapIn, part -> Impersonation, use -> literal])

from serde-xml-rs.

@sapessi, I can reproduce this with the current version. I'm looking into how to fix this.

I'll also see when this bug was introduced because to my recollection this used to work.

from serde-xml-rs.

Just why? Having a reusable tag is not insane, especially if the types are the same! Am I missing something?
This seems like a cop out.

from serde-xml-rs.

I made a pr to serde_with, so that deserialize_ignore_any can be imported instead of copy-pasted : jonasbb/serde_with#251

from serde-xml-rs.

@punkstarman I can provide valid xml examples that are libre sourced to test.

Joe's Window Manager

The Standard XDG menu is another example.
I tried all your solutions, but may have missed something. In the XDG menu, multiple Merge elements can exist, or not as a part of the standard example:

<-- snip-->
            <DefaultLayout>
              <Merge type="menus"/>
              <Merge type="files"/>
              <Separator/>
              <Menuname>More</Menuname>
            </DefaultLayout>
<-- snip-->

from serde-xml-rs.

@RReverser this issue still exists, as far as I can tell.
Specifically with both of the files I provide.
I have tried the enum, with the Other as well as using:

let mut de = serde_xml_rs::Deserializer::new_from_reader(file_string.as_bytes())
                                             .non_contiguous_seq_elements(true);

What am I missing if this is fixed?
JWM specifically uses out of place items:

    <Tray x="0" y="-1" autohide="off" delay="1000">
       <!-- Tray button 0 -->
       <TrayButton label="JWM">root:1</TrayButton>
       <!-- Spacer 0 -->
        <Spacer width="2"/>
       <!-- Tray button 1 -->
        <TrayButton label="_">showdesktop</TrayButton>
       <!-- Spacer 1 -->
        <Spacer width="2"/>
        <Pager labeled="true"/>
        <TaskList maxwidth="256"/>
        <Swallow width="32" height="32" name="xclock">xclock</Swallow>
        <Dock/>
        <Clock format="%l:%M %p"><Button mask="123">exec:xclock</Button></Clock>
    </Tray>

I have tried your tricks and they do not work. Which likely means I am missing something.
The Vec<enum> method does not work even with:

pub enum Stuff {
   //my enums
   #[serde(other, deserialize_with = "deserialize_ignore_any")]
   Other,
}
pub struct File {
  #[serde(rename = "$value")]
  pub items:Vec<Stuff>, // still get error "custom: duplicate field `<whatever>`"
}

from serde-xml-rs.

I've implemented parsing overlapped sequences in tafia/quick-xml#387 under a feature flag overlapped-lists. Be noted, that enabling that feature could lead to quadratic parsing and high memory consumption, so use it at your risk.

from serde-xml-rs.