JATS Import/Export about lens HOT 15 OPEN

JATS support is definitely not far away. With Substance you can register different converters so in addition to the LensXMLImporter + Exporter we could add a JATSXMLImporter + Exporter. In that case no XSLT would be needed. You could maintain that JATS converter yourself, that could also have some metadata conversion in place, which may be unique to PKP.

We need to discuss a bit about what node types are missing that are crucial for you. Should we make this a little project? The JATS converter but also PKP integration in general. What time frame do you have in mind?

from lens.

(Hi, @michael -- let me belatedly introduce you to @axfelix. And, though he won't understand why, James: @jmacgreg)

from lens.

Hello!

from lens.

Hi guys,

Glad we're all here!

So, for one thing, I think our timeline would be have to something usable in place by Q2 2016 -- right now we're mainly talking about using Substance in the context of our XML transformation stack, but I think we're really trying to have this ready in time for the first release of OJS 3.

As for maintaining something ourselves, I think we have a fairly vested interest in getting something upstream, for various reasons which include the lack of any nodejs folks on our team currently, so I would like to make sure we do something that aligns with your goals too.

Like I said, the only body element that immediately jumps out at me as missing from the current Lens Writer implementation is footnotes (@jmacgreg, @asmecher, anything I'm forgetting?). Whether we want to do build in more scaffolding around frontmatter and ref-list editing is up in the air. We definitely have a need to support citations of articles that don't have DOIs, so I expect we'll have to do something there, but I'm not sure if we want to commit to building an element-by-element citation editor ...

Likewise with front matter -- this interface already exists within OJS, and would be sufficient to staple some JATS together behind the scenes, but I was envisioning hooking Lens Writer into our XML stack first and then OJS making a call to the XML stack for all document editing operations, so it wouldn't stand alone as well if we rely on the OJS interface (right, @jalperin?)

from lens.

Hi everybody! Glad to meet you all :)

There is some time until Q2 2016. Oliver and I had some longer discussions about the Lens exchange formats yesterday. The LensXML as it is now is modelled after our current in-memory document model, thus the converters were really straight forward to implement. However, I do see that JATS support will be a fundamental feature. So we will get that in! As you mentioned, maintainability of the converters is something we should pay attention to. We need to evaluate this more but it may be a good idea to make JATS the primary input format for Lens, so we don't have to maintain two serialization formats. However we would support only a more strict subset of JATS. We would need some kind of priority list from you, which elements you need. Also which information must be preserved (even when not edited in Lens). We want to implement the converter against a testsuite, which will be based on the usecases (examples) you bring up to us. The converter can run client and server side! So with Substance you would be able to implement clean-up tasks that run on the server.

Footnotes: That's an easy one, we would not want to call it footnote though, since we are in the digital world. ;) How about just Note?

Tables: We had them in already in an earlier version (see http://cdn.substance.io/lens-writer/), but we dropped it until table editing is fully implemented. Full table support will be part of our next Substance beta.

References: We currently use bibitem elements that have Citeproc JSON as a source. And then we use citeproc for rendering them. Will your source XML files have structured ref elements or mixed citations?

Front matter editing: Title and abstract can be edited in Lens. Also other metadata but that would be more of a custom plugin then, as journals have different metadata.

Adding References that have no DOI: We are aware of that problem. The plan here is to allow importing common formats such as bibtex but also Mendeley, EndNote etc. We don't want to implement a reference editor! (Maybe one day as a community effort when everybody uses Lens already;)). However, what are your exact requirements here?

from lens.

Hm, I'm afraid "just note" might not be descriptive enough, and I'm not necessarily afraid of the evils of "footnote" in the digital world, seeing as the JATS elements are called <fn-group> and <fn>, but I won't press the issue; if we disagree on the labeling, that's definitely the sort of thing I'm comfortable maintaining as a tweak from upstream :)

Glad tables are already spoken for. That explains why the node type seemed to be in there already.

Front matter editing -- the more I think about this, the more I don't think it's necessary for a Lens implementation. If we're expecting regular authors to be using our XML stack to prepare a manuscript, it's not on them to get the journal's metadata completely right, and if we're expecting journal managers or editors to be using our XML stack (along with Lens), it'll be in the context of some other journal management interface. It's more important that we focus on allowing these metadata elements to be preserved / stapled in afterwards through our API and Lens' API when not being edited directly than it is that we try to accommodate everything here.

So that really just leaves references. We generate citeproc JSON as part of our stack, though we do it directly via ParsCit and some Bibutils calls (I'd like to say "for citations that can't be successfully looked up," but we haven't actually implemented a lookup yet, though that's on the to-do list) before styling it into a user's chosen CSL for our HTML/PDF/ePub outputs using Pandoc. But the problem is that there are cases where our stack fails to catch the ref-list at all (or just does a crap job of parsing it), and if we're envisioning Lens Writer being used as an all-purpose cleanup engine, I'm afraid we probably do need a lot more scaffolding around identifying (delineating?) which references are which, even if we don't go as far as making a popout Title-Year-Author-etc. editor for each reference -- which I also don't want to do :)

Thoughts? I think so far we've established that, as far as Lens development is concerned, we for-sure need footnotes and better JATS support. We need some further scoping (other folks I've pulled into this thread, please feel free to join in) of reference editing, and frontmatter and citation lookup stuff can both be handled within the context of our existing stack.

from lens.

Also pulling @MartinPaulEve in because I suspect he'll have opinions here...

from lens.

Juan here, from PKP—I do think a metadata editor is important, but agree it should be a plugin-type approach, since there could be a lot of variation on what is considered essential metadata between publishers.

from lens.

further to this:

Juan Pablo Alperin: i meant to comment on the substance issue
Alex: oh. nope, but please do
Juan Pablo Alperin: I think getting a metadata editor plugin is important
Alex: oh you bastard
Juan Pablo Alperin: agreed it should be a plug in type thing
Juan Pablo Alperin: well, of course i'd disagree with you ;)
Juan Pablo Alperin: agreed w/ Michael it should be a plugin, since each publisher will have different req
Alex: we'll see
Alex: my thinking is just that it's a trivial thing
Alex: and the expectation would be set that once in metadata fields within e.g. OJS
Alex: and have our plugin generate the frontmatter stub for the journal and staple it in every time
Juan Pablo Alperin: would be ideal to not make this dependent on OJS
Alex: I thought that at first too
Alex: but it's not so much that it's "dependent" on OJS
Alex: but that front matter is necessarily specific to journal workflows
Juan Pablo Alperin: still, this could make it stand-alone, which would be nice.
Alex: and I don't really want to overload lens with unnecessary interfaces
Juan Pablo Alperin: and seems not all that hard
Juan Pablo Alperin: it needs to at least be displayed
Alex: do people really need a "standalone" interface for putting in their journal and issue number?
Juan Pablo Alperin: so people can detect errors
Alex: yes, for sure, we can agree about that
Juan Pablo Alperin: and once you're displaying… its editable ;)
Alex: but a plugin workflow should also just replace frontmatter elements with known-good stuff
Alex: that was already input elsewhere
Alex: althooooough if we wanted our stack to automatically populate metadata fields in OJS...
Juan Pablo Alperin: also a good point
Juan Pablo Alperin: it's good to have in the plans
Alex: okay, so there's maybe more scoping here
Alex: I'm just going to paste this into the git issue
Juan Pablo Alperin: agreed that the other elements identified is more important right now

from lens.

OK, so at this point, I think we're looking at:

• Definitely adding support for footnotes and finalizing table editing.
• Definitely supporting the import/export of JATS while reserving some elements when passing the document to Substance/Lens through an API call (no reason this couldn't just be a CURL parameter or something).
• Possibly a minimal interface for editing front matter (this may be as simple as creating the framework for a pop-over UI window which has individual fields for different elements) -- unless, @asmecher, you think it'd be feasible to abstract out the existing front matter interface from OJS3 so that it's usable for writing to a JATS document (which could be manually constructed from OJS DB values if needed) as part of an API handoff when a user isn't actually making use of OJS for the rest of their workflow?
• Possibly a minimal interface for editing references -- again, could be a pop-over UI window with 5-10 basic fields for Title, Author, etc., for references that were present in the imported JATS but couldn't be looked up via DOI. Definitely want a structured editor if we go this route; no plaintext, as citeproc will be styling them later.

Should we move forward with figuring out how much work this would be?

from lens.

Maybe we meet on Skype next week and discuss the individual points, I think we can give a rough estimation then?

from lens.

That would be good. Most of us are on pacific time, and I believe you're in GMT or GMT+1, so it'll probably be early morning (8am or 9am) for us, and my early mornings are all open next week with the exception of Friday (which happens to be Christmas). Any preference?

from lens.

Hey we are GMT+1 yes. Monday 8am (Pacific time) would be fine for us. For call coordination just message us at info at substance io.

I'm looking forward to make this more concrete. Regarding timing, I think we really should only scope the JATS support for a start. We don't have enough resources to offer customization work atm... but we'd be glad to assist and give a hands-on-workshop. Don't you have a brave developer who wants to dive into that world? It's really not hard, and it would be fun. Who wants? :)

Talk soon!

from lens.

Hey are you still in for a call today? I haven't heard back. We are available at 9am (Pacific time) just ping us if you want to talk.

from lens.

Hey Michael,

Sorry I forgot to confirm -- we're available! Would you prefer 8am PST (in 25 minutes) or 9am, at this point?

from lens.