[REVIEW]: Elektra: universal framework to access configuration parameters about joss-reviews HOT 41 CLOSED

Thanks @eschnett for doing more than you probably thought you would when you agreed to be a reviewer :)

from joss-reviews.

@eschnett many thanks for reviewing this submission and @danielskatz thanks for editing ✨

@markus2330 - your paper is now accepted into JOSS and your DOI is http://dx.doi.org/10.21105/joss.00044 ⚡️ 🚀 💥

from joss-reviews.

OK, the reviewer is @eschnett

from joss-reviews.

/ cc @openjournals/joss-reviewers - would anyone be willing to review this submission?

If you would like to review this submission then please comment on this thread so that others know you're doing a review (so as not to duplicate effort). Something as simple as :hand: I am reviewing this will suffice.

Reviewer instructions

• Please work through the checklist at the start of this issue.
• If you need any further guidance/clarification take a look at the reviewer guidelines here http://joss.theoj.org/about#reviewer_guidelines
• Please make a publication recommendation at the end of your review

Any questions, please ask for help by commenting on this issue! 🚀

from joss-reviews.

@whedon commands

from joss-reviews.

Here are some things you can ask me to do:

# List all of Whedon's capabilities
@whedon commands

# Assign a GitHub user as the reviewer of this submission
@whedon assign @username as reviewer

# List the GitHub usernames of the JOSS editors
@whedon list editors

# List of JOSS reviewers together with programming language preferences and domain expertise
@whedon list reviewers

# Change editorial assignment
@whedon assign @username as editor

# Open the review issue
@whedon start review

🚧 Important 🚧

This is all quite new. Please make sure you check the top of the issue after running a @whedon command (you might also need to refresh the page to see the issue update).

from joss-reviews.

@danielskatz - I've updated the issue metadata (in the original post at the top of this issue now). Once you've found someone to help review this issue you can do:

@whedon assign @username as reviewer 

As there's no 'meta' issue for the reviewer assignment here you don't need to do the @whedon start review bit.

from joss-reviews.

@whedon assign @eschnett as reviewer

from joss-reviews.

@eschnett, you should now be able to do this review. You should review the reviewer instructions on http://joss.theoj.org/about. Let me know if you have any problems

from joss-reviews.

@eschnett 👋 is there anything we can do to help you with this review?

from joss-reviews.

Hi @eschnett - we would like to get some feedback on this, so that the authors can respond to it.

from joss-reviews.

Several of the self-tests were failing; this was reported and resolved in ElektraInitiative/libelektra#997 .

from joss-reviews.

The licence was difficult to find, since it was in a file with a non-standard name ElektraInitiative/libelektra#995 .

from joss-reviews.

Regarding the statement of need: I was initially confused as to the purpose of this library. The documentation speaks of "custom configuration files"; depending on context, this can mean many different things. For example, in one context this can refer to configuring some software ahead of time, and then letting the software executing in batch mode. This doesn't seem to be the use case for Elektra, since it offers features to dynamically updating configuration entries. In other cases, different parts of configurations need to be kept consistent, in a way that can be tested or enforced automatically. Elektra offers no features for this, so this doesn't seem to be part of the use case either. I assume now that Elektra is meant to dynamically configure some software, or several loosely related software installs. It would be good to clarify this, e.g. with an example, in the introduction or the summary.

See ElektraInitiative/libelektra#1054 for a concrete question.

from joss-reviews.

I assume the license issue has been resolved since you checked the box?

from joss-reviews.

Yes, the licence issue has been remedied.

from joss-reviews.

since statement of need is also checked, is that issue also resolved?

from joss-reviews.

It's resolved for the overall documentation, but the paper could be clearer. I didn't check the box there yet.

from joss-reviews.

I see, thanks

I hope you can make a clear statement here at some point of what issues the authors need to resolve.

from joss-reviews.

While testing functionality I've encountered ElektraInitiative/libelektra#1055 .

from joss-reviews.

Thank you for your effort and reporting helpful issues! I slightly rephrased the paper to clarify that it is about dynamic configuration (at run-time) and added a statement in doc/WHY.md about which user groups we target.

from joss-reviews.

libelektra is a large project, and I haven't yet found a way to explore it. I don't want to write (or "elektrify") one of my applications just to test it. Do you have a toy code to play with? Or a tutorial with commands to follow along? As is, I have trouble exploring libelektra's functionality.

from joss-reviews.

I have examined libelektra as submitted to the Journal of Open Science. I find that this submission meets or exceeds the requirements of the journal in almost all categories, with two exceptions:

(1) The software paper does not "clearly state what problems the software is designed to solve". In fact, this is one of the major issues with libelektra's documentation -- there is much documentation, but there are virtually no introductions. Much of the documentation remains opaque without context.

For example, the documentation speaks of "configuration" without stating exactly what the term means, and without giving examples. The "validation" tutorial does not state what validation is, only that a "configuration" can be "non-validating". The "compilation variants" tutorial begins with discussing "creat[ing] different variants of the same feature", without stating what a "variant" or a "feature" is.

(2) I was not able to confirm functionality or performance of this software. libelectra is a large project with many plugins, and it would be beyond the scope of such a review to test all parts of the software. However, I find that even a cursory examination of libelektra's functionality would have required a significant time investment from me, as there are no simple tutorials that would allow getting a first impression of libelektra.

There are only
(a) A video demonstration that seems slightly outdated (commands give different output from me than they do in the movie)
(b) An "application integration tutorial" that would have me modifying an existing application. However, there is no simple example application provided; instead, I would have to engage in a real programming project and learn libelektra's API
(c) A folder with about 15 tutorials that all seem to cover advanced topics such as e.g. "Validation"; none is a "first steps" tutorial.

After skimming much of the documentation, I conclude that libelektra is about configuring applications in the sense of managing user settings in a Gnome or KDE desktop environment for the various graphical applications there. While some settings are application specific, others are more general and apply to multiple applications, or to all of them. Configuration settings are seen as key-value pairs, and managing both the possible keys as well as the user's choices for the values is a complex task. libelektra addresses this. Its features are designed for this use case, and seem to include editing keys and values at run time, accessing keys and values from applications, managing a hierarchical key space, etc. One of the major features of libelektra seems to be that it can encompass existing configuration management software, including existing configuration files in various formats. libelektra claims to unify many existing mechanisms, instead of providing a new one.

I really wish that such a text could be found in libelektra's documentation.

I conclude that this project is not friendly for outsiders, as they have no starting point. I want to emphasize that the feedback I received on the libelektra bugtracker was quick and exceptionally friendly. In this review, will leave the boxes for "functionality" and "performance" in the functionality category, as well as for "statement of need" in the "paper" category unchecked.

from joss-reviews.

Thanks @eschnett. If there are no performance claims made, you can check the performance box. If there are claims made that you do not find convincing, that's a different story.

@markus2330, can you address these concerns? It seems like some introductory material is needed in the software paper, and that the software's documentation needs some content to make it more accessible to new users, such as is mentioned in @eschnett's point 2 above.

from joss-reviews.

@danielskatz Yes, no particular performance claims are made -- I have checked the "performance" box.

from joss-reviews.

Yes, I can address the concerns. We are currently writing a verification tool that checks that all commands in the tutorials produce the correct output (do not fail) and rewrite the tutorials to have a nice flow and assume less prior knowledge. Furthermore we develop a web page which makes navigation/reading of the tutorials easier.

Are there any time constraints when this should be finished? I am currently at a conference, thus I was not responsive the last days.

from joss-reviews.

And most importantly: A huge thanks to @eschnett for pointing us to the problems. I see that they are critical.

from joss-reviews.

I don't think there's a real hurry to this, other than that the longer it takes to update, the longer it will be before the paper can be accepted and published.

from joss-reviews.

Together with our latest release 0.8.19 I updated the paper and we created a tutorial that does not assume prior familiarity: http://tree.libelektra.org/doc/tutorials/mount.md

from joss-reviews.

@eschnett, back to you for updated review comments...

from joss-reviews.

ping: @eschnett

from joss-reviews.

@markus2330 The documentation is now much improved, thank you. However, the tutorials themselves are not quite there yet. I've provided some feedback in ElektraInitiative/libelektra#1189 . Since a tutorial is not required by JOSS, this feedback won't affect my review.

@danielskatz I have examined the functionality of libelektra in a cursory manner. The library offers a large API and script interface, and I have looked only at certain basic parts of it. Does this suffice to cover "Functionality: Have the functional claims of the software been confirmed?"?

from joss-reviews.

@eschnett yes, in my opinion, as I don't think a reviewer can be expected to exhaustively test the all possible functions of the software

@arfon will this suffice to meet the JOSS standards?

from joss-reviews.

@whedon ping @arfon

from joss-reviews.

@arfon will this suffice to meet the JOSS standards?

Yes I think this is acceptable.

If we're ready to move forward with accepting this submission then @markus2330 could you make an archive of the reviewed software in Zenodo/figshare/other service and update this thread with the DOI of the archive? I can then move forward with accepting the submission.

from joss-reviews.

I have checked the last box. I understand that @arfon will continue from here.

from joss-reviews.

Thank you for all your efforts! Can I archive the latest released version (0.8.19) or do I have to use the version when the review process started (0.8.17). In the last round @eschnett looked at 0.8.19.

from joss-reviews.

Thank you for all your efforts! Can I archive the latest released version (0.8.19) or do I have to use the version when the review process started (0.8.17). In the last round @eschnett looked at 0.8.19.

@markus2330 - please archive the latest released version (0.8.19)

from joss-reviews.

Thank you! Here is the DOI
https://doi.org/10.5281/zenodo.200894
links to
https://zenodo.org/record/200894

I also updated the paper.md (some time ago, but more recent than the pdf linked above)

If you are interested on some "failed" attempts to create a DOI you may continue to read...

Zenodo did not allow to create a DOI for an existing release thus I cloned the repo and created a new release for the existing 0.8.19 tag.

I also uploaded the signed tar on figshare:
https://figshare.com/articles/Elektra_0_8_19/4312394
which has the DOI:
https://dx.doi.org/10.6084/m9.figshare.4312394.v1
but unfortunately the page does not look nice ("sorry, we can't preview this file").

And I also created a DOI of the repository where the .tar.gz of the releases are:
https://doi.org/10.5281/zenodo.200884
but this is 91.6 MB and thus less useful.

from joss-reviews.

@whedon set 10.5281/zenodo.200894 as archive

from joss-reviews.

OK. 10.5281/zenodo.200894 is the archive.

from joss-reviews.