[REVIEW]: alchemlyb: the simple alchemistry library about joss-reviews HOT 52 OPEN

@orbeckst Thanks so much for the revision.
@srmnitc I read both the tutorial and how authors are now handling the list of developers. I agree that mentioning about the error estimation function is not required unless it was handled somehow differently within the alchemlyb. I am happy with the revision and I do not have any further comments.

Thanks @glycodynamics. In that case, could you please review your checklist again, and confirm that you recommend this paper for publication? Thanks a lot once again for your review!

@srmnitc I recommend this manuscript for publication.

from joss-reviews.

Thank you @srmnitc , great news!

Thank you @glycodynamics and @ryankzhu for the insightful and constructive reviews, it was a pleasure working with you, and your comments made the paper and alchemlyb better.

from joss-reviews.

@srmnitc Sorry for the late reply. I confirm I recommend this manuscript for publication.

from joss-reviews.

@mikemhenry and @philbiggin thank you for bringing this up. I will unassign you as reviewers. Thanks for your efforts nevertheless!

from joss-reviews.

@ryankzhu thanks for agreeing to be reviewer. You can start you review by using @editorialbot generate my checklist. Here, you can find some information about the checklist. Of course, feel fre e to ping/email me if you need any help. We aim to finish the review in 2-4 weeks. If you need more time, please let me know.

from joss-reviews.

@srmnitc I am sorry for the delay. I will write comments by the coming Tuesday.

from joss-reviews.

Thank you for your reviews @glycodynamics and @ryankzhu .

Would you be able to open issues in https://github.com/alchemistry/alchemlyb/issues for

• providing a single example case with input file, e.g. simple example, e.g., a self-contained jupyter notebook, to walk through the main functionalities.
• add link to docs inside README

We will address @glycodynamics 's comments in the paper

• Discussion of similar tools would be helpful for context and it's missing in the manuscript.
  • We do mention alchemical-analysis.py ; I am not aware of other tools that do the same as alchemlyb in terms of being applicable to many different file format, but we'll look carefully. If you have any specific software in mind please let us know!
• I noticed that the description of errors is missing from the paper, which could be a valuable addition.
  • Do you mean estimation of errors?

cc first author @xiki-tempula

from joss-reviews.

We will address @glycodynamics 's comments in the paper

• Discussion of similar tools would be helpful for context and it's missing in the manuscript.

  • We do mention alchemical-analysis.py ; I am not aware of other tools that do the same as alchemlyb in terms of being applicable to many different file format, but we'll look carefully. If you have any specific software in mind please let us know!

• I noticed that the description of errors is missing from the paper, which could be a valuable addition.

  • Do you mean estimation of errors?

cc first author @xiki-tempula

@orbeckst

• I agree that no other tools can handle output from multiple programs. It may be worth mentioning analysis tools of MD programs if they offer the same estimators and provide functionalities like checking convergence, error estimates, and overlap matrix. For example, analyse_freenrg from siremol is the one that came to my mind when I was testing and saw the plot of the overlap matrix.

• I meant to say estimation of errors, how are they obtained, internally in alchemylib or coming from the external program, such as bootstrapping in MBAR? I may have missed, but I didn't find good documentation of this.

Thanks and best wishes.

from joss-reviews.

We should be consistent between AUTHORS and doc footer and I raised an issue to address this discrepancy.

from joss-reviews.

We fixed the inconsistent author display in the docs by linking to the authoritative AUTHORS file. Please see cross-linked issue alchemistry/alchemlyb#383 .

from joss-reviews.

@orbeckst sorry for the delay from my side. I have opened an issue with some very small changes to the paper. Once you finish this, I can move ahead with the rest of the steps. Thanks again!

from joss-reviews.

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

from joss-reviews.

Software report:

github.com/AlDanial/cloc v 1.90  T=0.09 s (1054.2 files/s, 172073.0 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          50           2311           3704           6697
reStructuredText                34            502            370           1087
TeX                              1             40              0            280
Markdown                         1             41              0            186
YAML                             7             36             13            183
DOS Batch                        1              8              1             27
make                             1              4              6             10
-------------------------------------------------------------------------------
SUM:                            95           2942           4094           8470
-------------------------------------------------------------------------------

Commit count by author:

   139	Oliver Beckstein
    96	David Dotson
    82	Zhiyi Wu
    24	harlor
    12	Domenico Marson
     9	Ian Kenney
     7	Hyungro Lee
     6	trje3733
     5	shuai
     4	Jérôme Hénin
     4	Pascal Merz
     4	Victoria Lim
     3	Michael Shirts
     2	Irfan Alibay
     2	Mohammad Soroush Barhaghi
     2	Shuai Liu
     2	Tom Joseph
     2	hl2500
     1	Alexander Schlaich
     1	Bryce Allen
     1	David Mobley
     1	Wei-Tse Hsu
     1	brycestx
     1	helmut carter

from joss-reviews.

Paper file info:

📄 Wordcount for paper.md is 2661

✅ The paper includes a Statement of need section

from joss-reviews.

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/0021-9991(76)90078-4 is OK
- 10.1002/jcc.20290 is OK
- 10.1021/ct0502864 is OK
- 10.1016/j.softx.2020.100627 is OK
- 10.1021/jp807701h is OK
- 10.1021/acs.jcim.2c01052 is OK
- 10.21105/joss.01831 is OK
- 10.1063/1.1749657 is OK
- 10.1007/s10822-015-9840-9 is OK
- 10.1021/ct2003995 is OK
- 10.1063/1.3607597 is OK
- 10.1063/5.0014475 is OK
- 10.1021/jp102971x is OK
- 10.1016/j.softx.2015.06.001 is OK
- 10.1063/1.2978177 is OK
- 10.1063/1.1638996 is OK
- 10.1063/1.1740409 is OK
- 10.1007/s10822-019-00267-z is OK
- 10.33011/livecoms.2.1.18378 is OK
- 10.1021/acs.jctc.5b00784 is OK
- 10.25080/Majora-92bf1922-00a is OK
- 10.1016/j.cpc.2021.108171 is OK
- 10.1021/acs.jctc.8b00447 is OK
- 10.33011/livecoms.5.1.2067 is OK

MISSING DOIs

- No DOI given, and none found for title: Drug design: structure-and ligand-based approaches
- No DOI given, and none found for title: Scikit-learn: Machine Learning in Python
- No DOI given, and none found for title: API design for machine learning software: experien...
- No DOI given, and none found for title: Simulation techniques for solvation-induced surfac...

INVALID DOIs

- None

from joss-reviews.

License info:

✅ License found: BSD 3-Clause "New" or "Revised" License (Valid open source OSI approved license)

from joss-reviews.

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

from joss-reviews.

👋🏼 @orbeckst @glycodynamics, @mikemhenry, @philbiggin this is the review thread for the paper. All of our communications will happen here from now on.

As a reviewer, the first step is to create a checklist for your review by entering

@editorialbot generate my checklist

as the top of a new comment in this thread.

These checklists contain the JOSS requirements. As you go over the submission, please check any items that you feel have been satisfied. The first comment in this thread also contains links to the JOSS reviewer guidelines.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of just judging this submission. As such, the reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention openjournals/joss-reviews#REVIEW_NUMBER so that a link is created to this thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

We aim for reviews to be completed within about 2-4 weeks. Please let me know if any of you require some more time. We can also use EditorialBot (our bot) to set automatic reminders if you know you'll be away for a known period of time.

Please feel free to ping me (@srmnitc ) if you have any questions/concerns, thanks again for the submission, and for the reviews

from joss-reviews.

Review checklist for @glycodynamics

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the https://github.com/alchemistry/alchemlyb?
• License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@orbeckst) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
• Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

from joss-reviews.

@srmnitc is there a due date?

from joss-reviews.

It would be ideal to complete the review within about 2-4 weeks. If you need more time, please let me know.

@srmnitc is there a due date?

from joss-reviews.

@srmnitc So sorry but I will have to withdraw as a reviewer, I didn't look at the author list closely when I agreed to review (my bad!) and I have a COI from publishing with one of the authors in the last 4 years

from joss-reviews.

from joss-reviews.

@philbiggin This is the policy https://joss.readthedocs.io/en/latest/reviewer_guidelines.html#joss-conflict-of-interest-policy fyi

from joss-reviews.

from joss-reviews.

@editorialbot remove @mikemhenry from reviewers

from joss-reviews.

@mikemhenry removed from the reviewers list!

from joss-reviews.

@editorialbot remove @philbiggin from reviewers

from joss-reviews.

@philbiggin removed from the reviewers list!

from joss-reviews.

@editorialbot add @ryankzhu as reviewer

from joss-reviews.

@ryankzhu added to the reviewers list!

from joss-reviews.

Review checklist for @ryankzhu

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the https://github.com/alchemistry/alchemlyb?
• License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@orbeckst) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
• Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

from joss-reviews.

@glycodynamics @ryankzhu Thanks again for agreeing to review. Just a short reminder from my side. If you need any help from me, please let me know.

from joss-reviews.

@srmnitc Should I post my comments directly here?

from joss-reviews.

@glycodynamics The best option is to open separate issues directly on the code repository for each issue that you find. In this issue, it would be great to also add the link of this review issue so that they show up here. This way, its easier to keep track. If they are small issues, you could also simply write here.

from joss-reviews.

@srmnitc It is not related to code. The code worked perfectly fine for the test case I tried. My comments are related to the "State of the field" and "Example usage" sections, which I did not check off in my checklist.

• Discussion of similar tools would be helpful for context and it's missing in the manuscript.
• Although documentation is good, providing a single example case with input files would greatly benefit novice users in understanding the process.
• I noticed that the description of errors is missing from the paper, which could be a valuable addition.

For editor only: I noticed some co-authors contributed less code and commits than one who has been just acknowledged. Also, at least one co-author is not listed in the copyright section of the code but is listed as a co-author in the paper. I understand that the contributions of each author are best known to the corresponding author and not raising a flag but pointing it out as there was a "Contribution and authorship" section in the checklist.

from joss-reviews.

@srmnitc I am sorry for the delay. The codes have worked fine in my test case. The documentation is quite detailed and the manuscript reads nicely and concisely to me. I only have two small comments:

• It might be good to have a line (in addition to the badge) in the README to link to the documentation to make it more noticeable.
• I totally agree with @glycodynamics that it would be useful to have a simple example, e.g., a self-contained jupyter notebook, to walk through the main functionalities.

from joss-reviews.

I noticed some co-authors contributed less code and commits than one who has been just acknowledged. Also, at least one co-author is not listed in the copyright section of the code but is listed as a co-author in the paper. I understand that the contributions of each author are best known to the corresponding author and not raising a flag but pointing it out as there was a "Contribution and authorship" section in the checklist.

The PR alchemistry/alchemlyb#328 contains the full history of who decided to be an author and full rationale for all decisions regarding authorship. I am happy to explain in more detail.

from joss-reviews.

@glycodynamics Thanks for the review. I'm glad that you mentioned the free energy analysis tool in SOMD. So analyse_freenrg from siremol is deprecated and their more recent tool for doing free energy analysis is based on alchemlyb https://github.com/OpenBioSim/biosimspace/blob/devel/python/BioSimSpace/FreeEnergy/_relative.py#L1241.

I meant to say estimation of errors, how are they obtained, internally in alchemylib or coming from the external program, such as bootstrapping in MBAR? I may have missed, but I didn't find good documentation of this.

We just use the error estimation function from pymbar, so it might not be too useful to say in the paper that we just use pymbar for error estimation.

from joss-reviews.

@editorialbot generate pdf

from joss-reviews.

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

from joss-reviews.

@glycodynamics and @ryankzhu thanks for the review, and thanks @orbeckst and @xiki-tempula for agreeing to address the issues.

@glycodynamics Thanks again for looking closely at the author list. @orbeckst going through alchemistry/alchemlyb#328 it looks like the author list is justified. However, as @glycodynamics pointed out there are differences between copyright statement on the webpage and the author list. Is there a reason for this difference? Would it make more sense to be consistent?

from joss-reviews.

@glycodynamics could you please take a look again when possible and see if your issues have been addressed?

from joss-reviews.

@orbeckst Thanks so much for the revision.

@srmnitc I read both the tutorial and how authors are now handling the list of developers. I agree that mentioning about the error estimation function is not required unless it was handled somehow differently within the alchemlyb. I am happy with the revision and I do not have any further comments.

from joss-reviews.

@orbeckst Thanks so much for the revision.

@srmnitc I read both the tutorial and how authors are now handling the list of developers. I agree that mentioning about the error estimation function is not required unless it was handled somehow differently within the alchemlyb. I am happy with the revision and I do not have any further comments.

Thanks @glycodynamics. In that case, could you please review your checklist again, and confirm that you recommend this paper for publication? Thanks a lot once again for your review!

from joss-reviews.

@ryankzhu Thanks for your review again. Could you please confirm that you recommend this paper for publication?

from joss-reviews.

@orbeckst I have asked @ryankzhu also by email to confirm that they recommend the publication of this software. After that, I will go ahead with the rest of the steps. Thanks a lot for your patience!

from joss-reviews.

@glycodynamics and @ryankzhu thanks a lot for taking your time for the reviews!

from joss-reviews.

@editorialbot generate pdf

from joss-reviews.

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

from joss-reviews.

@editorialbot check references

from joss-reviews.

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/0021-9991(76)90078-4 is OK
- 10.1002/jcc.20290 is OK
- 10.1021/ct0502864 is OK
- 10.1016/j.softx.2020.100627 is OK
- 10.1021/jp807701h is OK
- 10.1021/acs.jcim.2c01052 is OK
- 10.21105/joss.01831 is OK
- 10.1063/1.1749657 is OK
- 10.1007/s10822-015-9840-9 is OK
- 10.1021/ct2003995 is OK
- 10.1063/1.3607597 is OK
- 10.1063/5.0014475 is OK
- 10.1021/jp102971x is OK
- 10.1016/j.softx.2015.06.001 is OK
- 10.1063/1.2978177 is OK
- 10.1063/1.1638996 is OK
- 10.1063/1.1740409 is OK
- 10.1007/s10822-019-00267-z is OK
- 10.33011/livecoms.2.1.18378 is OK
- 10.1021/acs.jctc.5b00784 is OK
- 10.25080/Majora-92bf1922-00a is OK
- 10.1016/j.cpc.2021.108171 is OK
- 10.1021/acs.jctc.8b00447 is OK
- 10.33011/livecoms.5.1.2067 is OK

MISSING DOIs

- No DOI given, and none found for title: Drug design: structure-and ligand-based approaches
- No DOI given, and none found for title: Scikit-learn: Machine Learning in Python
- No DOI given, and none found for title: API design for machine learning software: experien...
- No DOI given, and none found for title: Simulation techniques for solvation-induced surfac...

INVALID DOIs

- None

from joss-reviews.