Giter VIP home page Giter VIP logo

Comments (23)

mstimberg avatar mstimberg commented on August 16, 2024 2

πŸ‘‹πŸΌ @layik, @kirangadhave, @jibalamy this is the review thread for the paper. All of our communications will happen here from now on.

As a reviewer, the first step is to create a checklist for your review by entering

@editorialbot generate my checklist

as the top of a new comment in this thread.

There are additional guidelines/links in the message at the start of this issue.

Please feel free to ping me (@mstimberg) if you have any questions/concerns.

from joss-reviews.

jibalamy avatar jibalamy commented on August 16, 2024 1

Review checklist for @jibalamy

Conflict of interest

  • I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

General checks

  • Repository: Is the source code for this software available at the https://github.com/alan-turing-institute/setvis?
  • License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
  • Contribution and authorship: Has the submitting author (@layik) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
  • Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
  • Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
  • Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
  • Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

  • Installation: Does installation proceed as outlined in the documentation?
  • Functionality: Have the functional claims of the software been confirmed?
  • Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

  • A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
  • Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
  • Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
  • Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
  • Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

  • Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
  • A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
  • State of the field: Do the authors describe how this software compares to other commonly-used packages?
  • Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
  • References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

from joss-reviews.

layik avatar layik commented on August 16, 2024 1

Thanks @mstimberg will do and update this issue.

from joss-reviews.

ots22 avatar ots22 commented on August 16, 2024 1

Hi @jibalamy,

Thank you for the review so far.

Why the most recent version of Python and Bokeh are not supported?

The answer is because we decided to restrict it to the versions being tested within the CI setup. Happy to relax this condition but must discuss with other authors.

I think we can remove the upper bound on the Python version now; I'll submit a PR. We can control the versions we test in CI independently and we can add 3.12 there too.

I haven't tried bumping the Bokeh version, but something we can check.

I am stuck at step 5, "Making a selection". I am supposed to select rows with missing combinations, but these rows are not contiguous. I tried to select them with control or shift + mouse selection, but I am unable to select non contiguous rows.

I am going to ask for some help here from @ots22 if not will get you clarification from other authors offline and post it here.

It should work by holding shift with a mouse selection, which is just the default behaviour of the box select tool in Bokeh.

Do you have the same problem with multiple selections on this example from the Bokeh gallery?

Additionally, the combination heatmap is difficult to read, because there are column labels but no row labels. Is that normal?

Again, I will ask @ots22 but again will get some answers offline if not.

Each row visually represents a particular combination of present and missing values. There are as many rows as there are distinct combinations and these are shown one above the other as the rows. There isn't a meaningful row label that could be included here - the particular pattern of boxes is everything there is to know.

from joss-reviews.

editorialbot avatar editorialbot commented on August 16, 2024

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

from joss-reviews.

editorialbot avatar editorialbot commented on August 16, 2024

Software report:

github.com/AlDanial/cloc v 1.90  T=0.22 s (201.6 files/s, 519855.1 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
CSV                              5              0              0         107890
Python                          11            653           1338           1546
Jupyter Notebook                10              0           3100            374
TeX                              1             29              0            295
reStructuredText                 9            151            100            157
Markdown                         2             43              0             96
TOML                             1             10              2             69
JSON                             1              0              0             51
YAML                             2             12              3             50
DOS Batch                        1              8              1             26
make                             1              4              7              9
Bourne Shell                     1              4              4              8
-------------------------------------------------------------------------------
SUM:                            45            914           4555         110571
-------------------------------------------------------------------------------

Commit count by author:

   193	Oliver Strickson
    62	Layik Hama
    42	pwochner
    33	ots22
     5	Roy Ruddle
     1	Iain

from joss-reviews.

editorialbot avatar editorialbot commented on August 16, 2024

Paper file info:

πŸ“„ Wordcount for paper.md is 1081

βœ… The paper includes a Statement of need section

from joss-reviews.

editorialbot avatar editorialbot commented on August 16, 2024

License info:

βœ… License found: Apache License 2.0 (Valid open source OSI approved license)

from joss-reviews.

editorialbot avatar editorialbot commented on August 16, 2024
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.2312/eurova.20181110 is OK
- 10.1109/TVCG.2011.186 is OK
- 10.1109/TVCG.2016.2598496 is OK
- 10.1093/bioinformatics/btx364 is OK
- 10.1109/TVCG.2012.252 is OK
- 10.1109/TVCG.2008.144 is OK
- 10.1093/bib/bbab108 is OK
- 10.1016/j.intcom.2007.05.004 is OK
- 10.1109/TVCG.2019.2921544 is OK
- 10.1007/s10072-022-06457-2 is OK
- 10.1109/TVCG.2014.2346248 is OK
- 10.1136/bmjopen-2022-064887 is OK
- 10.1109/TVCG.2020.3030471 is OK
- 10.1109/TVCG.2015.2467051 is OK
- 10.1145/3512934 is OK
- 10.25080/Majora-92bf1922-00a is OK
- 10.5281/zenodo.3509134 is OK
- 10.5518/1150 is OK

MISSING DOIs

- No DOI given, and none found for title: Visual analytics of event data using multiple mini...
- No DOI given, and none found for title: The State-of-the-art of Set Visualization
- 10.1016/j.jclinepi.2020.12.004 may be a valid DOI for title: Methods for depicting overlap in overviews of syst...
- 10.1093/ije/dyx015 may be a valid DOI for title: Data resource profile: Hospital episode statistics...
- No DOI given, and none found for title: HES Data Dictionary - Admitted Patient Care
- No DOI given, and none found for title: Data quality report on comorbidity diagnostic pers...
- No DOI given, and none found for title: Single-cell transcriptome atlas of murine endothel...
- No DOI given, and none found for title: Identification of FN1 as a biological marker for d...
- 10.1109/tvcg.2018.2864812 may be a valid DOI for title: Rulematrix: Visualizing and understanding classifi...
- No DOI given, and none found for title: MOLBIOTOOLS - Molecular Biology Online Apps
- No DOI given, and none found for title: UpSetPlot
- 10.1542/peds.2020-021634 may be a valid DOI for title: Pediatric palliative care programs in US hospitals
- No DOI given, and none found for title: Bokeh: Python library for interactive visualizatio...

INVALID DOIs

- None

from joss-reviews.

editorialbot avatar editorialbot commented on August 16, 2024

πŸ‘‰πŸ“„ Download article proof πŸ“„ View article proof on GitHub πŸ“„ πŸ‘ˆ

from joss-reviews.

layik avatar layik commented on August 16, 2024

Hi

RE one of the general checks where the checkbox states:

If the paper contains original results, results are entirely reproducible by reviewers

Just to be proactive here, I am not sure if I include the relevant files (including notebooks) the reviewers would be able to reproduce the results of the tech-evaluation due to the heavy computation required. I have done the tech-evaluation to be reproducible. I would happily push them to the branch or upload them here if needed. Likewise would happily provide the output of the evaluation (csv files) that was used to generate the plots in the paper if that is deemed sufficient for the check.

from joss-reviews.

mstimberg avatar mstimberg commented on August 16, 2024

@layik Apologies for the late reply on this topic. Yes, please make the files to run the tech-evaluation available in some form. While reviewers might not be able/willing to run the full computation to verify the exact results, they might want to have a look at the benchmarking procedure itself. Ideally, the code would be easily adaptable for them to run a reduced version (e.g. just the first points) of the benchmark.

from joss-reviews.

mstimberg avatar mstimberg commented on August 16, 2024

πŸ‘‹ @jibalamy, @kirangadhave, please let me know if you have questions or need help with anything related to the review. @kirangadhave, please create your checklist even if you are not able to start the actual review right now – at least the first two check boxes should be easy to tick off πŸ™‚ Thanks !

from joss-reviews.

jibalamy avatar jibalamy commented on August 16, 2024

I add some difficulties for installing the module, but it's now Ok.
It is not available on Python 3.12, which is the default version on my Linux distribution (Arch Linux). Then running the tutorial requires Bokeh, but in a old version (it works with 2.4 but not with the latest version). Why the most recent version of Python and Bokeh are not supported? This would simplify the installation of the module.

from joss-reviews.

jibalamy avatar jibalamy commented on August 16, 2024

Then I tried the tutorial 1, but I am stuck at step 5, "Making a selection". I am supposed to select rows with missing combinations, but these rows are not contiguous. I tried to select them with control or shift + mouse selection, but I am unable to select non contiguous rows.
Additionally, the combination heatmap is difficult to read, because there are column labels but no row labels. Is that normal?

from joss-reviews.

layik avatar layik commented on August 16, 2024

Dear @mstimberg and @jibalamy. Please see the files added to the joss branch of the repository. I intend to remove this from the repository once the review is concluded if you do not object to it.
alan-turing-institute/setvis@aef7d6d

As for

Why the most recent version of Python and Bokeh are not supported?

The answer is because we decided to restrict it to the versions being tested within the CI setup. Happy to relax this condition but must discuss with other authors.

I am stuck at step 5, "Making a selection". I am supposed to select rows with missing combinations, but these rows are not contiguous. I tried to select them with control or shift + mouse selection, but I am unable to select non contiguous rows.

I am going to ask for some help here from @ots22 if not will get you clarification from other authors offline and post it here.

Additionally, the combination heatmap is difficult to read, because there are column labels but no row labels. Is that normal?

Again, I will ask @ots22 but again will get some answers offline if not.

Thank you very much for your time.

from joss-reviews.

kirangadhave avatar kirangadhave commented on August 16, 2024

Review checklist for @kirangadhave

Conflict of interest

  • I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

General checks

  • Repository: Is the source code for this software available at the https://github.com/alan-turing-institute/setvis?
  • License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
  • Contribution and authorship: Has the submitting author (@layik) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
  • Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
  • Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
  • Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
  • Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

  • Installation: Does installation proceed as outlined in the documentation?
  • Functionality: Have the functional claims of the software been confirmed?
  • Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

  • A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
  • Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
  • Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
  • Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
  • Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
  • Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

  • Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
  • A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
  • State of the field: Do the authors describe how this software compares to other commonly-used packages?
  • Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
  • References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

from joss-reviews.

kirangadhave avatar kirangadhave commented on August 16, 2024

Python Version

I would recommend supported Python versions (>= 3.8, <3.12) on the installation instructions page in this section.

Selections

Hi @jibalamy,

Thank you for the review so far.

Why the most recent version of Python and Bokeh are not supported?

The answer is because we decided to restrict it to the versions being tested within the CI setup. Happy to relax this condition but must discuss with other authors.

I think we can remove the upper bound on the Python version now; I'll submit a PR. We can control the versions we test in CI independently and we can add 3.12 there too.

I haven't tried bumping the Bokeh version, but something we can check.

I am stuck at step 5, "Making a selection". I am supposed to select rows with missing combinations, but these rows are not contiguous. I tried to select them with control or shift + mouse selection, but I am unable to select non contiguous rows.

I am going to ask for some help here from @ots22 if not will get you clarification from other authors offline and post it here.

It should work by holding shift with a mouse selection, which is just the default behaviour of the box select tool in Bokeh.

Do you have the same problem with multiple selections on this example from the Bokeh gallery?

Additionally, the combination heatmap is difficult to read, because there are column labels but no row labels. Is that normal?

Again, I will ask @ots22 but again will get some answers offline if not.

Each row visually represents a particular combination of present and missing values. There are as many rows as there are distinct combinations and these are shown one above the other as the rows. There isn't a meaningful row label that could be included here - the particular pattern of boxes is everything there is to know.

I am having trouble with selections as well. I can select multiple rows using Shift, but the selection behaves a bit janky. See the video below:

Screen.Recording.2024-07-16.at.3.23.12.PM.mov

When I select a few rows, I cannot select any new rows when pressing the Shift key. However, if I start the new selection and press the Shift key before I release the mouse button to complete the selection, the new selection is added to the previous one. I don't have this issue in the Bokeh example you linked; pressing Shift still allows me to start a new selection.

from joss-reviews.

layik avatar layik commented on August 16, 2024

Hi @kirangadhave, @mstimberg and others. Thank you so much for your reviews so far. Oliver (@ots22) has done a deps review and taken into account all your input. The PR is ready to merge and he will release a minor release ASAP.

alan-turing-institute/setvis#133

Can I please ask you to re-run your selection tests making sure you have bokeh 3.4? You can use the branch in the above PR or you can wait for the release.

I believe this would address most of the issues raised.

As for the tech evaluation, please let me know if you need me to do anything else.

One more time thank you very much for taking time to review the submission.

from joss-reviews.

ots22 avatar ots22 commented on August 16, 2024

Hi @kirangadhave and @mstimberg,

As @layik noted, we have now upgraded to Bokeh 3 and relaxed the required versions of some other packages which we think might help with the selection tool misbehaving.

Please could you upgrade setvis to the latest version, either from GitHub or PyPI (the latter should be version 0.1rc4) and see if this has fixed the problem? You should not need to manually upgrade Bokeh or the other dependencies, although you might want to check that you have ended up with bokeh >= 3.4 if you are reusing an environment.

from joss-reviews.

jibalamy avatar jibalamy commented on August 16, 2024

Hi, I successfully tested the Github version of SetVis with Python 3.12.4 and Bokeh 3.5.1.
I think I now understand how to read the heatmap; indeed it works somehow similarly to Upset, but with color instead of bars. I think you should describe better that adaptation/change; after reading the manuscript, I was thinking that Setvis was producing visualization similar to Upset, and then I was puzzled when I see the demo because the visualization was actually not the same. Possibly, it can be a good idea to display a simple dataset both with Upset and Setvis, for facilitating the understanding and the comparison.

from joss-reviews.

jibalamy avatar jibalamy commented on August 16, 2024

I now have difficulties to understand the various bar charts available in the other tabs. I looked at "Example - import data to visualize sets". In the "Value bar chart", it appears that set cardinalities range from 4 to 32. But in the "Value count histogram", set cardinalities range from 1 to 11... I cannot understand what is represented on these bars. Similarly, I cannot understand the other bar charts (some have decimal cardinality, such as 0.5 or 1.5, which is surprising... probably, 0.5-1.5 actually means 1 ?). Is there some explanations of these bar charts, ideally on the simple example ?

from joss-reviews.

jibalamy avatar jibalamy commented on August 16, 2024

I also obtained several warnings (see below) when running the tests with pytest, is it normal?

tests/test_plots.py::test_IntersectionBarChart
/home/jiba/src/setvis/setvis/membership.py:43: DeprecationWarning: in1d is deprecated. Use np.isin instead.
return list(np.array(universe)[~np.in1d(universe, selection)])

tests/test_plots.py::test_PlotSession
/usr/lib/python3.12/site-packages/tornado/ioloop.py:274: DeprecationWarning: There is no current event loop
loop = asyncio.get_event_loop()

tests/test_plots.py::test_PlotSession
/usr/lib/python3.12/site-packages/pydantic/main.py:1087: PydanticDeprecatedSince20: The dict method is deprecated; use model_dump instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.8/migration/
warnings.warn('The dict method is deprecated; use model_dump instead.', category=PydanticDeprecatedSince20)

tests/test_plots.py::test_SetBarChart
/home/jiba/src/setvis/setvis/plots.py:220: DeprecationWarning: in1d is deprecated. Use np.isin instead.
np.in1d(col_labels, selection.columns, invert=True)

tests/test_plots.py::test_SetCardinalityHistogram
/home/jiba/src/setvis/setvis/plots.py:351: DeprecationWarning: in1d is deprecated. Use np.isin instead.
np.in1d(col_labels, selection.columns, invert=True)

tests/test_plots.py::test_IntersectionBarChart
/home/jiba/src/setvis/setvis/plots.py:501: DeprecationWarning: in1d is deprecated. Use np.isin instead.
np.in1d(

from joss-reviews.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.