Structuring this repository about test-datasets HOT 13 CLOSED

If I understood well, I think @ewels was thinking about something like:

extra/
reference/
testdata/

But with a different branch for each pipeline

from test-datasets.

I think we can close this issue now - we've started building different branches for the different pipelines and it seems to be working well!

from test-datasets.

I suppose that if files are getting too big, we could try git-LFS

from test-datasets.

And we should list the different branches in the README.md on master

from test-datasets.

Git LFS would help to add files over 100MB, without git LFS, the commit will be blocked for files over 100MB. I believe there is a warning for files between 50MB and 100MB saying the max is 50MB.

from test-datasets.

Regarding git lfs: yes it would help but we should try to keep things as small as possible in all cases anyways. The Sarek way looks fine to me and the ExoSeq pipeline will do the same in the next days once I’m back at work :)

from test-datasets.

Cross linking my PR that I#d like to use as a first test point:
nf-core/exoseq#13

from test-datasets.

One particular problem I see is that the repository might become too big for hosting on GitHub.
Do you think that could be a problem?
https://help.github.com/articles/what-is-my-disk-quota/
Branches sound great - I suppose that should keep things clean

from test-datasets.

The branches idea came from @pditommaso - that way we can have a git submodule for a single test dataset without having to download the test data for every nf-core pipeline. Each branch should be an orphan - master can just have a readme with instructions.

Total repository size could become an issue, yes. I'm not sure how LFS really helps - GitHub still has the 1GB limit, even with LFS..? I guess we just need to keep things slim!

from test-datasets.

Other stuff - would be nice to have in the branch readme a list of all files and their sizes perhaps? Maybe even an md5sum file? Maybe this is overkill..

from test-datasets.

I think it might be overkill for test data.
But we can probably set up a script to pull all branches and generate the README, even with the md5sum ;-)

from test-datasets.

By the way, we set this up for Sarek:
https://github.com/SciLifeLab/Sarek-data

I think you'll all see where the inspiration comes from ;-)

from test-datasets.

Looking forward to this! Would be an amazing resource. I have a good set I've been using for scRNA-seq pipelines.

from test-datasets.