Comments (13)
If I understood well, I think @ewels was thinking about something like:
extra/
reference/
testdata/
But with a different branch for each pipeline
from test-datasets.
I think we can close this issue now - we've started building different branches for the different pipelines and it seems to be working well!
from test-datasets.
I suppose that if files are getting too big, we could try git-LFS
from test-datasets.
And we should list the different branches in the README.md on master
from test-datasets.
Git LFS would help to add files over 100MB, without git LFS, the commit will be blocked for files over 100MB. I believe there is a warning for files between 50MB and 100MB saying the max is 50MB.
from test-datasets.
Regarding git lfs: yes it would help but we should try to keep things as small as possible in all cases anyways. The Sarek way looks fine to me and the ExoSeq pipeline will do the same in the next days once Iām back at work :)
from test-datasets.
Cross linking my PR that I#d like to use as a first test point:
nf-core/exoseq#13
from test-datasets.
One particular problem I see is that the repository might become too big for hosting on GitHub.
Do you think that could be a problem?
https://help.github.com/articles/what-is-my-disk-quota/
Branches sound great - I suppose that should keep things clean
from test-datasets.
The branches idea came from @pditommaso - that way we can have a git submodule for a single test dataset without having to download the test data for every nf-core pipeline. Each branch should be an orphan - master
can just have a readme with instructions.
Total repository size could become an issue, yes. I'm not sure how LFS really helps - GitHub still has the 1GB limit, even with LFS..? I guess we just need to keep things slim!
from test-datasets.
Other stuff - would be nice to have in the branch readme a list of all files and their sizes perhaps? Maybe even an md5sum file? Maybe this is overkill..
from test-datasets.
I think it might be overkill for test data.
But we can probably set up a script to pull all branches and generate the README, even with the md5sum ;-)
from test-datasets.
By the way, we set this up for Sarek:
https://github.com/SciLifeLab/Sarek-data
I think you'll all see where the inspiration comes from ;-)
from test-datasets.
Looking forward to this! Would be an amazing resource. I have a good set I've been using for scRNA-seq pipelines.
from test-datasets.
Related Issues (20)
- Homo sapiens Genome.gtf test-dataset is invalid
- Add new test data for modkit_pileup
- What should we do with `docs/ADD_NEW_DATA.md` and `docs/USE_EXISTING_DATA.md` on pipeline branches?
- Request to add ONT-Amplicon seq for SARS-CoV-2:Wastewater
- CAT - Contig Annotation Tool custom database HOT 1
- add testdata for variantbenchmarking pipeline
- Test data required for testing scimap/spatiallda module
- Need testdata for mcmicro-celesta
- Testdata for xeniumranger modules
- Update test data for bcl_demultiplex subworkflow
- Modules test-data task force master list
- Set up a GitHub bot to automatically post links to new PRs into the #request-review or #test-datasets slack channels.
- Rearrange and rename the `delete_me` folder HOT 2
- Determine what we want to do with tools with very specific test data files
- Organise and document the files that we currently have better, rather than one massive README with single lines of text.
- Generate a page on the website to be able to view the current test data
- Very carefully delete any branches that are unnecessary (especially ones that have already been merged)
- If we are not allowing links to external git files, then we need to deal with all those files HOT 1
- Is there a better way to provide test-data for modules than we currently have, with the two stage process.
- aws s3 links
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
š Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ššš
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ā¤ļø Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from test-datasets.