Run a test on all modules which have been modified in that push / PR, all in parallel in separate jobs about modules HOT 16 CLOSED

If someone has an idea for it sure! I don't think the maintenance of the workflow files is enough to motivate me to write one workflow to rule them all.

Could just cheat and check how many jobs it would be and run pytest without the --tag flag if it's lots, so that it runs everything?

I also considered making a workflow that will run all of the on any releases/ pushes to master.

from modules.

This may or may not be worth it. See https://github.com/nf-core/modules/runs/1450514231?check_suite_focus=true

Mainly because there won't be a workflow for each individual software module anymore. It'll just be one big workflow software that will trigger whenever those files are changed, but it may get broken when there's a change to multiple software packages.

While I'm sure https://github.com/nf-core/modules/blob/master/.github/workflows/docker.yml will work to trigger the pipeline I don't know if it's the best way to do this for the modules even though it's slick.

from modules.

There's also another limitation: The matrix strategy only supports up to 256 jobs, while there can be unlimited workflow files.

(It could be debated, it it really makes sense to run everything in a separate job, since most of the CI time is probably spent setting up the environment to then run a test for 2 seconds)

If we stick with the one-workflow-per-module approach, we could maybe create a pytest-workflow github action to reduce redundant code.

If we go for one-workflow-per-repo, then the way this is done in the docker action seems like the way to go. In fact, I don't see why this wouldn't work with multiple changed modules. What is your concern here?

from modules.

Ooo, I'll definitely make the github action at least. We could also have a nextflow install action if we wanted too. I think that's the best way to cut down on the boilerplate.

For one-workflow-per-repo, sorry if I wasn't clear. With one-workflow-per-module, we get

With one workflow it'll just say Test software and you'll have to sort through all the recent tests, and they won't have a descriptive name like fastqc.

from modules.

ok, I see your point. With one-workflow-per-repo, this would have to be replaced with some nice logging to the test console.
But yes -- it's not as clear as with one-workflow-per-module.

from modules.

Yeah, sadly it's a limitation of GitHub actions, but I don't think it's too bad, I think we can cut it down and it should be easy to find replace all.

I'm perusing the github actions docs and found some useful things:

Creating a workflow template
So when a new module is added it's just a click of a button.

I'm also thinking we could do https://docs.github.com/en/free-pro-team@latest/github/creating-cloning-and-archiving-repositories/about-code-owners for modules, but more importantly subworkflows.(This could also be added to the configs repo)

We could also just use the nextflow docker container with https://docs.github.com/en/free-pro-team@latest/actions/learn-github-actions/finding-and-customizing-actions#referencing-a-container-on-docker-hub instead of creating a nextflow action.

I'm just full of yak shaving ideas here, but we could also test against multiple nextflow versions, with a test matrix.

from modules.

Creating a workflow template
So when a new module is added it's just a click of a button.

👍
Although at some point we would probably have nf-core module create to set-up the rest of the boilerplate as well?

We could also just use the nextflow docker container with https://docs.github.com/en/free-pro-team@latest/actions/learn-github-actions/finding-and-customizing-actions#referencing-a-container-on-docker-hub instead of creating a nextflow action.

Or create a docker-container-based action that used a mulled biocontainer with both nextflow and pytest-workflow and only needs to be provided with the path test.yml ;)

That being said, having a "setup nextflow" action will definitely be useful also outside this repo.

I'm also thinking we could do https://docs.github.com/en/free-pro-team@latest/github/creating-cloning-and-archiving-repositories/about-code-owners for modules, but more importantly subworkflows.(This could also be added to the configs repo)

I also like this idea. I know from bioconda, that they have the maintainer information stored in the meta.yaml files and bots take care of their notification. Codeowners seems more straight-forward!
Maybe create a separate issue for that?

from modules.

Definately a nf-core module create this would just be something to use while we wait for them to continue to mature before adding the nf-core tools stuff.

Yeah I thought about that too. I think that's probably better practice for plain reproducibility, but the github actions might make it easier for us to test with a matrix and then test other options. I'll get one started tomorrow hopefully.

See #86 now!

from modules.

Ooo, I'll definitely make the github action at least. We could also have a nextflow install action if we wanted too. I think that's the best way to cut down on the boilerplate.

@maxibor started on this so we already have https://github.com/nf-core/gh-actions-lint (though we never got as far as actually using it in the template yet). Could be good to kick it over the finish line.

from modules.

@ewels Sorry I updated @grst in slack.

I looked at creating a setup nextflow/pytest-workflow action, but decided against it for now. If we come up with more features maybe later, but we're not going to save any lines with them currently, and I don't see them saving us from any updates because the version would still have to be bumped across pipelines. Again if you think of other nice features for them to take care of sure, but nextflow does a pretty good job of handling the install itself.

Personally I think nf-core/tools#797 would better solve the nf-core linting, and make it easier to setup!

from modules.

Sounds good! Though as @grst said, I'm not too fussed about these things as once this all settles down a bit more (soon) we will start building automation / templates to generate these workflow files anyway. And in fact, this particular workflow will only ever be in this repo, right?

Small note regarding some of the discussion above about running in parallel in a matrix. I agree that the tear up / tear down times will probably mean that it doesn't make a huge benefit to run time. My original thoughts were more that it's annoying to have to create and maintain an actions workflow file each time we make a module. I'd rather that we have one file that runs tests according to which files are edited in a push / commit. I'd be quite happy for these tests to run in series, as most times this should only be one or two modules, having to dig through logs for hundreds of modules should be a rare occurrence..

Phil

from modules.

...though yeah, maybe writing it so that it runs in series actually isn't that easy 🤔 Not sure if it's possible to spawn steps dynamically like that, I've only done it with jobs.

Could just cheat and check how many jobs it would be and run pytest without the --tag flag if it's lots, so that it runs everything?

from modules.

Alright @ewels you win, after doing #119 this needs to be resolved. I found https://docs.github.com/en/actions/creating-actions/creating-a-composite-run-steps-action and I'll update if I get something going.

from modules.

Haha, that's a lot of copying and pasting! I've been there 😅

Note that the docs you've linked to are for creating a GitHub Actions action. I think this is probably overkill, and I think we can do everything in a GitHub Actions workflow (I hope I'm getting the nomenclature right here). In other words, just some .github/workflow YAML files and nothing fancier.

You can see an example of a workflow job output dynamically creating a matrix in the subsequent job in the nf-core/tools pipeline sync workflow.

I think it should be possible to do some git magic to find which paths have been changed, then this needs to be smushed into a JSON object so we can pass an array and use fromJson() in the next job matrix.

I always find this stuff very fiddly. Someone might have already made a GitHub action to simplify some of this? (eg. this one can return the changed paths for example - then we'd just need to filter that to make it tags and deduplicate).

Phil

from modules.

@ewels These are all great thoughts. I just had them about 10 hours ago 😜

https://github.com/nf-core/modules/tree/composite-runs

I made a composite action because I'm hoping to ge this working in the workflows also(Let me know if you have any ideas on how to fix my issue with the bin dir). I'd love to move the composite action over to nf-core.

I used https://github.com/dorny/paths-filter for finding what's changed. It's pretty slick, and I moved the filters into their own file. That will need to be updated with every new module for now, but open to automating it if someone figures that out.

The only thing I have left is to check if no files are modified, I just got tired last night and couldn't find the right syntax.

I also need to figure out how to handle passing multiple tags to pytest workflow elegantly in the composite step.

You also may love this that I found because I was tired of pushing stuff up last night, https://github.com/nektos/act . It didn't work though because of a broken symlink in a deprecated module I think.

from modules.

Great minds think alike! 🎉 😅 Sounds great - will take a look when I get a chance. The act thing looks super useful! (and easy to use, unlike all other run-actions-locally stuff)

from modules.