Proper way of running with own profile about rnaseq HOT 8 CLOSED

That's nice! But what about the function check_slots I mentioned in #107? Would it be possible to also incorporate this functionality in nextflow? That would be a really nice addition, because many facilities running SGE manages job memory based on the number of chosen slots/cores.

from rnaseq.

Hi @santiagorevale,

Thanks for the issue. It is indeed something I came across myself a while back when thinking about how to set up these configs. I came to the same conclusion - that adding the config as a profile was the best solution.

In short, I don't have any magic recommendations here except that you're welcome to add your config profile to the pipeline if you'd like. We can add you as a listed |contributor / user on the website](https://nf-co.re/about) then too as a bonus!

Note that we have discussed adding different institute's config profiles into the main template in the gitter chat) before. If we did this for you instead, then it would be propagated out to all nf-core pipelines which could be good.

Do you have any thoughts on how we should handle this?

Phil

from rnaseq.

Hi @ewels,

Yes, it would be good being able to add my centre's profile. I'll ask for permission just in case. How should I proceed to do that?

Regarding on how we should be properly handling this, let me familiarise some more with Nextflow's configuration files and I'll be back at you.

I like being able to collaborate in this type of projects. Seems that you and I are involved in the same kind of projects.

Cheers,
Santiago

from rnaseq.

Hi @ewels,

I believe I found a solution for the profiles stuff. By adding the following profile to the main nextflow.config file:

profiles {
  // ...
  // New CUSTOM profile. It allows to include any external profile file.
  custom {
    includeConfig 'conf/base.config'
    includeConfig "${params.custom_profile_file}"
    includeConfig 'conf/igenomes.config'
  }
  // ...
}

and then, you can run the pipeline like this:

nextflow run nf-core/rnaseq -r 1.1 \
  -profile custom \
  --custom_profile_file new_profile.config \
  [...regular options...]

What do you think? I tested it and it worked. Changes should be incorporated in every main nextflow.config file. And then also the new parameter custom_profile_file should be printed among the list of parameters.

By the way, would it be possible to generalise the parameter project by removing the references to UPPMAX?

Cheers,
Santiago

from rnaseq.

Nice 👍 Yes this could work. I'm hoping that we can circumvent this by refactoring how the profiles work - basically always using base.config and igenomes.config, see nf-core/tools#101 and my (now quite outdated) branch.

By the way, would it be possible to generalise the parameter project by removing the references to UPPMAX?

I'm not totally sure what you mean here sorry. Ah - you mean in the summary log? If so then yes, sure.

from rnaseq.

I was worried in the beginning that loosing the None profile would limit the flexibility. But after reading in detail both base.config and igenomes.config I realised that it's just a good default for the pipeline that could easily be overwritten. The only (minor) disadvantage is that it won't be possible to overwrite the current genomes on the list.

On the other hand, notice that adding the custom profile would still be a good addition, because by being able to include the file this way instead of using the -c option, we gain the advantage of also having access to the functions defined in nextflow.config, like check_max, which otherwise you wouldn't been able to see it.

Regarding the UPPMAX thing, yes, that's what I'm talking about. Thanks!

from rnaseq.

The only (minor) disadvantage is that it won't be possible to overwrite the current genomes on the list.

I think it should be possible, just:

params.genomes = null

we gain the advantage of also having access to the functions defined in `nextflow.config``

Yes this is true. However, I spoke to Paolo about check_max last week and we're hoping to build this into core nextflow functionality very soon (see nextflow-io/nextflow#640). I don't think that there are many (any?) others that you'd need.

from rnaseq.

I think this discussion should be going towards core nextflow or maybe nf-core/tools (and there for our template code). Thinking about having this discussion here: nextflow-io/nextflow#640 instead makes more sense than here specifically just for the rnaseq workflow? Thoughts? Similar things apply to #107, which is also rather something to potentially improve in core nextflow than a pipeline here I believe ...?

from rnaseq.