v1.22 does not work inside nextflow work directory about multiqc HOT 4 CLOSED

Can you help to reproduce this?

I re-enabled unit tests #2570 and the symlink search test is passing: https://github.com/MultiQC/test-data/blob/c3952131ec5ae7fc169577027a3c65d5326380ea/unit_tests/test_get_filelist.py#L51-L61

from multiqc.

FWIW I can confirm that multiqc can no longer find files for me after the update. This is probably it because I use nextflow which uses a lot of symlinks.

from multiqc.

I did a check of the file content searching work but most of it is just copy pastes from the old codebase. I can't find (by naked eye) any changes that would trigger this.

I tested on a symlinked directory: no issues.

I tested on a symlinked directory inside of a normal directory. No issues.

I would gladly fix any issues in the code I contributed, but I need to be able to reproduce the error in order to do so.

from multiqc.

This is probably it because I use nextflow

Right, turns out the original issue Phil had also had to do with running inside nextflow, Phil shared the workflow to reproduce: https://github.com/ewels/nf-test-multiqc

So it's actually not an issue with symlinks, but rather with the nextflow work folder being ignored in the config defaults: https://github.com/MultiQC/MultiQC/blob/main/multiqc/config_defaults.yaml#L124

fn_ignore_paths:
  - "*/work/??/??????????????????????????????" # Nextflow work directories - always same hash lengths

I'm not really sure why it worked in v1.21 - will check.

UPD:

The reason is the following. Nextflow changes into /work/d3/65b98a3d9c7f67b7f3a2f2b48b5a59 and just runs multiqc as follows:

multiqc .

v1.21 didn't expand the ., so the directory didn't match the exclusion pattern. v1.22 expands . and thus the directory is excluded.

UPD2:

Restored the v1.21 behavior of keeping config.analysis_dir relative, meaning that MultiQC will work as a Nextflow job again. Pushed a "bugfix" release, update accordingly:

pip install multiqc==1.22.1

from multiqc.