Is there a way to pass `external_data_dir` as an argument? about descqa HOT 5 CLOSED

oh, sorry @yymao, I should have clarified. Yeah, I think that for now I'll just hardcode in base.py as it's done for NERSC. If you think that we should use a different approach please feel free to reopen and assign the issue to me :)

from descqa.

If this is a feature that would be useful and is not available yet, I'd be happy to add it :)

from descqa.

external_data_dir is supposedly to be a module-wide variable (i.e., all individual tests should share the same value), and that's why it is coded in base.py and not configurable for each test.

Supposedly we can add add a new file descqa/descqa/config.py which stores its value:

__all__ = ["external_data_dir"]
external_data_dir = "/global/cfs/cdirs/lsst/groups/CS/descqa/data"

And then imported this in descqa/descqa/base.py

from . import config
...

class BaseValidationTest(object):
    ...
    external_data_dir = config.external_data_dir 

This way we just need to edit descqa/descqa/config.py at different sites. This approach will be similar to how descqarun and descqaweb do multi-site support; see:

• https://github.com/LSSTDESC/descqa/blob/master/descqarun/config.py
• https://github.com/LSSTDESC/descqa/blob/master/descqaweb/config.py

If we want something more sophisticated then we may want to adapt GCRCatalogs approach, which is certainly more involved:
https://github.com/LSSTDESC/gcr-catalogs/blob/master/GCRCatalogs/register.py#L82

from descqa.

Thanks for your reply @yymao! I see, that makes sense. My suggestion was for external_data_dir to become an attribute of the BaseValidationTest class with a default value, which I think it's still inherited by all tests, but I see that it could also cause some problems down the line. I think I'm happy to leave it as it is then. Thanks again!

from descqa.

@fjaviersanchez -- I saw you close this issue. Do you think the config approach is not needed? How are you planning to edit the external_data_dir then?

from descqa.