simonsobs / bbpipe Goto Github PK

pymaster should be required in the setup.py

I get this error when installing BBPipe. I am using python 3.7 and I think the issue is with cwlgen. (I can't install cwlgen with python 3.7). Works fine when I use 3.6.

(myenv) mabitbol@edison11:~/BBPipe> python setup.py install --user
running install
running bdist_egg
running egg_info
writing BBPipe.egg-info/PKG-INFO
writing dependency_links to BBPipe.egg-info/dependency_links.txt
writing entry points to BBPipe.egg-info/entry_points.txt
writing requirements to BBPipe.egg-info/requires.txt
writing top-level names to BBPipe.egg-info/top_level.txt
reading manifest file 'BBPipe.egg-info/SOURCES.txt'
writing manifest file 'BBPipe.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/bbpipe
copying build/lib/bbpipe/pipeline.py -> build/bdist.linux-x86_64/egg/bbpipe
copying build/lib/bbpipe/stage.py -> build/bdist.linux-x86_64/egg/bbpipe
copying build/lib/bbpipe/init.py -> build/bdist.linux-x86_64/egg/bbpipe
copying build/lib/bbpipe/main.py -> build/bdist.linux-x86_64/egg/bbpipe
creating build/bdist.linux-x86_64/egg/bbpipe/sites
copying build/lib/bbpipe/sites/init.py -> build/bdist.linux-x86_64/egg/bbpipe/sites
copying build/lib/bbpipe/sites/local.py -> build/bdist.linux-x86_64/egg/bbpipe/sites
byte-compiling build/bdist.linux-x86_64/egg/bbpipe/pipeline.py to pipeline.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/bbpipe/stage.py to stage.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/bbpipe/init.py to init.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/bbpipe/main.py to main.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/bbpipe/sites/init.py to init.cpython-37.pyc
byte-compiling build/bdist.linux-x86_64/egg/bbpipe/sites/local.py to local.cpython-37.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying BBPipe.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying BBPipe.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying BBPipe.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying BBPipe.egg-info/entry_points.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying BBPipe.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying BBPipe.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
bbpipe.pycache.stage.cpython-37: module references file
creating 'dist/BBPipe-0.0.5-py3.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing BBPipe-0.0.5-py3.7.egg
removing '/global/u2/m/mabitbol/.local/edison/3.6-anaconda-5.2/lib/python3.7/site-packages/BBPipe-0.0.5-py3.7.egg' (and everything under it)
creating /global/u2/m/mabitbol/.local/edison/3.6-anaconda-5.2/lib/python3.7/site-packages/BBPipe-0.0.5-py3.7.egg
Extracting BBPipe-0.0.5-py3.7.egg to /global/u2/m/mabitbol/.local/edison/3.6-anaconda-5.2/lib/python3.7/site-packages
BBPipe 0.0.5 is already the active version in easy-install.pth
Installing bbpipe script to /global/homes/m/mabitbol/.local/edison/3.6-anaconda-5.2/bin

Installed /global/u2/m/mabitbol/.local/edison/3.6-anaconda-5.2/lib/python3.7/site-packages/BBPipe-0.0.5-py3.7.egg
Processing dependencies for BBPipe==0.0.5
Searching for ruamel.yaml==0.13.13
Reading https://pypi.org/simple/ruamel.yaml/
Downloading https://files.pythonhosted.org/packages/d9/2d/f1773b60c7a78a832f6068761167907302927cb6c7279275a0dcee102e05/ruamel.yaml-0.13.13.tar.gz#sha256=7cfd653648a1d4a635ce7ae254b4dc6ec7df931a5a653c79e627547be5dda71b
Best match: ruamel.yaml 0.13.13
Processing ruamel.yaml-0.13.13.tar.gz
Writing /tmp/easy_install-1ik1vvkf/ruamel.yaml-0.13.13/setup.cfg
Running ruamel.yaml-0.13.13/setup.py -q bdist_egg --dist-dir /tmp/easy_install-1ik1vvkf/ruamel.yaml-0.13.13/egg-dist-tmp-lqbvnfbd
sys.argv ['/tmp/easy_install-1ik1vvkf/ruamel.yaml-0.13.13/setup.py', '-q', 'bdist_egg', '--dist-dir', '/tmp/easy_install-1ik1vvkf/ruamel.yaml-0.13.13/egg-dist-tmp-lqbvnfbd']
test compiling test_ruamel_yaml
Warning: 'keywords' should be a list, got type 'NoneType'
Traceback (most recent call last):
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/site-packages/setuptools/sandbox.py", line 154, in save_modules
yield saved
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
yield
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/site-packages/setuptools/sandbox.py", line 250, in run_setup
_execfile(setup_script, ns)
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/site-packages/setuptools/sandbox.py", line 45, in _execfile
exec(code, globals, locals)
File "/tmp/easy_install-1ik1vvkf/ruamel.yaml-0.13.13/setup.py", line 854, in
File "/tmp/easy_install-1ik1vvkf/ruamel.yaml-0.13.13/setup.py", line 843, in main
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/site-packages/setuptools/init.py", line 140, in setup
return distutils.core.setup(**attrs)
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/distutils/core.py", line 108, in setup
setup_distribution = dist = klass(attrs)
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/site-packages/setuptools/dist.py", line 370, in init
k: v for k, v in attrs.items()
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/distutils/dist.py", line 267, in init
getattr(self.metadata, "set" + key)(val)
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/distutils/dist.py", line 1203, in set_keywords
self.keywords = _ensure_list(value, 'keywords')
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/distutils/dist.py", line 40, in _ensure_list
value = list(value)
TypeError: 'NoneType' object is not iterable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "setup.py", line 26, in
install_requires=['pyyaml','parsl<0.6.0','cwlgen']
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/site-packages/setuptools/init.py", line 140, in setup
return distutils.core.setup(**attrs)
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/distutils/core.py", line 148, in setup
dist.run_commands()
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/distutils/dist.py", line 966, in run_commands
self.run_command(cmd)
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/site-packages/setuptools/command/install.py", line 67, in run
self.do_egg_install()
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/site-packages/setuptools/command/install.py", line 117, in do_egg_install
cmd.run()
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 415, in run
self.easy_install(spec, not self.no_deps)
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 657, in easy_install
return self.install_item(None, spec, tmpdir, deps, True)
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 704, in install_item
self.process_distribution(spec, dist, deps)
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 749, in process_distribution
[requirement], self.local_index, self.easy_install
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/site-packages/pkg_resources/init.py", line 777, in resolve
replace_conflicting=replace_conflicting
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/site-packages/pkg_resources/init.py", line 1060, in best_match
return self.obtain(req, installer)
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/site-packages/pkg_resources/init.py", line 1072, in obtain
return installer(requirement)
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 676, in easy_install
return self.install_item(spec, dist.location, tmpdir, deps)
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 702, in install_item
dists = self.install_eggs(spec, download, tmpdir)
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 887, in install_eggs
return self.build_and_install(setup_script, setup_base)
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 1155, in build_and_install
self.run_setup(setup_script, setup_base, args)
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/site-packages/setuptools/command/easy_install.py", line 1141, in run_setup
run_setup(setup_script, args)
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/site-packages/setuptools/sandbox.py", line 253, in run_setup
raise
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/contextlib.py", line 130, in exit
self.gen.throw(type, value, traceback)
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
yield
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/contextlib.py", line 130, in exit
self.gen.throw(type, value, traceback)
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/site-packages/setuptools/sandbox.py", line 166, in save_modules
saved_exc.resume()
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/site-packages/setuptools/sandbox.py", line 141, in resume
six.reraise(type, exc, self._tb)
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/site-packages/setuptools/_vendor/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/site-packages/setuptools/sandbox.py", line 154, in save_modules
yield saved
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/site-packages/setuptools/sandbox.py", line 195, in setup_context
yield
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/site-packages/setuptools/sandbox.py", line 250, in run_setup
_execfile(setup_script, ns)
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/site-packages/setuptools/sandbox.py", line 45, in _execfile
exec(code, globals, locals)
File "/tmp/easy_install-1ik1vvkf/ruamel.yaml-0.13.13/setup.py", line 854, in
File "/tmp/easy_install-1ik1vvkf/ruamel.yaml-0.13.13/setup.py", line 843, in main
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/site-packages/setuptools/init.py", line 140, in setup
return distutils.core.setup(**attrs)
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/distutils/core.py", line 108, in setup
setup_distribution = dist = klass(attrs)
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/site-packages/setuptools/dist.py", line 370, in init
k: v for k, v in attrs.items()
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/distutils/dist.py", line 267, in init
getattr(self.metadata, "set" + key)(val)
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/distutils/dist.py", line 1203, in set_keywords
self.keywords = _ensure_list(value, 'keywords')
File "/global/homes/m/mabitbol/.conda/envs/myenv/lib/python3.7/distutils/dist.py", line 40, in _ensure_list
value = list(value)
TypeError: 'NoneType' object is not iterable

"
you make a "DirFIle" type
that takes no extension
then you can do os.mkdir(self.get_output('chains_output')) and np.savez(self.get_output('chains_output')+'/chains_and_whatever'+timestamp+'.npz',chains=chains)
"

Create the pipeline stage that takes pre-processed split maps and generates all possible auto- and cross-correlations between different bands and splits. The output of this stage should be the collection of all possible cross-correlations as well as the mode-coupling matrix used to estimate the power spectra.

As part of this work we need to define a data format for the power-spectra. I would suggest using the flexible SACC format, but others may have other ideas.

A placeholder for this stage is currently here.

Create the pipeline stage that takes all the power spectra computed by BBPowerSpecter and combines them to produce all relevant null tests. The output of this stage could be either those null spectra and/or the statistics describing whether or not the null tests pass and possibly also some diagnostic plots.

The output format for null spectra should be the same format used by BBPowerSpecter (see #5 ), unless there are good reasons to have something else.

A placeholder for this stage is currently here.

Create the pipeline stage that takes computes the power spectrum covariance matrix. There are different ways to go about it:

1. Assume that the covariance matrix is given as an input (and that this stage is not necessary).
2. Use all the processed maps from simulations, computes their power spectra and use them to estimate the sample covariance matrix.
3. Use an analytical prediction for the covariance matrix from survey parameters.

The second option may be the most likely one. If so, this stage would also be an ideal one to estimate any necessary transfer functions. Alternatively, for more modularity, one could create an additional stage (e.g. BBSimSpectra) between BBPowerSpecter and BBCovFeFe that computes all the simulated power spectra, these are then passed to BBCovFeFe to estimate the covariance matrix, and to another additional stage (e.g. BBTransferFunc) that uses them to estimate a transfer function.

The output format for the covariance matrix should be able to talk well with the format used for power spectra. For this reason I would suggest using SACC too, but this is up for discussion.

A placeholder for this stage is currently here.

1. Do you plan to have any BB (as in "B-mode") specific stuff in this repo?
2. Related: will the actual B-mode pipeline scripts that e.g. calculate a PS or a covmat be in a different repo that imports this package?

If the answers are 1. no and 2. yes, then
(a) I'd like to start using this for lensing and maybe encourage @jcolinhill and @dpole to also think about using it for LAT component separation
and
(b) I have a feature suggestion that I can implement in a PR: have the logger outputs include git commit hashes and/or package versions for specified modules. (Really important for reproducible pipeline runs) This would work as follows: add an option in the main config to specify a set of python modules; for each of these modules, before the stages begin, the script checks if the modules are installed packages or git repos. If the former, note and log the package version number. If the latter, the script requires those repos to have clean working directories and errors out if not. If all specified repos have clean working directories, then note and log the branch name and commit hashes.

The definition of "clean working directories" could include the more restrictive requirement of no untracked files.

Let me know if this sounds like something good to add here, or if this is already an existing feature.

BBPipe is only a preliminary name for this package (as are the names of the pipeline stages, BTW). If you have any ideas for a better, cooler name, write them here!

Whoever creates the first stable pipeline stage should also set up some kind of unit testing framework, and unit tests should be a hard requirement for any PR that gets merged into master afterwards. I've had good experiences with unittest, but there are other options.

Create the pipeline stage that pre-processes all input maps (both data and simulations) and generates data that is ready to be ingested by the power spectrum estimator (this could be e.g. in the form of purified alms).

As part of this work we need to define a data format for the power-spectrum-ready maps (could be FITS alms format, but we may need something more sophisticated if we want to tie a given alm to its window function).

A placeholder for this stage is currently here.

We need to agree on data formats for the input data. A probably incomplete list of data whose format we need to agree on is:

• Maps: are we fine with fits files with either WCS images or healpix tables? Should we put all splits for the same band in the same file? Or create separate files per split? If the former, should all splits be in the same HDU or different HDUs.
• Bandpasses: something better than ASCII files would be desirable. Will we assume one bandpass file per band? Do we need separate bandpasses for diferent splits?
• Beams: same as above
• Transfer functions: before anything, we should probably figure out whether these are inputs or something we need to compute within the pipeline.
• Point source lists: for masking. What is the preferred format for these?

Create the pipeline stage that defines the window function to be used for power spectrum estimation. Minimally, this should include apodizing the raw input mask and including some kind of inverse-variance weighting.

The assumption is that the output format will be the same used for all maps (see #2), but this can be changed if needed.

A placeholder for this stage is currently here.

Can I modify the generate spectra and maps scripts to include a difference between the E and B foreground spectra? I think there should be different alphas in particular. (e.g. alpha_dust_EE, alpha_dust_BB, alpha_sync_EE, alpha_sync_BB)

To start the development of bbpower we'd like to have some simple mock data. Ideally, to separate bugs from errors in the methodology, we'd like these sims to be as simple as possible: Gaussian foregrounds, signal and noise, with constant spectral indices.

Generating O(100-1000) and putting them somewhere at NERSC would be good.

Create the pipeline stage that takes all the multi-frequency power spectra and covariance matrix and computes the likelihood for cosmological (r, A_lens), foreground (e.g. A_sync/dust, beta_sync/dust) and nuisance (e.g. bandpower shifts) parameters.

The output will probably be some parameter chains. The format should be whatever is sort-of standard nowadays (probably whatever cosmoMC uses).

A placeholder for this stage is currently here.

• Prepare skeleton for power-spectrum-based pipeline (maskpreproc, mappreproc, nulltester, powerspecter, covfefe, compsep, tensorlike(?)).
• Create issues for each stage and data format
• Write documentation and README, prepare some slides describing the process to create/modify stages and running

As described in https://www.overleaf.com/5129352367nnppxtnbtjxx we want to study to what level a frequency-dependent polarization angle will affect r in the presence of bandpass uncertainties.

To assess this we need to:

• Generate mock data that looks like SO.
• Extend bbpower to take in EE and EB data.
• Extend bbpower to marginalize over bandpass uncertainties (shifts and possibly gain).
• Extend bbpower to include a frequency-dependent polarization angle in each frequency.
• Use all of the above, together with a model for phi(nu) to quantify the impact on r and sigma(r) of different levels of bandpass uncertainties.

I notice that cori.py does a module load for Python 3.5

# Remember to update this if you are using a different Python version                                                              
                           # on client side. Client and workers should have the same python env.                                                              
                           "overrides": """#SBATCH --constraint=haswell                                                                                       
                           module load python/3.5-anaconda ; source activate parsl_env_3.5"""

, but this repo uses f-strings (supported only from 3.6+). Does this not end up mattering because none of the actual bbpipe code runs on the compute nodes (which I think you call workers)? Why is there an insistence then that the client (login node?) and worker should have the same python version?

Say that I run:

bbpipe test.yml

which has stages stage1 and stage2, and config file config.yml. Currently, it is not obvious to me that the information in test.yml and config.yml is fully saved for future reference. Also, the outputs from the two stages end up in the same directory by default, which makes it difficult to e.g. sync just some of the expensive stages to a different compute system. Here's a proposal for a slight restructuring.

output_dir will be the root directory. Pipeline products will be written to sub-directories in output_dir with the same name as the stages. log_dir does not have to be specified. Instead, it is always written to a file $output_dir/run_{TIME}/log.txt, where TIME is some identifier for the time bbpipe was run. Similarly, test.yml and config.yml are copied into $output_dir/run_{TIME}/. This will save more info about the submission. Let me know what you think and I can submit a PR.