lock file already exists causing hang about crds HOT 7 CLOSED

@jaytmiller i've filed another one

from crds.

I think this problem may be not catching / setting up Keyboard interrupt appropriately before multiprocessing pool. Bear with me:

https://stackoverflow.com/questions/11312525/catch-ctrlc-sigint-and-exit-multiprocesses-gracefully-in-python

The lockfile creation did fail, but the Keyboard interrupt is happening inside the multiprocessing pool wait. I saw this during HSI I think and the key was to do a join as is seen farther down in the web link. (So I think maybe any exception occuring inside your code may get stuck. Read the thread and see what you think.)

There is a related mystery though of how the temp lock file was already created. That's bogus.

from crds.

The keyboard interrupt was done on purpose to get out of the hang situation; the interrupt was not causing the hang.

However, to note which I also noted in the other call, I was running this on linux machines all last week without issue. I'll try the run on my mac today, with the latest code, and see if I can reproduce.

from crds.

OK. so last week I was not doing any multiprocessing. I've just confirmed, on my mac, that the issue, with today's conda-dev environment, still exists. To clarify, with CRDS_PATH set to a local folder, no other CRDS environmentals are defined, and the local folder is empty, the following script will cause the hang condition. Run this script in the test data folder I had sent instructions to download earlier; ping me again if you need that info. If I pre-load the local cache using bestrefs, all works fine.

from multiprocessing import Pool
from jwst.pipeline import Spec2Pipeline
from jwst.associations import load_asn
from jwst.associations.lib.update_path import update_path

def spec2run(fname):
    Spec2Pipeline.call(
        fname, 
        config_file='cfgs/calwebb_spec2.cfg', 
        output_dir='level2b'
    )

with open('jw93135-o011_20170721t013702_spec2_002_asn.json') as fp:
    lvl2_asn = load_asn(fp)
update_path(lvl2_asn, 'level2a')

with Pool(processes=4, maxtasksperchild=1) as process_pool:
    process_pool.map(
        spec2run,
        [
            product['members'][0]['expname']
            for product in lvl2_asn['products']
        ]
    )

from crds.

Another fly-in-the-ointment. I have also discovered that no matter what, running the equivalent to the above script in a jupyter notebook causes the hang to occur.

from crds.

Just tried this all out on a linux box; all worked fine. Seems just to be os x.

from crds.

Should be resolved by commit #93a8f716

from crds.