Comments (7)
@jaytmiller i've filed another one
from crds.
I think this problem may be not catching / setting up Keyboard interrupt appropriately before multiprocessing pool. Bear with me:
The lockfile creation did fail, but the Keyboard interrupt is happening inside the multiprocessing pool wait. I saw this during HSI I think and the key was to do a join as is seen farther down in the web link. (So I think maybe any exception occuring inside your code may get stuck. Read the thread and see what you think.)
There is a related mystery though of how the temp lock file was already created. That's bogus.
from crds.
The keyboard interrupt was done on purpose to get out of the hang situation; the interrupt was not causing the hang.
However, to note which I also noted in the other call, I was running this on linux machines all last week without issue. I'll try the run on my mac today, with the latest code, and see if I can reproduce.
from crds.
OK. so last week I was not doing any multiprocessing. I've just confirmed, on my mac, that the issue, with today's conda-dev environment, still exists. To clarify, with CRDS_PATH set to a local folder, no other CRDS environmentals are defined, and the local folder is empty, the following script will cause the hang condition. Run this script in the test data folder I had sent instructions to download earlier; ping me again if you need that info. If I pre-load the local cache using bestrefs, all works fine.
from multiprocessing import Pool
from jwst.pipeline import Spec2Pipeline
from jwst.associations import load_asn
from jwst.associations.lib.update_path import update_path
def spec2run(fname):
Spec2Pipeline.call(
fname,
config_file='cfgs/calwebb_spec2.cfg',
output_dir='level2b'
)
with open('jw93135-o011_20170721t013702_spec2_002_asn.json') as fp:
lvl2_asn = load_asn(fp)
update_path(lvl2_asn, 'level2a')
with Pool(processes=4, maxtasksperchild=1) as process_pool:
process_pool.map(
spec2run,
[
product['members'][0]['expname']
for product in lvl2_asn['products']
]
)
from crds.
Another fly-in-the-ointment. I have also discovered that no matter what, running the equivalent to the above script in a jupyter notebook causes the hang to occur.
from crds.
Just tried this all out on a linux box; all worked fine. Seems just to be os x.
from crds.
Should be resolved by commit #93a8f716
from crds.
Related Issues (20)
- crds issues warning on python 3.12 due to `ast.Str` deprecation
- resolve SonarScan issues
- Improve download options HOT 2
- Retire Python 2 HOT 1
- jwst regression tests are failing because crds cannot find jwst dependency HOT 3
- crds does not complain when it does not find the local crds cache and just exits HOT 1
- Feature Request: Download bestref from AWS S3 buckets
- Mapping._parse_header_selector error under python 3.8 HOT 4
- Add documentation section on crds get_synphot
- Improve documentation to methods
- Move Roman Tests To Unique File
- Failure downloading files that are visible in the web interface HOT 3
- Add Windows support for CRDS client HOT 4
- crds sync warns about unusual CRDS file state 'delivered' HOT 2
- StrConfigItem returns None instead of 'none' if os.environ is modified
- DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')` HOT 3
- Deprecation raised during the install of CRDS
- crds.getreferences does not verify local files HOT 1
- Remove references to ICD-47 in users guide
- `crds list --status` contains summary with what appears to be a warning.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from crds.