Comments (2)
I put together something functional using the joblib package a while back that executed plot creation for one collection per thread. It was over a year ago so I used gcpy prototype code in a Jupyter notebook that I had not yet put into the gcpy repo. The non-parallelization part is all very outdated at this point, and I did some extraneous stuff such as storing some input info in a config file. But it did work for parallelization and I was able to create a full suite of collection comparisons within minutes. I am not sure if joblib is the best option, then or now, but here are the files for reference.
Job submit script:
make_benchmark_plots.run.txt
Script called in job:
plot_collections.py.txt
Config file:
config_bench.py.txt
Script with functions used (can now just use gcpy):
v1102e_1hr.py.txt
from gcpy.
Parallel functionality has been added to compare_single_level and compare_zonal_mean, where variables are now plotted and merged into a single PDF in parallel. Parallel functionality has also been added to several benchmark functions, where separate PDFs for different categories of variables are created and populated in parallel. These changes have resulted in a 2-4x speedup of benchmark plotting.
from gcpy.
Related Issues (20)
- GCPy 1.4.1 usages of seaborn-talk is deprecated in older matplotlib versions HOT 2
- Problems creating virtual environment compatible with GCPy 1.4.1 HOT 1
- Plot parallelization off failing in GCPy 1.4.1 HOT 4
- [FEATURE REQUEST] Transport tracer benchmark improvements HOT 2
- Make GCPy a conventional Python package HOT 7
- file_regrid.py creates GCHP restart files with variable name `DELPDRY` instead of `DELP_DRY` HOT 1
- pip install geoschem-gcpy==1.4.2 fails with error due to awscli version HOT 2
- [FEATURE REQUEST] Clean up start/end time usages in benchmark scripts HOT 1
- [FEATURE REQUEST] Transport Tracers benchmark for less than one year HOT 2
- Feature request: Add benchmark results paths and other related info to the the `config` object
- Combine Ref and Dev in the same file for TransportTracers mass conservation tables HOT 1
- Bugs about automatic regridding for cube-sphere when plotting HOT 5
- Feature request: Add a script to scrape timing info from benchmark simulation log files HOT 4
- Can't generate zonal differences for global vs nested model output
- GCPy not properly recognizing 0.125x0.15625 lat-lon grid HOT 4
- Replace whitespace with underscores in benchmark plot/table version labels and file names HOT 1
- RnPbBe budget table benchmark code assumes initial restart is run resolution HOT 1
- Documentation updates for GCPy 1.5.0 HOT 1
- Combine Ref and Dev output in the TransportTracers budget table
- GCPy installation from PyPI (via Pip) does not work due to not finding xesmf version 0.5.1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gcpy.