Comments (12)
Can you paste the command you used to produce this output?
from charm4py.
I just ran this snippet in pycharm.
from charm4py.
Aah, I see. When starting Charm4Py scripts, you actually want to run the charmrun.start
program that becomes available when you installed Charm4Py. Here is an example of the command I used to run the above square.py
script using 4 cores on my machine:
python3 -m charmrun.start +p4 ./square.py
The +p4
argument is used to specify that 4 PEs should be used for execution. You can think of PEs as processes that run on a core of your machine, so 4 PEs will use 4 cores.
In the example when charm.pool
is used, 1 PE is reserved as a worker, according to the documentation. Therefore, when using charm.pool
on a machine with n
cores, you may want consider passing n+1
as the value to the +p
argument shown above.
As for running this in PyCharm, you might be able to set up the run configuration such that it will allow you to run Charm4Py programs directly from within the IDE: https://www.jetbrains.com/help/pycharm/creating-and-editing-run-debug-configurations.html
from charm4py.
python3 -m charmrun.start +p4 ./square.py
Is this the only way to run it in order to exploit the benefits of charm4py?
How to do this in a case where there is a main program that launches a child process and this child process needs to run in parallel?
from charm4py.
@ZwFink So I tried the way you mentioned above in my own program and getting this error:-
from charm4py.
How to do this in a case where there is a main program that launches a child process and this child process needs to run in parallel?
Can you explain what you mean by this a little further?
Regarding the timeout error, this issue happens because charmrun
uses ssh
to start processes. For this to work, you need to be enable passwordless ssh for your machine, so that running ssh localhost
doesn't require a password. What operating system are you using?
from charm4py.
I should mention that this is a gap in the Charm4Py documentation and will make sure it gets added.
from charm4py.
@ZwFink Thanks for your reply and clarification.
To explain further on my doubt earlier, what I meant is:-
I have a main script say main.py
This main.py spawns a child process say childprocess.py
and now this child process needs to exploit charm4py to achieve faster execution.
E.g. main.py
child_process = Process(target=(...), args=(...))
child_process.start()
<some other code>
child_process.join()
How to use charm4py in this case?
from charm4py.
Is this an existing codebase where it's not possible/feasible to make main.py
a Charm4Py program and perform the processing of both child/parent processes therein? Doing this would be nice as the Charm++ scheduler can schedule all of the work the program does, which may improve performance. The fork/join model you propose above is trivial to express in Charm4Py.
Otherwise, in the above example can child_process
not simply be a call to invoke a Charm4Py program such as python3 -m charmrun.start +p4 ./square.py
?
from charm4py.
"The fork/join model you propose above is trivial to express in Charm4Py."
Can you please share an example?
from charm4py.
Sure! I transformed your example above into a Charm4Py program:
from charm4py import charm, Chare, Future, Array, Reducer
import time
class ChildProcess(Chare):
def __init__(self, arg, doneFuture):
self.arg = arg
self.doneFuture = doneFuture
@coro
def start(self):
# our "work" here is just to perform a reduction and send it to the
# future, but anything can be done
self.reduce(self.doneFuture, self.thisIndex[0], Reducer.sum)
# if the main process doesn't need the result, we may simply do:
# self.reduce(self.doneFuture)
def main(args):
childProcessFuture = Future()
childProcessArg = 1
# child_process = Process(target=(...), args=(...))
# Here we specify that 20 chares will be created to perform the work.
childProc = Array(ChildProcess, 20, args=[childProcessArg, childProcessFuture])
# alternatively, the constructor of ChildProcess can call doWork which saves some message passing
# this represents the child process creation that does the work
# child_process.start()
childProc.start()
# some other code, in this case just sleep
# <some other code>
time.sleep(3)
# similar to 'child_process.join()' in the example you provided
# child_process.join()
childResult = childProcessFuture.get()
print(f'Child result: {childResult}')
charm.exit()
charm.start(main)
In the above example, the code executed by the childProc.start()
call will happen asynchronously and in parallel with the code executed in <some other code>
, which is what you were achieving above. The work will be distributed among the processors that you use for execution. The call to childProcessFuture.get()
is blocking and will return as soon as the array childProc
has finished processing.
After saving the above in example.py
, the following command executes the above code using 4 processors:
python3 -m charmrun.start +p4 ./example.py
. I can also provide an example using charm.pool
if you think that would be good to see.
from charm4py.
@karankakwani Just following up on this. Have you been able to use something similar to the above to solve your problem?
from charm4py.
Related Issues (20)
- Crash with co-routine suspension within charm.iwait
- Charmrun.start can only run Python programs
- Local message optimization breaks correctness
- Failure to install on macOS Catalina HOT 2
- Local Message Optimization Should be Supported
- Make LocalFuture Available to User Code
- Pip install in new environment fails HOT 4
- Charm4Py needs CKLocal equivalent
- Mac build defaults to x86-64 HOT 1
- Asynchronous GPU Support with HAPI
- Charm++ `smp` does not seem supported HOT 1
- LiveViz Should be Supported
- Update to Cython 3.0 HOT 2
- New release for Charm4Py 1.1
- python environment requirements file
- Use `pip install` not `python setup.py` for installation
- Ray support: is_initialized function
- Charm Pool Examples: Add check for more than 1 PE
- Importing charm4py from project root fails
- Migrate from setuptools to meson-python
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from charm4py.