Comments (3)
Sorry my mistake.. The additional processes came up when my Database cluster scaled up
from deco.
Do you mean this is still an issue, or were you able to resolve it?
muliprocessing.Pool
is supposed to default to the number of physical cores the machine has, but you can explicitly pass in a number of processes to deco:
@concurrent(processes=36)
def do_work():
...
You may also be hitting some other bottleneck in either deco, or multiprocessing.Pool
. I've never attempted to run deco on anything with 36 cores myself, so seeing someone trying it is awesome!
from deco.
Thank you for getting back to me so quickly. I've had to manually set the number of CPUs. It didn't seem to auto-detect. I can try that again and see if is true or if I made a premature judgement there.
I didn't realize I was pointed to the wrong end point (write vs read) and my Database wasn't scaling up. Since the database server was maxed out at 100%, the other Deco processes were sitting idle. Once I correct that, the cluster scaled up and I was able to use all 36 processors..
Great work btw.. I've never had such an easy time scaling up python.. 👍
from deco.
Related Issues (20)
- Can deco decorates nested functions? HOT 1
- conc_test.py was eror in my python 3 HOT 1
- Can I specify the number of processes to use manually? HOT 1
- Decorators from deco cannot be used on class methods HOT 3
- KeyError with peterbe.com easy example on WinPython 3.6.3.0-64 HOT 4
- Bug with deco function call HOT 2
- wiki problems HOT 1
- Issues With Hanging Processes / Restart HOT 2
- Is nbody.py a Deco example? HOT 1
- Specify fixed number of CPUs? HOT 1
- User problem: where to look for when passing and receiving complex data structures? HOT 2
- Is it a good idea to pass read-only variables in a global like manner to the @concurrent function? HOT 3
- "posonlyargs" missing from arguments HOT 3
- Code slows down without sleep HOT 7
- get() without synchronized returns a tuple? HOT 2
- Multiple for loops HOT 4
- TypeError: cannot pickle '_nrt_python._MemInfo' object when passing a numba Dict to the @concurrent function HOT 2
- error_callback possible for deco?
- Setting and handling timeout HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deco.