generalise to run provided async function on all files? about readdir-cluster HOT 5 CLOSED

I don't believe you can pass a function to the worker, however I'd happily be proven wrong by a PR as this seems a useful addition :-) Upon review we can decide if it makes sense to be part of it or outside of it. Good plan?

from readdir-cluster.

Yep, the documentation mentions JSON, and I can confirm that functions do not survive transit:
https://nodejs.org/api/child_process.html#child_process_event_message

a parsed JSON object or primitive value

I'm playing with some other ideas...

from readdir-cluster.

Okay, so eval() is obviously a non-starter, but I found that I can pass a filePath, and have the worker require() it on the other side. If there is valid JavaScript in the content found at that filePath, and it exports a function, then the worker has no trouble executing it.

I'm not sure this makes sense for your project, and it might even be valuable turning it into a generic cluster-powered iterator. Thanks for the rubber-duck debugging, anyhow. :)

I'll close this unless you really want a PR for this idea.

Side note: you may not necessarily need to use the cluster module, as you don't need port-level load-balancing for this (as you don't even open ports). I wonder if just using the childProcess built-in instead would be faster?

from readdir-cluster.

. If there is valid JavaScript in the content found at that filePath, and it exports a function, then the worker has no trouble executing it.

I was suspecting this, perhaps the worker file path can be provided as an option. Such that the user can copy the worker template and modify the stat action to their desire which would more or less accomplish this as needed. It's a bit hacky, but would work.

I'd be more inclined to just boot up another set of workers with the generated file list at the end or on the fly.

I'm not sure this makes sense for your project, and it might even be valuable turning it into a generic cluster-powered iterator. Thanks for the rubber-duck debugging, anyhow. :)

Hehe, no worries.

from readdir-cluster.

In case you were curious, here's what I came up with so far for loading in callbacks via paths:
https://github.com/jokeyrhyme/child-process-async.js

It's missing error handling and worker clean ups, so it's not production-ready, but the concept is there.

from readdir-cluster.