Ensure that parallel flows are executed independently about lib-bpmn-engine HOT 7 CLOSED

I want to move the technical implementation of the task services out of the BPMN engine. For this purpose, I am considering publishing tasks to REDIS topics and retrieving the results. My concern arises when the documentation states that service-task cannot be paused, but is that really the case? It seems it's not if i correctly understand the code https://github.com/nitram509/lib-bpmn-engine/blob/main/pkg/bpmn_engine/jobs.go#L58C3-L58C3

Regarding the parallel execution of task services, I agree with you; it's not a priority as long as services-tasks can be paused to avoid being blocking. A kind of cooperative multitasking.

Regards,

Ken

from lib-bpmn-engine.

Thanks for your explanation, Ken.

You're right, the documentation might be misleading in the sense of synchronous vs. asynch.
As a matter of fact, calling the explicit .Complete() function, made it possible to implement asychn. behavior,
and this has been intended since the beginning.
So, pausing and resuming os absolutely possible.
With the wording of BPMN, this means: the job activity remains in the state active ;)

I will make a note to improve the documentation there.

from lib-bpmn-engine.

Hi,

on your question, I'm not planning to implement parallel task execution any time soon.
That said, during my current rework, I did realize a bug when handling parallel gateways, and thus it might be, with the next release your use case could be addressed. What I have in mind (but too early to promise yet),
when after a parallel gateway, multiple service tasks (e.g. parallel: one, two, three) are placed,
and the execution of one is not completed, then two could be started, etc.
Would this idea help you for you case?

Regards
Martin

from lib-bpmn-engine.

If I understand correctly, you are suggesting an approach similar to user task handling, but exclusively for service tasks behind a parallel gateway? This would indeed circumvent my issue and remove the roadblock (managing to buffer REDIS messages from an external go-routine and go forward in service task handler on-purpose).

Regarding parallel task execution implementation, what are the major bottlenecks beyond resource sync issues for multi-threaded access ?

from lib-bpmn-engine.

If I understand correctly, you are suggesting an approach similar to user task handling, but exclusively for service tasks behind a parallel gateway? This would indeed circumvent my issue and remove the roadblock (managing to buffer REDIS messages from an external go-routine and go forward in service task handler on-purpose).
👍

Frankly said, I'm not yet fully understand, what goal/problem you want to implement, using the Redis pub/sub.
That sounds to me like you want to execute and wait for a response of long-running ext. tasks, which are integrated via a message queue pattern.
If so, there are some best practices for designing BPMN, documented on Camundas homepage - e.g how to deal with events and interestingly also with timeouts (which are often present in real-world business) -
see https://docs.camunda.io/docs/components/best-practices/modeling/modeling-beyond-the-happy-path/#dealing-with-events

As an alternative, as you already mentioned, you can write a single task handler, which sends a message and receives a message ... which means the task handler will only complete until the expected message is received from Redis.
The polling would happen via engine.RunOrContinue() - which means definitely not inside the handler itself, but rather on the engine level.

I hope that helps somewhat.

Regarding parallel task execution implementation, what are the major bottlenecks beyond resource sync issues for multi-threaded access ?
Besides the technical challenge, of ensuring state modification from/with multiple threads is correct, also the execution's main loop need to be more thread-aware and potentially need to synch execution thread-workers.
E.g. an exclusive gateway is like a "join" operation and needs to wait for all prior execution paths (incl. tasks) to be completed before continuing.
Currently, I consider this too much overhead for the benefit to gain.

I'm in a massive rework, where I use a command pattern, instead of implicitly looping through the workflow.
Also, with a new object "activity" introduced (actually an implementation note from the BPMN spec itself), I'm closer on supporting better and more standard elements.
I consider support for e.g. exception throw + catch and sub-workflows more important than multi-threading.

from lib-bpmn-engine.

On your question, what was the bug?
Here's the test, which covers the scenario...

from lib-bpmn-engine.

Looks, like we can close this question - if not, feel free to re-open.

from lib-bpmn-engine.