camunda / camunda-8-js-sdk Goto Github PK
View Code? Open in Web Editor NEWThe Camunda 8 JavaScript SDK for Node.js
Home Page: https://camunda.github.io/camunda-8-js-sdk/
License: Apache License 2.0
The Camunda 8 JavaScript SDK for Node.js
Home Page: https://camunda.github.io/camunda-8-js-sdk/
License: Apache License 2.0
A future release of Zeebe will support migrating a running process instance to a newer version of the process model. Simple process migration is targeted for 8.4.
The client API for this has been added in stub form with this PR: camunda/camunda#15199.
Add this API stub to the Zeebe Node client.
On a push to alpha and main branches, we only need to run the publish workflow.
For PRs, create a single parallelised test without the publish step in it.
Something to think about: we can only run one SaaS test at a time.
The Tasklist API is moving to the Zeebe Gateway. See here: camunda/camunda#15622
How this is implemented in the SDK will depend on the API surface area that we expose to developers.
It may be possible to do this with no change to the API surface area.
This will be targeted to post-8.5.
No error should pop up
A cluster is created, multi-tenancy enabled. 2 tenants are created, blue and green.
A script is created:
Process instances are created, and workers execute jobs, but from time to time, an Error: 16 UNAUTHENTICATED: is visible in the log. Even so
, all service tasks are completed.
No solution
Create a cluster, with two tenants red and green
create a client ID / Client Secret
Run the script
Multi-tenancy usage
Log visible:
##### Started/Job RED: 0/0, GREEN: 1/2
------ Green process instance started: [2251799813686010]
>>>> Execute GreenJob PI=[2251799813686010] tenant=[green]
15:42:07.412 | zeebe | [service-task] ERROR: Grpc Stream Error: 16 UNAUTHENTICATED: Expected Identity to provide authorized tenants, see cause for details
##### Started/Job RED: 0/0, GREEN: 2/3
------ Green process instance started: [4503599627371180]
>>>> Execute GreenJob PI=[4503599627371180] tenant=[green]
15:42:12.524 | zeebe | [service-task] ERROR: Grpc Stream Error: 16 UNAUTHENTICATED: Expected Identity to provide authorized tenants, see cause for details
##### Started/Job RED: 0/0, GREEN: 3/4
(node:226) UnhandledPromiseRejectionWarning: Error: 16 UNAUTHENTICATED: Expected Identity to provide authorized tenants, see cause for details
at callErrorFromStatus (/mnt/d/pym/CamundaDrive/Support & Consulting/S-19554 Avertra NodeJS/avertra-test/node_modules/@grpc/grpc-js/build/src/call.js:31:19)
at Object.onReceiveStatus (/mnt/d/pym/CamundaDrive/Support & Consulting/S-19554 Avertra NodeJS/avertra-test/node_modules/@grpc/grpc-js/build/src/client.js:192:76)
at listener.onReceiveStatus.processedStatus (/mnt/d/pym/CamundaDrive/Support & Consulting/S-19554 Avertra NodeJS/avertra-test/node_modules/@grpc/grpc-js/build/src/call-interface.js:78:35)
at Object.onReceiveStatus (/mnt/d/pym/CamundaDrive/Support & Consulting/S-19554 Avertra NodeJS/avertra-test/node_modules/zeebe-node/dist/lib/GrpcClient.js:97:36)
at InterceptingListenerImpl.onReceiveStatus (/mnt/d/pym/CamundaDrive/Support & Consulting/S-19554 Avertra NodeJS/avertra-test/node_modules/@grpc/grpc-js/build/src/call-interface.js:73:23)
at Object.onReceiveStatus (/mnt/d/pym/CamundaDrive/Support & Consulting/S-19554 Avertra NodeJS/avertra-test/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:360:141)
at Object.onReceiveStatus (/mnt/d/pym/CamundaDrive/Support & Consulting/S-19554 Avertra NodeJS/avertra-test/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:323:181)
at process.nextTick (/mnt/d/pym/CamundaDrive/Support & Consulting/S-19554 Avertra NodeJS/avertra-test/node_modules/@grpc/grpc-js/build/src/resolving-call.js:99:78)
at process._tickCallback (internal/process/next_tick.js:61:11)
for call at
at ServiceClientImpl.makeUnaryRequest (/mnt/d/pym/CamundaDrive/Support & Consulting/S-19554 Avertra NodeJS/avertra-test/node_modules/@grpc/grpc-js/build/src/client.js:160:32)
at ServiceClientImpl.<anonymous> (/mnt/d/pym/CamundaDrive/Support & Consulting/S-19554 Avertra NodeJS/avertra-test/node_modules/@grpc/grpc-js/build/src/make-client.js:105:19)
at Promise (/mnt/d/pym/CamundaDrive/Support & Consulting/S-19554 Avertra NodeJS/avertra-test/node_modules/zeebe-node/dist/lib/GrpcClient.js:272:47)
at process._tickCallback (internal/process/next_tick.js:68:7)
(node:226) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:226) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
--
Tests in GitHub CI for Self-Managed are failing.
The cause seems to be a race condition in token refresh. At the moment, the OAuth component caches the token and compares the expiry time to the current time, and requests a new token if the expiry time is greater than or equal to the current time.
If the token expires in 1ms it will be used for a call, but this will probably result in it expiring before it hits the service.
To deal with this, I am adding a new configuration field: CAMUNDA_OAUTH_TOKEN_REFRESH_THRESHOLD_MS
. It defaults to 1000 (1 second).
This represents the lead time to refresh the token. So, by default a cached token will be refreshed 1 second before it expires, and this can be tuned by the user depending on their environment.
The SDK uses the debug
library for debugging. Users can set a DEBUG
environment variable to enable debugging information to help trace issues.
Make all the values normalized and document them.
camunda:tasklist
camunda:oauth
camunda:grpc
camunda:worker
camunda:modeler
camunda:adminconsole
camunda:optimize
camunda:zeebeclient
camunda:operate
The workflow is like this:
All development work should be done via PRs against the alpha
branch.
When a PR is merged to alpha
, the tests are run, then semantic-release
runs to determine if a new release is required. If it is, then an alpha package is published to NPM.
Production releases are accomplished by opening a PR from alpha
to main
.
When a PR is merged into main, semantic-release
runs and if a new package release is required, a package is published to NPM.
At the moment there is some sideways configuration of the grpc channel in the Zeebe client. This means that configuration cannot be overridden via the constructor.
This needs to be threaded down into the components.
The Node.js client provides a BroadcastSignalCommand for broadcasting tenant-aware signals in Zeebe. These commands should support multi-tenancy by exposing an optional tenantId property/method.
The following error codes may be returned:
PERMISSION_DENIED
(code: 7)
when a user attempts to broadcast a signal of a tenant they are not authorized for, when multi-tenancy is enabled.
INVALID_ARGUMENT
(code: 3)
For a provided tenant id, when multi-tenancy is disabled
For a missing tenant id, when multi-tenancy is enabled
For an invalid tenant id (i.e. doesn't match the pre-defined format), when multi-tenancy is enabled.
Java client issue: camunda/camunda#13558
This method should take a more generic signature.
Operate
The following methods work
searchFlownodeInstances
getFlownodeInstance
It looks like methods don't call correct endpoints
searchFlownodeInstances
calls the flownodes/search
but should call the flownode-instances/search
getFlownodeInstance
calls the flownodes/${key}
but should call the flownode-instances/${key}
"@camunda8/operate": "^8.4.0"
The repository needs to be moved to the Camunda GitHub organisation, and all the references and links need to be updated.
The SDK API docs task needs to run in CI and automatically update the docs.
We'll run it on main, and tag the methods to indicate what release they were implemented in.
Create a top-level configuration key for a custom useragent string. Add this to the standard string in a centralized place.
This one is a stretch goal for the release.
The new clients have a clean structure with all interfaces in a Dto
namespace.
The ZeebeGrpcClient has interfaces all over the place. It will be a significant documentation win to coalesce them all into a Dto
namespace.
"Randomly", a test will fail in the suite with a 401 Unauthorised response from an API.
Example:
console.log
Failed to search for process instances for 2251799813685806
at src/zeebe/lib/cancelProcesses.ts:16:12
console.log
HTTPError: Response code 401 (Unauthorized) (request to http://localhost:8081/v1/process-instances/search)
at Request.<anonymous> (/Users/jwulf/workspace/c8-sdk/node_modules/got/dist/source/as-promise/index.js:118:42)
This is probably due to an expired token being used for the request.
To debug this, I'll need to write a lifecycle unit test for the OAuth token expiry strategy.
Similar to #80, the Zeebe worker variable payload may contain int64
number values.
Without a user-provided Dto class to leverage #78, we can't know the variable type - but we can detect an unsafe number value if we parse with lossless-json.
An option could be to specify via configuration how variables with a number value that is unsafe are handled.
If there is no Dto provided, we could default to throwing during the parse operation. This stops the application from operating with bad data.
And we could allow the user to set a default behaviour, "coerce to BigInt" or "coerce to string".
Here are the things that need to be dealt with:
These three mean that the web browser environment requires distinct strategies from the server-side environment.
Creating and maintaining differential strategies for the two environments (client- and server-side) is a significant engineering effort.
The SDK package tracks Camunda Platform 8 minor versioning. Feature releases to support current Platform minor version features result in a patch release of the SDK.
The repository uses []semantic-release
](https://github.com/semantic-release/semantic-release) to create releases. Because we track the Camunda 8 Platform minor version, we treat feature implementation during a minor release cycle as a patch release rather than a minor release.
Creating a commit with a feat
commit message will cause the package version patch release number to increment. To update the minor version, a commit with the type minor
is needed.
A commit with the type release
will trigger a patch release even if there are no features or fixes. This can be used on the alpha
branch to test automation if needed.
Getting an OAuth token to use the Modeler API has some significant differences in ergonomics between Self-Managed and SaaS that have an impact on the design of the SDK.
On SaaS, the env var for an admin console client credential id is CAMUNDA_CONSOLE_CLIENT_ID
.
On SaaS, the env var for a cluster application client credential id is ZEEBE_CLIENT_ID
.
If the user is accessing Modeler on SaaS, we need to use the CAMUNDA_CONSOLE_CLIENT_ID
. We can determine if the application is talking to SaaS by the OAuth URL, which is a known value for SaaS.
If the user is accessing Modeler on Self-Managed (ie: using an OAuth URL other than SaaS), we will use the ZEEBE_CLIENT_ID
to request a token. If Modeler is moved out of the application credential pool on Self-Managed in the future, we will deal with it at that point.
api.cloud.camunda.io
.On SaaS, again detected via the OAuth endpoint URL, we will set the audience. On Self-Managed, we will omit the audience field, unless an explicit CAMUNDA_OAUTH_MODELER_AUDIENCE
value is set.
See here for details: camunda-community-hub/camunda-saas-oauth-nodejs#21.
Remove expiry timer and reimplement as an expiry check on token request.
The following repositories need to be archived:
Packages that need to be deprecated with a notice:
deployProcess is deprecated and has been replaced by deployResource.
This will impact the tests, which use a lot of deployProcess.
DeleteResource is in the gRPC API protocol, and support for deleting deployed Forms is coming.
Add ZBClient.deleteResource and test.
Create a complete integration test scenario using all the things in a coordinated way.
Tests of Modeler API on Self-Managed currently fail due to not being authorized.
This could be because the OAuth token for Modeler on SaaS needs an audience of api.cloud.camunda.io
and Self-Managed, apparently, it should be requested with no audience. See here: https://docs.camunda.io/docs/next/apis-tools/web-modeler-api/authentication/?authentication=self-managed
Write a quickstart for the Camunda docs and raise a PR.
Integration tests are still intermittently failing on authorisation.
Periodically a unit test will fail with 401: UNAUTHORISED.
If I "re-run failed jobs" is will reliably pass on the second run.
My hypothesis is that the API client involved in the test is attempting to make its calls with an expired token.
Token expiry is handled in the SDK in the OAuth component. This component encapsulates retrieving tokens from the token endpoint, caching them in memory and on disk, and providing a token to an API client when the API client wants to make an API call.
The OAuth client should be checking if it has a token in-memory or on-disk (the on-disk caching is for when applications are restarted), then checking if the token is expired or is likely to expire soon (there is a threshold setting that represents "this might expire before the call makes the roundtrip") and either requesting a new token from the endpoint to pass on or passing on the cached token.
Some tests - notably the Tasklist ones - will fail multiple calls when they do fail.
Otherwise, I've noticed that it happens later in the test suite, which leads me to think that it happens when a 300 second validity token expires and there is some race condition or logic error that means it is not correctly refreshed before being passed to the API client.
This is difficult to reproduce reliably.
Maybe it needs some specific test of the token refresh timing logic? Or maybe there is something obvious in the code that I am missing.
Workflow fragment:
Post-Failure:
needs: Build_nightly
if: failure()
runs-on: ubuntu-latest
steps:
- name: Post to a Slack channel
uses: slackapi/[email protected]
with:
channel-id: ${{ secrets.SLACK_CHANNEL_ID }}
slack-message: "Nightly build failed. <https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}|Go to the build.>"
env:
SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
This is a smoke test for the release.
Replace the zeebe-node dependency in the Desktop Modeler with the new SDK, and raise a PR for Desktop Modeler.
npm ci
respects the package-lock.json
file, resulting in deterministic installs.
However, users will install the package using npm i
, which does not.
So, running npm ci
in automated tests is not going to replicate the user's experience.
The lossless Json parser from #78 needs to be used for the convenience method getJSONVariablesforProcess
.
User variables may contain int64
values.
We should allow the user to optionally pass in a class that extends LosslessDto
to be able to get the variable data back with no loss of precision.
As a fall-back (when no user Dto is passed in), we could parse it using lossless-json, and then convert any unsafe numbers into BigInt or string.
Making it a string
would make it consistent with how we handle int64 in the SDK - however, without metadata about the payload typing, we are reduced to examining the actual payload and we can only detect int64 when the value exceeds the range of the JS number
type.
This means that the type of the field would vary depending on the value - which is strange behaviour. This could lead to application errors where string concatenation could occur when arithmetic addition was expected.
Hi ! I know this version is in alpha, but I wanted to submit my issue in case of.
An error is thrown when starting a NestJS project with camunda8/[email protected] or camunda8/[email protected]
Error: Cannot find module 'lib'
Require stack:
- /home/xxx/workspace/projects/edl/poc/camunda/node_modules/.pnpm/@[email protected]/node_modules/@camunda8/sdk/dist/admin/lib/AdminApiClient.js
- /home/xxx/workspace/projects/edl/poc/camunda/node_modules/.pnpm/@[email protected]/node_modules/@camunda8/sdk/dist/admin/index.js
- /home/xxx/workspace/projects/edl/poc/camunda/node_modules/.pnpm/@[email protected]/node_modules/@camunda8/sdk/dist/index.js
- /home/xxx/workspace/projects/edl/poc/camunda/dist/process.service.js
- /home/xxx/workspace/projects/edl/poc/camunda/dist/app.controller.js
- /home/xxx/workspace/projects/edl/poc/camunda/dist/app.module.js
- /home/xxx/workspace/projects/edl/poc/camunda/dist/main.js
All
The 'lib' is found
The 'lib' is not found
the path of 'lib' (or any other require() in the compiled js) seems to target a file in a same folder.
NodeJS 20
NestJS@10 project
PNPM
The ZBClient
component supports a lot of backward-compatible constructor options, including multiple constructor signatures.
This makes modification and maintenance a challenge. While adding multi-tenancy support to the Zeebe client, this is causing a lot of friction and accidental complexity.
For the 8.5 release, I will simplify the constructor signature to be a Partial<ZBClientConfig>
, and then make the configuration hydrator do the following:
Read all environment variables into a config map, then overwrite that map with any explicit configuration passed to the constructor.
This makes the configuration explicit and simple. The zero-conf constructor is default. All configuration from environment variables is applied, then anything that is explicitly passed to a ZBClient constructor overrides it.
Check out the guide here: https://docs.github.com/en/code-security/getting-started/dependabot-quickstart-guide#enabling-dependabot-for-your-repository
Then, for code scanning GitHub Advanced Security. Check out the guide at https://docs.github.com/en/code-security/code-scanning/enabling-code-scanning
Desktop Modeler CODE_SCANNING action for reference as well: https://github.com/camunda/camunda-modeler/blob/develop/.github/workflows/CODE_SCANNING.yml
OAuth component, and Zeebe client (if it doesn't use the OAuth component)
Has the ability to specify the token scope via env var and constructor to support custom identity providers. See: camunda/camunda-modeler#4102
OAuth hardcodes the scopes per component.
The JSON number value passed by the REST API for processInstanceKey (for example) is of type $int64.
This can't be reliably represented by the JavaScript number type, which is a 2^53 range floating point value.
This means that systems that are running for a long time can generate keys that cannot be represented by the SDK.
The gRPC library deals with this by parsing int64 to a string representation. Since we don't do arithmetic on keys, it might as well be a string.
The REST library that I am currently using, however, (got) represents these as number, leading to two issues:
The imprecision issue that will become an L1 in production when a system hits key values that cannot be represented as the JS number type.
The impedance mismatch that ZeebeGrpcClient returns long keys as type string, and the other APIs expect a number type as input; and vice versa.
I'm looking for alternative REST clients that can serialise long integer types as JavaScript string.
This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.
These updates are currently rate-limited. Click on a checkbox below to force their creation now.
@typescript-eslint/eslint-plugin
, @typescript-eslint/parser
)@commitlint/cli
, @commitlint/config-conventional
)@typescript-eslint/eslint-plugin
, @typescript-eslint/parser
)uuid
, @types/uuid
)These updates have all been created already. Click a checkbox below to force a retry/rebase of any.
docker/docker-compose-modeler.yaml
docker/docker-compose-multitenancy.yml
docker/docker-compose.yml
zeebe-extra/docker/docker-compose.yml
camunda/zeebe 8.3.3
.github/workflows/commitlint.yml
actions/checkout v4
actions/setup-node v4
.github/workflows/multitenancy.yml
actions/checkout v4
actions/setup-node v4
.github/workflows/saas.yml
actions/checkout v4
actions/setup-node v4
.github/workflows/singletenant.yml
actions/checkout v3
actions/setup-node v3
.github/workflows/tag-and-publish.yml
actions/checkout v3
actions/setup-node v3
JamesIves/github-pages-deploy-action v4
docker.elastic.co/elasticsearch/elasticsearch 7.17.5
camunda/zeebe 8.4.5
zeebe-extra/.github/workflows/build-docs.yml
actions/checkout v3
actions/setup-node v3
JamesIves/github-pages-deploy-action v4
docker.elastic.co/elasticsearch/elasticsearch 7.17.5
camunda/zeebe 8.3.3
zeebe-extra/.github/workflows/test-camunda-saas-push.yml
actions/checkout v1
zeebe-extra/.github/workflows/test.yml
camunda/zeebe 8.3.3
package.json
@grpc/grpc-js 1.9.7
@grpc/proto-loader 0.7.10
chalk ^2.4.2
console-stamp ^3.0.2
dayjs ^1.8.15
debug ^4.3.4
fast-xml-parser ^4.1.3
got ^11.8.6
lodash.mergewith ^4.6.2
long ^4.0.0
lossless-json ^4.0.1
neon-env ^0.1.3
node-fetch ^2.7.0
promise-retry ^1.1.1
reflect-metadata ^0.2.1
stack-trace 0.0.10
typed-duration ^1.0.12
uuid ^7.0.3
@commitlint/cli ^18.4.3
@commitlint/config-conventional ^18.4.3
@semantic-release/changelog ^6.0.3
@semantic-release/git ^10.0.1
@sitapati/testcontainers ^2.8.1
@types/debug ^4.1.12
@types/jest ^29.5.11
@types/lodash.mergewith ^4.6.9
@types/node ^20.9.4
@types/node-fetch ^2.6.11
@types/promise-retry ^1.1.6
@types/uuid ^9.0.8
@typescript-eslint/eslint-plugin ^6.14.0
@typescript-eslint/parser ^6.14.0
commitizen ^4.3.0
cz-conventional-changelog ^3.3.0
eslint ^8.55.0
eslint-config-prettier ^9.1.0
eslint-plugin-import ^2.29.1
eslint-plugin-prettier ^5.0.1
husky ^8.0.3
jest ^29.7.0
lint-staged ^15.2.0
prettier ^3.1.1
semantic-release ^22.0.12
ts-jest ^29.1.1
tsconfig-paths ^4.2.0
typedoc ^0.25.9
typedoc-plugin-include-example ^1.2.0
typedoc-plugin-missing-exports ^2.2.0
typescript ^5.3.3
zeebe-extra/package.json
@grpc/grpc-js 1.9.7
@grpc/proto-loader 0.7.10
chalk ^2.4.2
console-stamp ^3.0.2
dayjs ^1.8.15
debug ^4.2.0
fast-xml-parser ^4.1.3
fp-ts ^2.5.1
got ^11.8.5
long ^4.0.0
promise-retry ^1.1.1
stack-trace 0.0.10
typed-duration ^1.0.12
uuid ^7.0.3
@camunda8/operate ^8.4.0
@sitapati/testcontainers ^2.8.1
@types/debug 0.0.31
@types/got ^9.6.9
@types/node ^18.19.3
@types/promise-retry ^1.1.3
@types/stack-trace 0.0.33
@types/uuid ^3.4.4
delay ^4.3.0
jest ^27.2.3
jest-environment-node-debug ^2.0.0
typedoc ^0.21.10
node >=16.6.1
In the published package, type information is missing.
This is because the path mapping in the project is not transpiled.
Adding these two fixes it:
https://github.com/nonara/ts-patch
https://github.com/LeDDGroup/typescript-transform-paths
Installation and Configuration docs:
https://docs.camunda.io/docs/next/self-managed/console-deployment/configuration/
Docker Image:
https://registry.camunda.cloud/harbor/projects/28/repositories
This will not happen until after the 8.5 release.
Zeebe
The Zeebe Gateway has a new RPC StreamJobs to open a long-lived stream that essentially "pushes" jobs to the worker.
This obviates polling loops.
Environment variable to enable it:
ZEEBE_CLIENT_WORKER_STREAMENABLED
See also the documentation about the general concept, including how back pressure works and can be implemented for your custom client: https://stage.docs.camunda.io/docs/next/components/concepts/job-workers/#job-streaming
Unit tests can no longer be run without an integration environment. The global setup and teardown should have code in them to skip the Operate and Zeebe client creation if they are not running in an integration test.
At the moment all tests are running on the TypeScript source. The package entry point and any misconfigurations in the transpilation are not tested. This means that a package can be released via automation that doesn't work (See #114).
To make sure that this doesn't happen, we need a test that transpiles the code and then runs a smoke test on it using the package entry point.
This is the official supported-by-Camunda Nodejs SDK for Camunda Platform 8.
The Node.js SDK will not always support all features of Camunda Platform 8 immediately upon their release. Complete API coverage for a platform release will lag behind the platform release.
Prioritisation of implementing features is influenced by customer demand.
A Self-Managed Multi-tenancy stack, running locally or in CI, should function as a test environment for multi-tenancy tests that prove the operation as expected of the SDK in the multi-tenancy environment.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.