OTel Auto-instrumentation for BullMQ

License: Apache License 2.0

JavaScript 3.98% TypeScript 96.02%

bullmq nodejs opentelemetry traces contributions-welcome

opentelemetry-instrumentation-bullmq's Introduction

OpenTelemetry Bullmq Instrumentation for Node.js

This module provides automatic tracing instrumentation for BullMQ.

Compatible with OpenTelemetry JS API and SDK 1.0+.

Installation

npm install --save @jenniferplusplus/opentelemetry-instrumentation-bullmq

Supported Versions

[1.90.1, 2.x, 3.x, 4.x, 5.x]

It's likely that the instrumentation would support earlier versions of BullMQ, but I haven't tested it.

Usage

OpenTelemetry Bullmq Instrumentation allows the user to automatically collect trace data from Bullmq jobs and workers and export them to the backend of choice.

To load the instrumentation, specify it in the instrumentations list to registerInstrumentations. There is currently no configuration option.

const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const { registerInstrumentations } = require('@opentelemetry/instrumentation');
const { BullMQInstrumentation } = require('@jenniferplusplus/opentelemetry-instrumentation-bullmq');

const provider = new NodeTracerProvider();
provider.register();

registerInstrumentations({
  instrumentations: [
    new BullMQInstrumentation(),
  ],
});

Emitted Spans

Name	BullMQ method	Description
`{QueueName.JobName} Queue.add`	`Queue.add`	A new job is added to the queue
`{QueueName} Queue.addBulk`	`Queue.addBulk`	New jobs are added to the queue in bulk
`{QueueName.FlowName} FlowProducer.add`	`FlowProducer.add`	A new job flow is added to a queue
`FlowProducer.addBulk`	`FlowProducer.addBulk`	New job flows are added to queues in bulk
`{QueueName.JobName} Job.addJob`	`Job.addJob`	Each individual job added to a queue
`{WorkerName} Worker.run`	`Worker.run`	While a worker is accepting jobs
`{QueueName.JobName} Worker.{WorkerName} #{attempt}`	`Worker.callProcessJob`	Each job execution by a worker's processor function

Useful links

For more information on OpenTelemetry, visit: https://opentelemetry.io/
For more about OpenTelemetry JavaScript: https://github.com/open-telemetry/opentelemetry-js
For help or feedback on this project, open an issue or submit a PR

License

Apache 2.0 - See LICENSE for more information.

Contributing

Contributions are welcome. Feel free to open an issue or submit a PR. I would like to have this package included in opentelemetry-js-contrib at some point. Until then, it lives here.

BullMQ has a hard dependency on Redis, which means that Redis is (for now) a test dependency for the instrumentations. To run the tests, you should have a redis server running on localhost at the default port. If you have docker installed, you can just do docker-compose up and be ready to go.

opentelemetry-instrumentation-bullmq's People

Contributors

Stargazers

Watchers

Forkers

cainaleaouk loancrate screencastify h2atecnologia kevinlillyjv jerrywgray tomdugue alexf4dev afranioce roggervalf zachcr-ws

opentelemetry-instrumentation-bullmq's Issues

Add Support for Worker Metrics and Queue Counts

hey all,

Thank you for making this package :)!!

We have a use case for tracking performance of each worker and each queue's job counts. We would like to scale up/down our instances using queue depth and/or performance of workers.

Would it be possible to add support for these?

Bullmq has metrics option that could be enabled on a worker. They also have a function queue..getJobCounts(). I am not sure how easy or hard it would be to integrate these. I am not very familiar with opentelemetry instrumentation otherwise I would push a PR

Thank you for your help!

Support for bullmq 4.x

Propagate trace context to workers

Hello @jenniferplusplus 👋

I was starting to wonder how I could pass the trace coming from an API to the consumers, and I noticed this package to instrument with Otel our BullMQ.

Would it make sense to instrument this package in a way it detects the trace ID on the add() method, and set the trace context in the worker doing the process? (is it even possible?)

What I had in mind was to use propagation to detect the current trace (https://opentelemetry.io/docs/languages/js/propagation/) and pass it in the data object with a custom property (_otel: { traceparent: <traceparent>}). Then when the worker picks up a job, we could remove the field _otel and instruct the new context to use the traceparent. What do you think?

I would be curious to give it a try!

Why not have a trace for every job?

Very strange setup here - there is a single trace for the entire queue processor, and each job processed is merely a span in that huuuge trace:

Why not have a trace for every job instead? We want to be able to figure out which particular jobs are slow so we can identify them.

No Spans are send

Hi,

i dont get it to work with my nestjs application.

Might need anything?

Tracer code:

  SimpleSpanProcessor,
} from '@opentelemetry/sdk-trace-base';
import { tracing, NodeSDK } from '@opentelemetry/sdk-node';
import * as process from 'process';
import { HttpInstrumentation } from '@opentelemetry/instrumentation-http';
import { ExpressInstrumentation } from '@opentelemetry/instrumentation-express';
import { NestInstrumentation } from '@opentelemetry/instrumentation-nestjs-core';
import { Resource } from '@opentelemetry/resources';
import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';
import { KafkaJsInstrumentation } from 'opentelemetry-instrumentation-kafkajs';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { BullMQInstrumentation } from '@jenniferplusplus/opentelemetry-instrumentation-bullmq';

const exporter = new OTLPTraceExporter({
  url: 'http://locahost:4318/v1/traces',
});

export const otelSDK = new NodeSDK({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]:
      "campaign-publishing-control",
    [SemanticResourceAttributes.SERVICE_NAMESPACE]: process.env.CUSTOMER_ID
  }),
  spanProcessor: new SimpleSpanProcessor(exporter),
  traceExporter: new tracing.ConsoleSpanExporter(),
  instrumentations: [
    new HttpInstrumentation(),
    new ExpressInstrumentation({
      enabled: true
    }),
    new NestInstrumentation({
      enabled: true
    }),
    new KafkaJsInstrumentation(),
    new BullMQInstrumentation({
      enabled: true
    })
  ],
});

// gracefully shut down the SDK on process exit
process.on('SIGTERM', () => {
  otelSDK
    .shutdown()
    .then(
      () => console.log('SDK shut down successfully'),
      (err) => console.log('Error shutting down SDK', err),
    )
    .finally(() => process.exit(0));
});

Node: 16

"@opentelemetry/exporter-metrics-otlp-http": "^0.46.0",
    "@opentelemetry/instrumentation-express": "^0.34.0",
    "@opentelemetry/instrumentation-http": "^0.46.0",
    "@opentelemetry/instrumentation-nestjs-core": "^0.33.3",
    "@opentelemetry/resources": "^1.19.0",
    "@opentelemetry/sdk-node": "^0.46.0",
    "@opentelemetry/sdk-trace-base": "^1.19.0",
    "@opentelemetry/semantic-conventions": "^1.19.0",
    ```

jenniferplusplus / opentelemetry-instrumentation-bullmq Goto Github PK