The sections of this tutorial are structured as follows

• Goal
• Pre-requisites
• Clone the repository
• Directory structure of the project
• Overview of the application
• Building, running and testing the application locally.
• Viewing and validating the results
• End

In each section, we'll describe the required steps to take in order to reach the goal.

The purpose of this tutorial is to showcase the instrumention technique (custom instrumentation) for a node.js application based on thrift with Datadog and opentracing. The original code used in this tutorial is based on the official Apache code example available here https://thrift.apache.org/tutorial/nodejs.html

• About 20 minutes
• Node.js (17.9.0), npm (8.5.5), thrift (0.18.1) on Ubuntu 18.04
• Git client
• Docker installed (optional)
• A Datadog account with a valid API key
• Your favorite text editor or IDE (Ex Sublime Text, Atom, vscode...)

[root@pt-instance-2:~/]$ git clone https://github.com/ptabasso2/nodethrift
[root@pt-instance-2:~/]$ cd nodethrift
[root@pt-instance-2:~/nodethrift]$ 

The example below is the structure after having cloned the project.

[root@pt-instance-2:~/nodethrift]$ tree -L 2
.
├── README.md
├── clienttracing.js
├── gen-nodejs
│   ├── Calculator.js
│   ├── SharedService.js
│   ├── sharedtracing_types.js
│   └── tutorialtracing_types.js
├── node_modules
│   ├── @datadog
│   ├── dd-trace
│   ├── opentracing
│   ├── thrift
...
│   ├── through
│   ├── ws
│   └── yocto-queue
├── package-lock.json
├── package.json
├── servertracing.js
├── sharedtracing.thrift
└── tutorialtracing.thrift

The main components for this project can be described as follows:

• A client (clienttracing) and a server (servertracing) communicating with each other using thrift. The client making various calls to the server
  
• In order to allow the communication using thrift we need to use beforehand the compiler it provides to generate the stub layers which are generated from the contract/IDL specified in both tutorialtracing.thrift and sharedtracing.thrift
• Once executed, the corresponding *.js files will be automatically generated and placed into the gen-nodejs directory

First set your API Key:

[root@pt-instance-2:~/nodethrift]$ export DD_API_KEY=<Your api key>

Then let's run the agent. As docker is installed on our environment, we will use a dockerized version of the agent.

But if you wish to have it deployed as a standalone service you will want to follow the instructions as per Datadog Agent installation

By default, the Datadog Agent is enabled in your datadog.yaml file under apm_config with enabled: true and listens for trace data at http://localhost:8126

[root@pt-instance-2:~/nodethrift]$ docker run -d --network app --name dd-agent-dogfood-jmx -v /var/run/docker.sock:/var/run/docker.sock:ro \
-v /proc/:/host/proc/:ro \
-v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
-v /home/pej/conf.d/:/etc/datadog-agent/conf.d \
-p 8126:8126 -p 8125:8125/udp \
-e DD_API_KEY \
-e DD_APM_ENABLED=true \
-e DD_APM_NON_LOCAL_TRAFFIC=true -e DD_PROCESS_AGENT_ENABLED=true -e DD_DOGSTATSD_NON_LOCAL_TRAFFIC="true" -e DD_LOG_LEVEL=debug \
-e DD_LOGS_ENABLED=true \
-e DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=true \
-e DD_CONTAINER_EXCLUDE_LOGS="name:datadog-agent" \
-e SD_JMX_ENABLE=true \
gcr.io/datadoghq/agent:latest-jmx
Unable to find image 'gcr.io/datadoghq/agent:latest-jmx' locally
latest-jmx: Pulling from datadoghq/agent
8dbf11a29570: Pull complete 
Digest: sha256:c7fe7c8d15f259185ab0c60dbfb7f5cbc67d09b5749af0d2fee45cefe2ccb05f
Status: Downloaded newer image for gcr.io/datadoghq/agent:latest-jmx
2d1eec89c2196d298d1e3edf1e9f879c0fc3be593d96f1469cfacc2cacfc18b4

In order to instrument our services, we will also need to use the node.js tracing library (dd-trace-js).

For more details you may want to check the following repository dd-trace-js

To install the node.js tracing library, we will add it by using npm (npm install dd-trace). The opentracing api will also be used together with dd-trace therefore the corresponding package (opentracing) will need to be added as well.

Lastly, we will add the thrift package as this will be used to handle the communication operations between the client and the server.

Build step (optional)

These steps assume that you have a node.js, npm and thrift installed and configured for your environment. This tutorial has been tested with node v17.9.0, npm 8.5.5 and thrift 0.18.1.

The build step essentially consists of running the following commands that call out the thrift compiler. This step is optional as this repository already has them, therefore you can simply reuse them and run the application.

[root@pt-instance-2:~/nodethrift]$ thrift -r --gen js:node tutorialtracing.thrift
[root@pt-instance-2:~/nodethrift]$ thrift -r --gen js:node sharedtracing.thrift 

This will create the gen-nodejs/ and add four new files as per what has been defined in the contract files.

├── gen-nodejs
│   ├── Calculator.js
│   ├── SharedService.js
│   ├── sharedtracing_types.js
│   └── tutorialtracing_types.js

Running and testing the application locally

At this stage we have the Datadog agent running and configured to send telemetry and traces data. We can now move on and launch the server component.

[root@pt-instance-2:~/nodethrift]$ node servertracing.js

Open an another shell and run now the client:

[root@pt-instance-2:~/nodethrift]$ node clienttracing.js
ping()
InvalidOperation InvalidOperation: InvalidOperation
15-10=5
1+1=undefined
Check log: 5

After the sequence of calls, you should be able to see the following output on the server side

[root@pt-instance-2:~/nodethrift]$ node servertracing.js
ping()
calculate( 1 , { num1: 1, num2: 0, op: 4, comment: null } )
calculate( 1 , { num1: 15, num2: 10, op: 2, comment: null } )
add( 1 , 1 )
getStruct( 1 )

This means that the communication takes place and works as expected. You may want to run the command a few times to generate a bit of load.

After having run the client several times and after 20/30 seconds, we can check that our services are being instrumented and that the details are reflected in this trace flamegraph.

What we actually see above is our two services producing spans that are stitched together. That is the result of the context being propagated from the client to the server.

The yellow spans are those of the client and the context of the lower spans are being carried out through the rpc calls and are retrieved on the server end. The spans generated on the server side then becoming child spans.

This process implies changing the signature of the of the original methods so that for each method invocation we add an argument that contains the "carrier" that contains the span context to be accessed.

This means that the IDL (.thrift files) and the code has to be adapted accordingly.

Here is the method before the change

  client.getStruct(1, function(err, message){
    console.log('Check log: ' + message.value);

    //close the connection once we're done
    connection.end();  
  });

Here is the method after the change

  const structCarrier = {};
  tracer.inject(structSpan, opentracing.FORMAT_TEXT_MAP, structCarrier);

  client.getStruct(1, structCarrier, function(err, message){
    console.log('Check log: ' + message.value);

    //close the connection once we're done
    connection.end();

    // End all spans when the connection is closed
    calcSpan.finish();
    structSpan.finish();
    span.finish();
  });

In the latter code snippet, you can see that we've added the carrier that contains the current span context that will be retrieved on the server side as shown in the below example for the getStruct() method

In this case the context is extracted first to retrieve the context associated to the lower level span on the client side. Once accessed we create/start a new span that will be marked as a child span (childOf) and tie the rest of the activity we would like to track to it and finish it.

  getStruct: function(key, headers, result) {
    console.log("getStruct(", key, ")");
    const parentSpanContext = tracer.extract(opentracing.FORMAT_HTTP_HEADERS, headers);
    const span = tracer.startSpan('getStruct', { childOf: parentSpanContext });

    result(null, data[key]);
    span.finish();
  }

This was the code before the changes were applied

  getStruct: function(key, result) {
    console.log("getStruct(", key, ")");
    result(null, data[key]);
  }

Besides we can also visualize the topology representation of this call