AWS Lambda & serverless support about apm-agent-nodejs HOT 2 CLOSED

Another potential problem is that flushing does not happen when I expect it to.

In the experiment I have set maxQueueSize: 0. I then call endTransaction(), which I would expect should flush the transaction to APM immediately. But this does not happen before the function/request is fully complete. This may be a problem if the Lambda function terminates before the actual flush happens.

Experiment:

var transaction = apm.startTransaction('myTrans', 'custom');
transaction.result = 200;

apm.endTransaction();
apm._instrumentation._queue._flush();

var now = new Date().getTime();
while(new Date().getTime() < now + 10000){ /* do nothing */ }

Data is not sent to the APM server before the 10 second sleep has completed.

The result is the same with or without apm._instrumentation._queue._flush().

from apm-agent-nodejs.

@StianOvrevage Thanks for creating this issue.

Before diving into the details of AWS Lambda, I thought I'd fest get a few things out of the way:

If the agent is started up before the http (or https) module is required for the first time, it's able to be notified of incoming HTTP requests to the server and automatically start a new transaction for each of them. The moment the http.ServerResponse stream end for a given request, the corresponding transaction is also ended by automatically calling transaction.end(), which in turn adds the transaction to the queue.

As you also mention, you can set maxQueueSize to 0 when configuring the agent. This should be functionally equivalent to manually calling the private apm._instrumentation._queue._flush() function as you do in the code snippet above, and is always recommended instead of relying on a private function which might change without warning.

So on to your questions:

I have come so far as to conclude that code within a Lambda function is executed after nodejs starts the transaction. Therefore calling apm-agent-nodejs within the nodejs function never records/starts the transaction.

When testing locally with serverless offline, the very first transaction is not recorded, but since the server keeps running with APM loaded the consequent requests generate Transactions.

Maybe AWS Lambda is doing something very low level for the first request which means we don't get access to it. It does sound weird though. We'll have to investigate further.

Is there a way to "hook" onto an already ongoing nodejs request/transaction and gather whatever information and timings is visible within the functions own bubble?

No, but you can always start a transaction manually, though we should preferably "fix this" in the agent, so you don't have to deal with this.

Another potential problem is that flushing does not happen when I expect it to. [...] This may be a problem if the Lambda function terminates before the actual flush happens.

Even if effectively disabling the internal queue by setting maxQueueSize to zero, bundling up the payload and sending it off to the APM Server is by its very nature an asynchronous process. There's no way around that.

One solution to this would be to allow you to hook into the agent to be notified when the queue had finished flushing. This would allow you to wait notifying the Lambda function that you're done until you're sure that the APM Server had received the transaction.

So for now there's no officially supported way to achieve this. You could potentially replace your synchronous while-loop with an asynchronous 10 second setTimeout, which should "get around the issue", but that's obviously not a nice solution. Just dug a little more into this and it seems that AWS Lambda by default will wait terminating the function until the Node.js event loop is empty, so this shouldn't be necessary unless you explicitly disable that feature.

from apm-agent-nodejs.