Question

JS Logging in post webhook async function

Posted on June 27, 2024
Logging Ubuntu JavaScript
Asked by Larry Singleton

I have a javascript server that exposes a webhook endpoint that pushes the request body onto a message queue and returns a response code immediately. Asynchronously, the message is then processed off the queue. For some reason, logging output of the async process isn’t always being displayed. After a fresh restart of the App (flush cache), logging occurs for the first 2 events, then stops. When I run the process locally, logging is not interrupted after any amount of events processed. What could be going on here? Any ideas or suggestions would be appreciated.

Submit an answer

This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

Larry Singleton • June 27, 2024

Accepted Answer

So, it turns out that logging wasn’t the issue. The queue was the culprit.

I am using “fastq” and declared the queue globally like this:

const fastq = require("fastq");
const queue = fastq(processMessage, 2);

Notice the “2” - this is supposed to be the number of concurrency threads. However, after the 2nd message, no further processing was occurring (not even logging). Perhaps an “undocumented feature”.

I replaced the above two lines with the below and sanity was returned, allowing >2 messages to be processed.

const queue = require("fastq").promise(processMessage, 2);

#fastq

Matt Arnold • June 27, 2024

This sounds like it could be a caching issue with the logging mechanism. Since local runs don’t show the problem, it might be related to your deployment environment. Try increasing the log verbosity or disabling any log rotation happening after the first few messages.

Bobby Iliev

Site Moderator

• June 27, 2024

Hey!

This is quite interesting, one guess here would be that the logging lib that you are using might be buffering output and not flushing it immediately.

Can you confirm what is the logging library that you are using?

Usually you could try forcing a flush after each log write. For example if you are using winston, you can set { exitOnError: false } and use transports that flush logs promptly:

If false, handled exceptions will not cause process.exit

One other suggestion here could be to make sure that your async functions are properly handling errors and not silently failing, for example you can try to wrap your async code in try/catch blocks and log any errors:

async function processMessage(message) {
  try {
    // Your processing logic here
    console.log('Processing complete:', message);
  } catch (error) {
    console.error('Error processing message:', error);
  }
}