Collect & forward logs from self-managed MongoDB to DigitalOcean's managed OpenSearch using Fluent Bit

Introduction

In this tutorial, you will learn how to send logs from Fluent Bit installed on a self-managed MongoDB running on an Ubuntu Droplet to DigitalOcean’s managed OpenSearch database.

OpenSearch is an open-source search and analytics suite that originated as a fork of Elasticsearch and Kibana. It supports real-time data ingestion, advanced querying, and robust security, making it ideal for e-commerce, IT monitoring, and finance applications. With features like SQL query support, machine learning, and alerting, OpenSearch continuously evolves through active community development.

Fluent Bit is a lightweight, open-source log processor and forwarder that collects, processes, and ships log data. Optimized for performance, it’s ideal for resource-constrained environments like containers and edge computing. Fluent Bit efficiently aggregates and forwards logs to various destinations, making it popular for real-time logging and monitoring.

Prerequisites

Before you begin, ensure you have:

• A DigitalOcean Cloud account with access to Managed OpenSearch.
• Basic knowledge of OpenSearch and Fluent Bit.
• Access to the OpenSearch Dashboard.

Installing Fluent Bit

There are multiple platforms where Fluent Bit can be installed. You will be using an Ubuntu Platform here. Run the following command on your Droplet’s Terminal.

curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

You can access the official documentation for more information.

Configuring FluentBit

The Fluent Bit config file is stored at /etc/fluent-bit/fluent-bit.conf by default. You’ll change the fluent-bit.conf to send the logs to OpenSearch.

FluentBit Inputs

Fluent Bit offers a variety of input plugins that enable it to collect log and event data from different sources. Since you will be sending log from logs files, you will be using tail Input plugin.

Update the fluent-bit.conf file as follows:

[INPUT]
   name tail
   Path /var/log/mongodb/mongod.log

You can read more about Input plugins on Fluent Bit’s official Manual.

FluentBit Outputs

Similar to input plugins, Fluent Bit provides an output plugin that sends collected and processed logs to different destinations. Since we are sending logs to Opensearch, let’s make use of the Opensearch Output plugin.

[OUTPUT]
   Name  opensearch
   Match *
   Host opensearch hostname
   port 25060
   HTTP_User doadmin #DO Managed OpenSeach username
   HTTP_Passwd XXXX #Managed OpenSeach Password
   Index mongodb
   tls On
   Suppress_Type_Name On

You can read more about Output plugins on Fluent Bit’s official Manual.

Once the configurations are set up, start Fluent Bit Service.

systemctl enable fluent-bit.service
systemctl start fluent-bit.service
systemctl status fluent-bit.service

Access MongoDB logs in OpenSearch Dashboard

• Navigate to the OpenSearch Dashboard on your DigitalOcean Cloud account.
• Using the hamburger icon, click on Dashboard Management under Management section.
• Click Index patterns and click on Create Index pattern.
• Give the index pattern the same name used in the fluent-bit.conf file.
• Once the index pattern is ready, you can access the MongoDB logs from the Discover tab.

Conclusion

Congratulations! You’ve successfully learned to use Fluent Bit to collect and forward self-hosted MongoDB logs to a DigitalOcean Managed OpenSearch database. We covered configuring Fluent Bit and creating an index pattern in OpenSearch to efficiently monitor and analyze your MongoDB logs.