Batch manages complex data sets easily with DigitalOcean

“The simplicity is huge. As a small team, everyone has to participate in on-call. In DigitalOcean, even our front-end developers can participate and check if things are working properly. Members don’t have to be specifically trained in all aspects of the platform.”

— Fritz Stauffacher, Batch

Batch manages complex data sets easily with DigitalOcean

Batch is a data observability and replay platform enabling organizations to gain insight into their messaging systems. In addition to enabling developers to view data-streams in real time, the SaaS offering also allows users to replay raw historical data from any location, ensuring companies can recover from any outages, data loss, or bugs. Batch initially began their proof of concept in another major cloud provider, but the company was pre-revenue, and the costs quickly became prohibitive. The complexity of the large cloud provider also made it difficult for the team to move as quickly as they preferred. The organization needed a provider that was reliable, scalable, easy to use, and cost-effective.

A few team members had positive experiences using DigitalOcean for personal projects. They knew the platform could provide the ease of use and scalability they needed, along with the cost savings necessary for that stage of the project. Within five months of beginning the proof of concept, the team migrated their application to DigitalOcean.

Small team, simple solutions

The initial migration to DigitalOcean was straightforward. The team was already using Terraform and Kubernetes, so once they had the basic architecture laid out, they could redeploy on DigitalOcean easily and without outside help. Batch is a five-person team that consists primarily of highly-skilled developers, with one person dedicated to infrastructure. As they began to dig into all that the platform had to offer, the team realized that DigitalOcean’s managed solutions and simple-to-use API enabled them to build faster and remain lean.

“The simplicity is huge. As a small team, everyone has to participate in on-call. In DigitalOcean, even our front-end developers can participate and check if things are working properly. Members don’t have to be specifically trained in all aspects of the platform.” – Fritz Stauffacher, Batch

With DigitalOcean’s API, Batch can quickly test and pivot as they build out their product. The API allows Batch to programmatically manage their resources, perform various actions and automate certain flows. With the API’s automation and ease of use, they can easily spin up or spin down their dev environments or add nodes if needed and ideate on their efforts rapidly.

Simple and scalable architecture

The application is built with a microservices architecture. The main components for Batch live in DigitalOcean’s Managed Kubernetes (DOKS). DOKS is used for anything that isn’t data-dependent or can’t lose state. Any of the primary services they manage go through containerized Kubernetes.

DigitalOcean Load Balancers are used to ingress traffic, and then Collector Droplets dispatch data to Kafka. Kafka pushes messages to the appropriate writer or manager depending on the context of the message. From there, data makes its way to the ElasticSearch cluster, which also interacts with the API and search functionalities. The API also works with Droplets supporting various functions of the Batch platform. Those Droplets are also backed by RabbitMQ and ETCD clusters as well as PostgreSQL databases and TimescaleDB running on Droplets.

,“Managed Kubernetes is amazing. It has exceeded our expectations. We haven’t had any issues with it, and we’ve run really big workloads in it. The Kubernetes product is exceptionally good.” – Fritz Stauffacher, Batch

Security is essential for Batch, and the team uses DigitalOcean’s cloud firewall to easily secure its infrastructure. As an organization responsible for so much customer data, they often work with auditors to ensure everything is secured correctly. By using DigitalOcean’s cloud firewall, it’s easy to use the API to view the security rules in one central location and then modify them if needed. With DigitalOcean’s VPC offering, they can attach everything to the private network and block all public traffic.

Support for every phase of the business

As Batch continues to grow, the team is confident that DigitalOcean can help them accomplish their goals. They have never had a scaling problem and don’t anticipate any in the future. The simplicity of DigitalOcean’s interface, along with its variety of offerings, flexibility, and reliability, make it a perfect platform for startups looking to grow.