ScraperAPI leverages App Platform and DigitalOcean Kubernetes for fast deployments

ScraperAPI, a service designed to make public data collection simple for everyone, has been leveraging DigitalOcean’s infrastructure for years. Initially using DigitalOcean’s Droplets and Managed Databases, ScraperAPI now also leverages App Platform for faster deployments and easy scalability, along with DigitalOcean Kubernetes for more advanced configurations, which allows them to release more features while spending less time managing infrastructure.

A lean team leverages Managed Databases

Founded as a bootstrapped startup, ScraperAPI provides an easy-to-use API to collect publicly available data from the internet on a large scale. Businesses utilize ScraperAPI for e-commerce pricing analytics, search engine data collection, training AI engines, and more.

ScraperAPI has been a heavy utilizer of DigitalOcean Managed Databases for several years. Before joining ScraperAPI, CTO Zoltan Bettenbuk had extensive experience maintaining databases in previous roles, where he found self-managing databases to be a significant burden. Zoltan shared, “The weight of self-managing databases on you and your team is incredible. You’re responsible for making sure that the data, which is the most valuable asset of a company, is always safe, available, has backups, and has failover. I would honestly never do it again.”

Seeking a better solution, ScraperAPI adopted DigitalOcean Managed Databases, utilizing Managed PostgreSQL and Managed Redis. This transition allowed them to scale their database operations effortlessly. “The best thing here is that if I want to scale up to even 2x, I can do it in about a minute,” said Zoltan.

Deploying quickly with App Platform and DigitalOcean Kubernetes

After being acquired by another company, ScraperAPI was looking to bring the efficiency they’d found with Managed Databases to other parts of their tech stack, and began exploring App Platform, DigitalOcean’s Platform as a Service solution, to enhance their deployment processes. Zoltan described the transition: “We found that going to market with a new feature or service is much faster and easier with the App Platform. It really is a matter of a few minutes.” This rapid deployment capability has allowed ScraperAPI to innovate and release new features swiftly.

Speed is one of the main benefits ScraperAPI has realized with App Platform, which enables new features to go live within minutes. Zoltan emphasized the ease of use, saying “It literally takes two to three minutes from the engineer writing the last line of code to the feature being live.”

App Platform has now become an incubator for new services at ScraperAPI. Initially deploying features on the App Platform allows for fast-paced development and testing. Once features scale and require more robust infrastructure, they are transitioned to Managed Kubernetes for more efficient resource management and robust performance. This dual approach has allowed ScraperAPI to release more features while retaining a scalable infrastructure as they grow.

It also enables ScraperAPI to balance the use of premium features for critical workloads, with more cost-effective solutions for less demanding tasks. Because ScraperAPI is a bandwidth-intensive business, they also appreciate how much bandwidth DigitalOcean provides to customers with Droplets and App Platform, estimating “We’d probably be paying three times as much on AWS because we work with such a large amount of data.”

ScraperAPI is experiencing 30-35% year-over-year revenue growth and is expanding its focus to enterprise customers with structured data needs. The combination of DigitalOcean’s Managed Databases, App Platform, and Managed Kubernetes has been integral to their success. They have also been pleased with the support DigitalOcean has provided them as they grow, saying that “Generally speaking, DO support has been available for us basically all the time we needed it. This is usually not the case with other, even larger vendors.”

The team looks forward to future enhancements that DigitalOcean is bringing to App Platform, and will continue to leverage the Platform as a Service solution as a way to quickly deploy new ideas without complexity. Zoltan is also working on new projects, like startup SendStreak, and he is excited to continue to leverage the simplicity and ease of deployment of DigitalOcean’s solutions for these additional use cases.