How ScraperAPI scaled their data-heavy business with DigitalOcean Managed Databases

ScraperAPI is a proxy solution for web scraping that helps companies collect clean, insightful data from any HTML webpage without being blocked. Whether you’re looking to scrape marketplace pricing, stock market data, job listings, or simply keep an eye on your competitors, you can scrape the data you need with a simple API call.

Founded in 2018, ScraperAPI is used by over 10,000 companies today—including startups, small businesses, and large enterprises. The scale at which ScraperAPI operates is immense. 36 billion web scraping requests are initiated on their platform each month and it scrapes over 14,000 websites per second. Needless to say, databases are a critical part of their business.

Zoltan Bettenbuk, the CTO of ScraperAPI, understands the pain involved in self-managing databases. Before joining ScraperAPI, he had first-hand experience maintaining databases in prior roles. It was a significant burden on business.

“The weight of self-managing databases on you and your team is incredible. You’re responsible for making sure that the data, which is the most valuable asset of a company, is always safe, available, has backups, and has failover. I would honestly never do it again.” — Zoltan Bettenbuk, CTO of ScraperAPI

Ultimately, he sought to find a better solution than burdening the ScraperAPI team with that immense task. He found that solution with DigitalOcean Managed Databases.

Scaling simply with Managed Databases

In his previous job, Zoltan used a bare metal service provider where scaling a database could take upwards of two weeks; the provider had to provision new hardware. That experience cemented his decision to find a cloud option instead. Today, ScraperAPI uses DigitalOcean Managed PostgreSQL and Managed Redis, which have allowed them to scale with ease. If ScraperAPI needs to scale the database to get more storage or compute power, they can do so on-demand and with the click of a button.

“The best thing here is that if I want to scale up to even 2x, I can do it in about a minute.” — Zoltan Bettenbuk, CTO of ScraperAPI

Despite scaling their capabilities significantly, ScraperAPI has maintained a lean engineering team. When Zoltan joined the company, they had two employees. ScraperAPI has since grown to 12 employees, five of which manage the cloud infrastructure—a significantly smaller team size than some of their competitors. DigitalOcean’s simplicity allows for scale, while keeping their ratio of revenue to number of employees high.

Finding security peace-of-mind

Managed Databases offer benefits like automated backups, updates, and automatic failovers. Of these benefits, automatic updates is the one that Zoltan most appreciates. DigitalOcean updates the database if there’s a new version, a hotfix, or a security patch. This provides Zoltan and his team with tremendous peace of mind. Rather than having to monitor the threat landscape, this process is automated and handled by DigitalOcean.

“One of the most valuable aspects of having a managed database is the security and safety that that’s given. We don’t need to deal with updating versions and patches and applying fixes. That type of thing would definitely require a full time person, not only to apply these changes every day, but to constantly monitor these vulnerabilities and apply patches. This is something that would be super painful.” — Zoltan Bettenbuk, CTO of ScraperAPI

Pricing that beats the giants

Bettenbuk mentions that they also chose DigitalOcean for its affordable pricing. ScraperAPI is a network intensive application, currently consuming one petabyte of outbound bandwidth per month and an equally high amount of inbound bandwidth. With DigitalOcean’s low cost bandwidth pricing, they are able to keep their bandwidth costs in check.

“The cost of running ScraperAPI would be 250% more on AWS, as compared to DigitalOcean.” Zoltan Bettenbuk, CTO of ScraperAPI

ScraperAPI is seeing 30-35% year-over-year revenue growth, and has their eyes set on expansion. Alongside their existing business, Zoltan and his team are currently pursuing enterprise customers, focusing on larger clients with structured data needs. Reflecting on their journey thus far, using Managed Databases is a decision they’re glad to have made from the start.

Put your databases on autopilot

Databases are a critical part of your business and there’s a huge amount of overhead in managing them—whether you’re using PostgreSQL, MySQL, Redis, or MongoDB. DigitalOcean’s Managed Databases allows you to offload essential but time-intensive database administration tasks to us. We handle set up, backups, failovers, updates, and more so that you can focus on building great apps.

See whether DigitalOcean Managed Databases is right for your business. View our pricing page to find the solution that fits your budget.