Arcesium Uses Amazon EBS and Amazon FSR for Faster Database Refreshes
Case Study
Snowflake Secure Data Sharing, the foundation for Arcesium’s product offering, provides data accessibility to Arcesium clients via direct database queries rather than data transfers via API calls.
Arcesium can enable nearly any amount of computing power for virtually any number of users.
With Snowflake, Arcesium retains and increases customer data security.
Arcesium used Amazon EBS and Amazon FSR to create seamless backups and reduce its database refresh time by 59 percent, enabling it to offer multiple staging environments with production data so clients can run various large transformation projects simultaneously.
Arcesium, a global financial technology and professional services firm, recognized that speedier database refreshes would benefit its clients. The company ultimately turned to Amazon Web Services (AWS) to employ Amazon Elastic Block Store (Amazon EBS), a simple-to-use and high-performance block storage service, and Amazon EBS Fast Snapshot Restore (Amazon FSR). As a result, the company achieved rapid scaling, high performance, ample storage, and better resource management capabilities.
In 2014, Arcesium was in the process of spinning off from the D. E. Shaw group, a global investment and technology development firm. At that time, Arcesium considered whether to opt for an on-premises data center or a cloud provider for its computing needs. It ultimately made the decision to use AWS, largely because of the length of time AWS had been in the market and its available suite of dynamic services. “We serve the world’s most sophisticated financial institutions with very complex workflows, and we needed high-performing infrastructure that could scale and meet our data processing needs. We ran benchmarks with various providers,” says Manikantan Madhunapantula, associate director of Arcesium and head of the company’s database engineering group. “We compared on-premises and AWS instance sizes and throughput—input/output operations per second specifically. AWS had a significant advantage over other providers—especially for running our database workloads.”
One of the first challenges Arcesium faced was finding a way to refresh its multiterabyte nonproduction database servers on AWS without using expensive filer software, degrading performance, or incurring a lot of downtime. “Without the capability to refresh with the latest production data, clients would have to invest a lot of time and energy to create data to successfully test their daily, weekly, and monthly data-dependent workflows,” says Manikantan. “Whatever solution we embarked on had to provide a consistent, point-in-time view of multiple databases across the multiple volumes in our environment.”
As Arcesium started its cloud journey, it planned to run database workloads on Amazon Elastic Compute Cloud (Amazon EC2) instances, using Amazon EBS volumes to satisfy its performance requirements. Because Arcesium runs multiterabyte databases, it needed the most powerful instances available from Amazon EC2, a web service that provides secure, resizable compute capacity in the cloud.
Although Amazon EC2 and Amazon EBS offer high performance and ample storage, Arcesium recognized the need to develop an innovative solution to handle an increasing number of refreshes. Arcesium initially made copy-on-write backups for a consistent point-in-time view of databases, then mounted the shadow copy to an Amazon EC2 instance and copied all data over to another disk drive to accomplish refreshes. While this worked fine, Arcesium also wanted to provide the capability to manage multiple environment refreshes over the same weekend to enable clients to run numerous transformative projects simultaneously. That’s when Arcesium reached out to AWS and was introduced to what would become Amazon FSR, as it enables fast restores.
Arcesium wanted to achieve application-consistent snapshots for its databases, so it explored Amazon EBS snapshots using AWS Systems Manager—which provides visibility and control of a customer’s infrastructure on AWS—to discover a way to make a copy-only backup without affecting the backup chain. Arcesium was also able to further streamline the process by reducing the snapshots of all Amazon EBS volumes attached to the instance to just those that were needed. Using Amazon FSR, the company facilitated refreshes of up to 30 multiterabyte environments within the weekend time window while successfully reducing refresh costs by 58 percent and refresh time by 59 percent.
This innovative use of the technologies means that the company can now refresh multiple Amazon EBS databases over the weekend without the addition of filer-based software or other costly solutions. Refreshing client environments used to take the entire weekend, but now Arcesium only needs a few hours. This increase in both speed and performance has resulted in increased satisfaction for Arcesium’s clients, who can now successfully test their data-dependent workloads quickly and efficiently. “The capability to refresh multiple client environments provides clients the capacity to run multiple large transformation projects at the same time—for example, testing new system integration in one staging environment while testing an upgrade of an accounting system in another,” says Manikantan.
In the technology and professional services industry, it’s vital to have a plan in case of a disaster. For extensive databases, full backups take a lot of time, but Arcesium discovered that it could use application-consistent Amazon EBS snapshots and Amazon FSR for database backups as well. Using Amazon EBS snapshots, Arcesium can pause the input/output while taking the backups, and it no longer needs to stop the database. When the company uses Amazon FSR while restoring from the Amazon EBS snapshots, the target incurs a downtime of only 10–15 minutes, versus 15–20 hours for a traditional backup/restore for large databases. If the group wants to go back to a certain point in time, it can use the snapshot it has on hand—and, depending on the size, this can actually be done within 4 hours. “Using application-consistent Amazon EBS snapshots as database backups would be really helpful for any AWS customers running databases on top of Amazon EC2 instances,” says Manikantan.
By using Amazon EBS and Amazon FSR, Arcesium can provide faster database refresh times for more clients without compromising performance or incurring downtime. Arcesium is looking to further explore what it can do on AWS. “We’re looking to create a strategy in which we can quickly move hundreds of terabytes on AWS by using complete copy clones,” says Manikantan. That type of speed and data storage would mean even faster and more efficient services for Arcesium’s global clientele.
“Using application-consistent Amazon EBS snapshots as database backups would be really helpful for any AWS customers running databases on top of Amazon EC2 instances.”
—Manikantan Madhunapantula, associate director and head of the database engineering group, Arcesium
