Mutual of Enumclaw Insurance Company has been supporting communities with insurance for over 100 years. Mutual of Enumclaw provides commercial and personal property/casualty insurance in six western states (Washington, Oregon, Idaho, Utah, Arizona, and Montana). Through subsidiaries Mutual of Enumclaw Insurance Company and Enumclaw Property and Casualty Insurance, the company writes a variety of lines, including auto, homeowners, liability, and umbrella insurance.

Driving Growth with Microservices

Mutual of Enumclaw engineers were tasked with expanding their solution to more states. As the company is 100+ years old, some of the systems were more up to date than others. The company’s primary policy systems ran on an AS/400 and a Z-mainframe, both leveraging forms of Cobol and both fundamental to how Mutual of Enumclaw does business.

Full Visibility All the Time

For Mutual of Enumclaw, it was extremely important to quickly discover and eliminate code vulnerabilities, as well as have full visibility into its cloud environment at all times. Mutual of Enumclaw wanted to control traffic through access restrictions and run a dynamic, cloud-native environment where workloads could run anywhere.

“Mutual of Enumclaw wanted to control traffic through access restrictions and run a dynamic, cloud-native environment where workloads could run anywhere.”

Mutual of Enumclaw currently runs its containerized infrastructure on AWS Amazon Elastic Container Service (ECS) (a multi-AZ deployment in the US regions) that integrates with a serverless environment, running on AWS Lambda. In addition, the team is utilizing AWS ancillary services such as RDS, S3, IAM, KMS, DynamoDB, and API Gateway.

Mutual of Enumclaw chose AWS for its customer support and its extensive range of products to make cloud computing easier and safer. “As an insurance company that uses the cloud, it makes sense to use a cloud provider everyone knows and is familiar with,” Nikody Keating, Chief Architect, said.

Using a modern application approach, Mutual of Enumclaw was able to provide a scalable solution to more customers, while maintaining a rapid pace of development. Under pressure to deliver, the engineers knew that the additional infrastructure overhead would incur more cost and would take time to deploy. Continuing to build on-premises would slow down the company’s efforts. Thus, Mutual of Enumclaw decided to build an architecture based on microservices with Amazon Elastic Container Service clusters and AWS Lambda.

From AS/400 to Modern Applications Running on ECS and AWS Lambda

Mutual of Enumclaw decided to work on transforming their policy system. The business went from large queues of batch processes to real-time and from running on an AS/400 and Z-Series Mainframe to running a modern application using AWS Lambda and ECS.

Mutual of Enumclaw saw the transition to AWS Lambda and ECS as an opportunity to be more business-focused, greatly increase the productivity of their developers, and significantly reduce their infrastructure costs.

Mutual of Enumclaw initially set a small team of architects and developers to focus on the project. The first task was to design a scalable and modern architecture, based on services such as AWS Lambda, ECS, API Gateway, SQS, SNS, S3, DynamoDB, and more. The new application had to connect to the legacy application via existing APIs.

After the initial design, Mutual of Enumclaw started the implementation. The developers built new services one-by-one, making sure that each piece of the service was working correctly before moving on to the next piece. At that point, they started to encounter observability challenges since there were so many moving pieces in the architecture. These challenges affected the developers’ velocity. As a result, logging and ad hoc querying capabilities in highly distributed, containerized production and development environments were needed. 

“At that point, Mutual of Enumclaw started to encounter observability challenges since there were so many moving pieces in the architecture. These challenges affected the developers’ velocity.”

Solution – Distributed Tracing on AWS

As Mutual of Enumclaw started looking at its transformation, the company knew it didn’t want to do what it had always done. The developers wanted a system that would self-document, provide visibility into what was going on, and required less effort to maintain and support. This was where Epsagon fit in. Epsagon’s visibility capabilities are not intrusive to the development experience and provide value at all stages of the product life cycle. During initial development with Epsagon, Mutual of Enumclaw could see messages and logs for a specific processing path, even if it gets complex (for example, Lambda to SNS to SQS to Lambda).

“Epsagon’s visibility capabilities are not intrusive to the development experience and provide value at all stages of the product life cycle.”

Distributed Modern Architectures, Visualized by Epsagon

As a result, developers could identify where unexpected events occurred in a chain and could reproduce the issue when running a function on their local machine. In production, they use Epsagon’s tagging capabilities to allow either help desk or developers obtain visual representation of specific customers’ problems and then work with the customer to resolve issues in a short time period (even while on the call). Given the challenges of cloud and microservices, visibility with Epsagon was crucial to providing an insurance experience that could not be found elsewhere.

Finally, from a development perspective, Mutual of Enumclaw found that Epsagon support was highly responsive and expert. The elegance of the Epsagon solution and its setup in less than 5 minutes also made it much simpler to work with than other solutions, Nikody said.  With Epsagon, Mutual of Enumclaw could build environments and tear them down at a moment’s notice, without complications.

“The elegance of the Epsagon solution and its setup in less than 5 minutes also made it much simpler to work with than other solutions.”

Monitoring AWS Lambda Functions

As Mutual of Enumclaw expanded its modern architecture footprint, they encountered additional challenges. One blind spot was the timeouts and out-of-memory conditions that can occur when getting started with AWS Lambda. Choosing the maximum running time and memory limit for the function was always a challenge, since the developers didn’t know how much time or memory the function needed. Using Epsagon’s functions analysis, they could easily identify these issues.

Identify Unique Modern Application Problems: Timeouts and Out-of-Memory

“Staying on top of these issues with Epsagon contributed to developer velocity and ensured the health of the applications.”

Monitoring Containerized Services

Managing Container Usage and Resource Consumption

The Claim Center, a major part of the company’s application, ran on ECS clusters. Using Epsagon, engineers could correlate problems from the traces to the environment and infrastructure the center is running on. Epsagon automatically mapped all running services and their tasks, so that developers could see the health of the application, Nikody explained.

“Epsagon automatically mapped all running services and their tasks, so that developers could see the health of the application.”

Listing Services and Tasks with Easy Navigation to Data in AWS

With the ability to drill down into a specific service and detect spikes in CPU or memory usage, developers could better understand how the application was operating. Using Epsagon, Mutual of Enumclaw monitored disk I/O operations (bytes and counts), in addition to the health checks.

Detecting Spike in Traffic by Looking at CPU and Memory Consumption

Improving Service Quality

Mutual of Enumclaw’s engineering teams ran multiple environments in several stages (dev, sandbox, preproduction, and production), which results in tens of ECS clusters spread across multiple regions and accounts. Understanding the current health status of clusters, including performance trends for each and every task, was crucial to ensure application reliability and meeting SLAs.

By integrating Epsagon CloudFormation stack, all clusters were scanned automatically and displayed in a unified dashboard that presented both infrastructure data and application metrics, including logs and traces, Nikody noted.

“By integrating Epsagon CloudFormation stack, all clusters were scanned automatically and displayed in a unified dashboard that presented both infrastructure data and application metrics, including logs and traces.”

In addition, Epsagon helped company teams to understand the application processes running on the ECS cluster, by using Epsagon’s distributed tracing engine that automatically collects infrastructure data from the ECS task metadata endpoint.

Information from the ECS Task Metadata

Expanding Services and National Reach

Mutual of Enumclaw continued expanding its applications and its modern application footprint. The goal was to add more and more services into their modern application architecture and extend cost, efficiency, and productivity benefits.

Mutual of Enumclaw also expanded its solution to more states. Using a modern application approach, the business was able to provide a scalable solution to more customers, while maintaining a rapid rate of development.

Modern applications provided Mutual of Enumclaw with a competitive edge in the market. By using AWS Lambda, ECS and other AWS services, the business was able to adjust to changes quicker than ever.

Epsagon helped Mutual of Enumclaw to ensure business and technical success:

  • Business Impact: up to 80% reduction in error rates, ensuring excellent user-facing performance across global platforms.
  • Technical Impact:  sped up MTTR (Mean Time to Repair) – up to 85% reduction in troubleshooting time, freeing engineering teams to pursue business value.
  • Architecture Map: understanding which components are involved in handling each of the services and minimizing the silos across teams.

“Epsagon delivered up to 80% reduction in error rates, ensuring excellent user-facing performance across global platforms and up to 85% reduction in troubleshooting time, freeing engineering teams to pursue business value.”