Monitoring Modern Applications on AWS: How Bastian Solutions Gains Observability Into their IoT Supply Chain

Bastian Solutions, a Toyota Advanced Logistics company, is an independent material handling system integrator providing automated solutions for distribution, manufacturing, and order fulfillment centers around the world. Since 1952, Bastian Solutions has grown from a Midwest company into a global corporation with multiple U.S. offices as well as international offices in Brazil, Canada, India, Mexico, and Spain.

Sai Ponnaganti is an IoT Cloud Systems Developer at Bastian Solutions. His team, based in Indianapolis, is developing IoT trackers, a software platform, and custom analytics for the supply chain. They leverage the power of AWS Lambda to gather sensor data, implement tracking features, and alerts their customers.

“As we sell IoT trackers and services, it is crucial for us to monitor each tracker – how it behaves, what type of data it reports, and under which conditions”, says Sai. This monitoring is done via their serverless functions.

The Problem: Troubleshooting Serverless Applications

One of Bastian Solutions’ Serverless Applications

When describing troubleshooting Bastian’s serverless applications, Sai believes there was no easy or clear way of troubleshooting the serverless functions. “With a large and sophisticated architecture, it’s hard for us to figure out where the error is happening in the pipeline. The basic offerings from our Cloud provider gave some insight to where the errors were occurring, but finding the exact error was still tedious.”

We had to dig deep into logs – we call it log hell. We would iterate through each log stream in AWS CloudWatch to look for errors, which was very time-consuming. The timestamps are in UTC, so we had to do calculations. We couldn’t start fixing the code until we spent at least a few hours to find where the error was in our architecture.

Alerts were another problem. We would get email alerts from CloudWatch if a threshold has reached a certain point, but beyond that, we still had to go through the CloudWatch streams and logs.”

The Proposed Solution – Epsagon

Epsagon is an all-in-one serverless monitoring product. It is well-designed to help companies have a birds-eye view into their modern architecture, as well as individual applications.

Epsagon traces each service call, provides many meaningful statistics, automated diagrams, deep integrated insights to help them enhance and refactor code, and real-time alerting system – all with a simple setup.

As can be seen in the diagram above, Bastian Solution uses not only Lambda functions but also regular Python, container-based services. They used Epsagon to connect and trace those services as well.

The Setup Process

CloudFormation: Bastian deployed the CloudFormation template provided by Epsagon so that it can automatically monitor Bastian’s Lambda functions and provide statistics.

Lambda wrapper: Bastian wrapped each Lambda with an Epsagon Wrapper using the Epsagon plugin for Serverless Framework. With doing so, Epsagon can trace each API call that a Lambda function makes.

How Epsagon is helping Bastian Solutions, as described by Sai:

Real-time alerts

The real-time alerts are straight to the point. They helped us to quickly identify the type of error and the occurrence time. The ability to go to the specific transaction was a complete game changer for us. Not only did we know the error immediately, but we also knew where it was and how it happened through tracing capabilities. Now our developers can focus on fixing the code, instead of searching for the problem.

Monitoring

Serverless Issues Manager

The Issues Manager that Epsagon released has also been a turning point for Bastian Solutions. It has enabled us to look at the state of each Lambda function with an issue, all in one place, and displays clear and concise messages as to what has gone wrong. We were finding errors that we never thought would be an issue. We are glad that Epsagon is finding these insights for us before the code has been released to production.

Insights

Meaningful insights and statistics about each Lambda function have made Bastian us more aware of what the Lambda functions are doing., Epsagon made it for us to look at the flow of information clearly. Sometimes I look at the logs and think, “It can’t get any more transparent than this.”

Troubleshooting

One of the best troubleshooting features is the capability to query all of the logs based on application, a particular IoT device and error type. We are using Trace Search to search for payloads and quickly root-cause problems.

Clear visibility

Epsagon has designed its product in a way that any issue can be easily understood, Whether it is for managers or developers, the clean dashboard will display where the error has occurred and what triggered it.

Time savings

Without exaggeration, Bastian’s time spent troubleshooting has been cut down from hours to seconds. As soon as Bastian receives an error message, I’m immediately off to code and can begin implementing the change. Epsagon It makes our development cycle faster.

Lessons Learned

  • Before you start anything, see if adopting Serverless is right for your company/department. Know the Pros and Cons.
  • Before you start developing, research and implement performance tests with different Lambda languages.
  • If using Serverless Framework, research some best practices.
  • Don’t forget to have tests in your pipeline. Have an automated way to do unit tests, integrations tests, and acceptance tests.
  • Implement a CI/CD tool that has built-in integration with the deployment tool that you use.
  • Use Epsagon to help visualize, debug, and solve problems in your serverless architecture faster.