Bastian Solutions, a Toyota Advanced Logistics company, is an independent material handling system integrator, providing automated solutions for distribution, manufacturing, and order fulfillment centers around the world. Since 1952, Bastian Solutions has grown from a Midwest company into a global corporation with multiple U.S. offices as well as international offices in Brazil, Canada, India, Mexico, and Spain.

Monitoring IoT Trackers Via Serverless

Sai Ponnaganti is an IoT Cloud Systems Developer at Bastian Solutions. His team, based in Indianapolis, is developing IoT trackers, a software platform, and custom analytics for the supply chain. The team leverages the power of AWS Lambda to gather sensor data, implement tracking features, and alert customers.

“As we sell IoT trackers and services, it is crucial for us to monitor each tracker: how it behaves, what type of data it reports, and under which conditions,” said Sai. This monitoring is done via their serverless functions.

“As we sell IoT trackers and services, it is crucial for us to monitor each tracker: how it behaves, what type of data it reports, and under which conditions.”

The Problem: Troubleshooting Serverless Applications

One of the Bastian Serverless Applications

When troubleshooting Bastian’s serverless applications, there had been no easy or clear way of troubleshooting the serverless functions, Sai explained.  “With a large and sophisticated architecture, it’s hard for us to figure out where the error is happening in the pipeline. The basic offerings from our Cloud provider gave some insight to where the errors were occurring, but finding the exact error was still tedious.”

We had to dig deep into logs. We call it log hell. We would iterate through each log stream in AWS CloudWatch to look for errors, which was very time-consuming. The timestamps are in UTC, so we had to do calculations. We couldn’t start fixing the code until we spent at least a few hours to find where the error was in our architecture.”

We had to dig deep into logs. We call it log hell.”

Alerts were another problem. “We would get email alerts from CloudWatch if a threshold has reached a certain point, but beyond that, we still had to go through the CloudWatch streams and logs.”

 Epsagon’s Visibility into Architectures and Applications

Epsagon is an all-in-one serverless and microservices monitoring solution. It is well-designed to provide companies with a birds-eye view into their modern architecture, as well as individual applications.

Epsagon traces each service call, provides many meaningful statistics, automated diagrams, deep integrated insights to help enhance and refactor code, as well as a real-time alerting system—all with a simple setup.

As can be seen in the diagram Bastian Serverless Applications, Bastian Solutions uses not only Lambda functions but also regular Python, container-based services including Amazon ECS. They used Epsagon to connect and trace those services as well.

The Setup Process

CloudFormation: Bastian deployed the CloudFormation template provided by Epsagon so that it can automatically monitor Bastian’s Lambda functions and provide statistics.

Lambda wrapper: Bastian wrapped each Lambda with an Epsagon Wrapper using the Epsagon plugin for Serverless Framework. As a result, Epsagon can trace each API call that a Lambda function makes.

Real-time alerts

According to Sai, Epsagon’s real-time alerts are right on target. The alerts helped Bastian to quickly identify the type of error and the occurrence time. “The ability to go to the specific transaction was a complete game changer for us. Not only did we know the error immediately, but we also knew where it was and how it happened through tracing capabilities. Now our developers can focus on fixing the code, instead of searching for the problem.”

“With Epsagon’s real-time alerts, the ability to go to the specific transaction was a complete game changer for us….Now our developers can focus on fixing the code, instead of searching for the problem.”

 

Monitoring

Serverless Issues Manager

Epsagon Issues Manager

Epsagon’s Issues Manager, Sai noted, has also been a turning point for Bastian Solutions. “It has enabled us to look at the state of each Lambda function with an issue, all in one place, and displays clear and concise messages as to what has gone wrong. We were finding errors that we never thought would be an issue. We are glad that Epsagon is finding these insights for us before the code has been released to production.”

“We were finding errors that we never thought would be an issue. We are glad that Epsagon is finding these insights for us before the code has been released to production.”

Insights

Meaningful insights and statistics about each Lambda function have made Bastian us more aware of what the Lambda functions are doing, Sai explained. “Epsagon made it easy for us to look at the flow of information clearly. Sometimes I look at the logs in Epsagon and think, ‘It can’t get any more transparent than this.’”

“Epsagon made it easy for us to look at the flow of information clearly. Sometimes I look at the logs in Epsagon and think, ‘It can’t get any more transparent than this.’”

Troubleshooting

One of the best troubleshooting features is the capability to query all of the logs based on application, a particular IoT device and error type, Sai said. “We are using Trace Search to search for payloads and quickly root-cause problems.”

“We are using Trace Search to search for payloads and quickly root-cause problems.”

Clear visibility

Epsagon has designed its product in a way that any issue can be easily understood. Whether it is for managers or developers, the clean dashboard will display where the error has occurred and what triggered it.

Time savings

Without exaggeration, Sai said, Bastian’s time spent troubleshooting has been cut down from hours to seconds. “As soon as Bastian receives an error message, I’m immediately off to code and can begin implementing the change. Epsagon makes our development cycle faster.”

“Epsagon makes our development cycle faster.”

Lessons Learned

  • Before you start anything, see if adopting Serverless is right for your company/department. Know the pros and cons.
  • Before you start developing, research and implement performance tests with different Lambda languages.
  • If using the Serverless Framework, research some best practices.
  • Don’t forget to have tests in your pipeline. Have an automated way to do unit tests, integrations tests, and acceptance tests.
  • Implement a CI/CD tool that has built-in integration with the deployment tool that you use.
  • “Use Epsagon to help visualize, debug, and solve problems in your serverless and container-based architecture faster.”

“Bastian’s time spent troubleshooting has been cut down from hours to seconds.”