Troubleshooting AWS Lambda 101

To understand the various issues you encounter with serverless apps, rewiring your brain is a must. First of all, you don’t have access to stack traces anymore, nor do you have any installed agents to help you. And you will quite often end up staring at blank API responses that just say “internal server error” with no context whatsoever.

There are a few steps you can take to troubleshoot such issues. This article will show you those first steps needed to troubleshoot and debug your serverless applications. We’ll also discuss using the AWS console to trigger AWS Lambda functions with custom payloads. Of course, this only covers testing function invocations, while not showing you the logs for that particular invocation. Because of this lacking in the AWS Lambda console, we’ll address the need for CloudWatch, AWS’ built-in system for logs, as well. But even CloudWatch, despite being a great help, has its limits.

But first, let’s explore troubleshooting APIs, in general, to give you a gentle introduction.

Troubleshooting APIs

There are many API tools out there for you to use. Postman and Insomnia are two of the most popular. But you can always CURL the API Gateway endpoints tied to your function for easy testing as well.

By hitting a GET endpoint event that triggers a “Hello World!” function, you’ll see this in your terminal.

$ curl https://1234567890.execute-api.us-east-1.amazonaws.com/dev/
> Hello World!

Or, if you need to send a POST request, you can add some data to the request.

$ curl -x POST -d '{"username":"xyz","password":"xyz"}' \
https://1234567890.execute-api.us-east-1.amazonaws.com/dev/login
> Login Successful!

As you can see, this is a nice but rather painful way of testing if you have to do extensive troubleshooting. So Postman is an excellent alternative.

Postman

Postman gives you a clean, nice interface for setting the request type, headers, body, and tests. You don’t have to fiddle with the terminal; just use Postman instead! This approach works great for functions tied to API endpoints as event triggers, such as API Gateway. But not all Lambda functions are used this way. Often, you’ll end up having functions that are triggered asynchronously by other services, such as SQS, Kinesis, or even CloudWatch. What then?

Well, you move to the AWS Lambda Console and CloudWatch in order to perform proper debugging.

Debugging Lambda Using the AWS Console

AWS has done a great job giving you the ability to debug and test your functions through the console. It also gives you several options for configuring how your function will be triggered. The benefit of this is that you can get an instant feedback loop while debugging so as to better understand your functions.

Debugging using the AWS console

By pressing the test button you can insert a sample value and trigger your function on command. The event samples that you can use range from Kinesis to SNS and CloudWatch. Basically, anything you would ever need to test is available to you. If you want to troubleshoot triggering the Lambda functions with an SES email event, it’s just as easy, as you can see below.

Configure a test event

But what about metrics? Easy. You just need to move to the Monitoring tab in the Lambda console.

AWS console AWS Lambda function monitoring

Here’s where you can see function metrics, including error percentages, invocation count, and duration. Keep in mind that this is just a quick glance into the monitoring of your function. If you want to go into more detail, head on over to CloudWatch metrics to get more insight.

Checking Logs With CloudWatch

AWS Lambda pushed all logs, by default, to CloudWatch. Here, every Lambda function creates something called a Log Group. This is where all logs for that particular Lambda function are sent. Further on, these Log Groups get separated into Log Streams. And every Log Stream represents the logs that get generated from a particular Lambda instance.

NoteA Lambda instance is basically a container that will run the code that you supplied to the Lambda function. Multiple Lambda instances can be created at the same time if there are lots of concurrent requests hitting your functions.

By jumping over to the CloudWatch console on AWS, you can see all Log Groups by pressing Logs in the left sidebar.

CloudWatch Log Groups

You can then drill down further by clicking a single Log Group to see all Log Streams for a particular function.

CloudWatch Log Streams

For even more detail, click on a Log Stream, and check out all log events for that particular Log Stream.

CloudWatch Log

Every function invocation will generate log events right here. This gives you a nice and quick feedback loop when you’re debugging your functions. It also makes it easy to troubleshoot when something breaks, because the logs appear almost in real time.

However, this does raise another issue. You still don’t have any insight across the entire system. One function may be causing a failure, which then only shows itself further down the road in yet another function. CloudWatch isn’t well suited to handle such a problem.

In cases where you need a proper monitoring solution, you’ll need to turn to a third-party tool such as Epsagon. Epsagon gives you full insight into your serverless functions with support for traces and alerts. The coolest feature is Epsagon’s architecture overview, where you can see how your serverless functions interact with each other.

Full Observability With Epsagon

If you have an application running in production, allowing your customers to experience downtime is simply unacceptable. So knowing immediately when something breaks is priceless. Epsagon can deliver this peace of mind, enabling you to monitor and troubleshoot all of your functions in just a few minutes.

Epsagon Functions

The onboarding for Epsagon is quick and painless. You can even choose auto-instrumentation to deploy a CloudFormation stack to your AWS account, which will ship all of your logs to your new Epsagon dashboard. Or you can manually instrument your functions with agents.

By using Transaction IDs, you can also get a complete visual overview of your function architecture and actually see what’s going on! This visual tool will help you find issues faster, while also giving you the insights you need to stay on top of all of your functions. A transaction is literally a list of traces that are connected together, giving you a representation of the entire transaction for clear viewing and oversight.

Epsagon Transaction

One of Epsagon’s most powerful feature is the Issues Manager. This shows you a list of all errors, warnings, timeouts, and anything else that could be a risk to the health of your system. This means that you can get a quick glance of all errors without having to sift through tons of logs. This feature alone is priceless, as it saves you from wasting time on senseless troubleshooting.

Epsagon Issue Manager

By using these tools, you can achieve significant overview and insight into your application system’s health and performance, providing an easy and intuitive way of debugging serverless applications.

Conclusion

Troubleshooting serverless applications is not a walk in the park. In fact, it’s more like fumbling around in the dark. We’ve all been there, and we can all confirm that the feeling is horrible.

Developing serverless applications requires you to re-think troubleshooting in general. Often, all you get back from a failing function is a nice “internal server error” message. “Thanks a lot!” would be anybody’s reaction, while proceeding to throw a laptop out the window.

So here we’ve given you a few steps to stop the madness. First of all, learn about CloudWatch– how it works and how to read logs. Then make sure to understand how to test your AWS Lambda functions from the AWS Console. This is crucial for recreating failed invocations. Lastly, feel free to rely on a third-party solution to do it all for you. Epsagon is easy to set up and will give you full observability in just a few clicks.

Thanks for taking a look at this brief introduction to serverless debugging. If you have any questions, here are a few links and resources you can check out.