How to setup AWS Lambda with SQS - everything you should know!

Ran Ribenzaft
AWS recently introduced SQS integration to Lambda. We have been waiting for this for a long time now. In the following post, I'm going to share a quick-and-simple tutorial on how to get started with message distribution between Lambda functions, using SQS.

In addition, I will also compare between SQS and SNS (i.e. why should we choose one over the other), and present an in-depth performance analysis of using SQS as a message distributor.


Setup

In this tutorial, I will use the Serverless Framework, to send a message from one Lambda function to another Lambda function via SQS.

Prerequisites:

  1. npm installed.
  2. Serverless Framework installed.
  3. Python - Feel free to use your favorite programming language.
  4. AWS account (duh).

Let's start by creating the serverless project:

serverless create --template aws-python3 --path sqs-lambda-tutorial
cd sqs-lambda-tutorial


At this point, we can configure our serverless.yml and handler.py files. The code can be found in the following GitHub repo.

serverless.yml

This yml file configures our functions, permissions and SQS resource.

handler.py

This is where our code runs. It contains two functions. Each function represents a different Lambda function. One will be triggered manually, and the other one will be triggered by an SQS message.

Deploy!

This is the magic of Lambda. Just run the following command while you are in the same folder as serverless.yml:

sls deploy

The output should look roughly like this:

Now we can invoke the "start" function by running the following commands:
sls invoke -f start-lambda
sls logs -f start-lambda
sls logs -f end-lambda


You will be able to see the CloudWatch Logs of the Lambda functions with the SQS message.

We are done. Congrats! 🎉👏⚡️

Event structure

If you plan to use SQS in your application, you will need the event structure and data. You can find it in the following Gist.

SQS vs. SNS

When applied to Lambda only, this is a tough call. Up to this moment, only SNS supported triggering Lambda functions. Now both SNS and SQS can trigger. Let's compare the relevant parameters for Serverless applications:


The main advantage for SQS is the batch messaging, which can also lead to cost reduction.

Performance analysis

Distributing messages between services can be complicated. Understanding the performance impact on our applications is almost impossible. At Epsagon, we love to see the bigger picture for highly distributed applications. In a previous post, we compared several methods of distributing messages between functions (direct sync invocation, direct async invocation, SNS and Kinesis), and now, using our serverless performance monitoring tool, it's a great time to introduce the results for SQS:
  • Operation duration (how long we wait to publish message): ~10ms.
  • Total duration (how long it takes from publish until it reaches the destination): ~45ms.

Compared to the results from the previous post - this is much better!

Conclusion

Lambda integrations are great. The more, the better. In this post, we learned how to deploy Lambda functions that distribute messages using SQS.

We also saw an updated comparison between SQS and SNS for Serverless applications and understood the performance implications of distributing messages between functions.

Considering features, cost and performance, SQS is the best choice for the task of message distribution between Lambda functions.

Tags

AWS Serverless AWS Lambda AWS SQS SQS Lambda trigger SQS Lambda tutorial Python Performance Distributed Systems Distributed Tracing Serverless Monitoring Lambda Monitoring

You may also like

Ran Ribenzaft
Mar 31, 2018
The right way to distribute messages effectively in serverless applications
When scaling a serverless application into a real architecture, that is composed of many resources and elements — it becomes hard to design it properly…
Gal Bashan
Apr 25, 2018
Lambda Internals: Exploring AWS Lambda
AWS Lambda is an excellent environment for rapid and scalable development. As a developer, I love using it. The main advantage of Lambda is…
Ran Ribenzaft
Feb 15, 2018
How to make Lambda faster: memory performance benchmark
Configuring a Lambda function is considered as a very complex task. Among all parameters, you will have to pick the memory size for your function, which is very confusing…

Contact Us

Want to see your serverless application in action?