AWS teams help us to continuously improve our experience and capabilities with Lambda functions. This time, AWS announced the AWS Lambda support for EFS, which gives us a great opportunity to handle much larger files than before. According to the AWS Lambda limits, the `/tmp` directory could handle up to 512MB of data, which in many cases is not sufficient. Other alternatives could be storing these large files in S3 and handling them in batches, which introduces complications, and performance impacts.
With the new release today, EFS can be attached to Lambda functions, and it exposes a new directory under `/mnt/<name>`, that can handle any file size. In addition, these file systems can be shared across multiple functions and services. It can become very useful for AI/ML functions that use large data models, or any other processing functions that handle large files.
Setup EFS File System for Lambda
The setup is relatively simple, and consists of creating an EFS file system, adding permissions to your function, and attaching the EFS file system to your Lambda function.
Creating the file system
Head to the EFS console on AWS, click “Create file system”, select a VPC, and the desired AZs:
In the next step, make sure that the General Purpose is selected, and click next:
Now create an access point to the file system. You can see an example details to fill in, but make sure that you understand the configuration, and adjust it accordingly:
Confirm and create the file system:
Now let’s edit the IAM role of the desired Lambda function. Head to the permissions tab and click on your role name to open it in the IAM console:
In the IAM console, click on Attach Permissions, and add the following permissions:
Your role should look like this:
Head back to the Lambda function, and at the bottom part you’ll be able to find the new File system section and click on the add file system:
Insert the file system details that you’ve created, and save:
Ultimately it should look like this:
Now you can access files on the mount you’ve created!
As an AWS Lambda service ready partner, Epsagon makes sure the new configuration is visible for you in the same dashboard where you can see your function metrics and invocations. Once configured, you’ll be able to see the attached file system, and get all of the configuration details:
And get the full configuration when you click on the file system:
Try it yourself here.
Amazon EFS provides shared file storage in the AWS Cloud. With that, you can create a file system, mount the EFS file system from an AWS Lambda function, in a very simple configuration. This allows us to leverage Lambda functions for more scenarios where we need to handle larger files. When using it, verify to you set the right configuration for the file system and benchmark the impact on your current performance.