AWS Python

How to build an AWS Lambda function with Python 3.7 (the right way)

In this short article, we will see what’s the best way to build a Python 3.7 app which runs on AWS Lambda and requires additional packages (i.e. urlib3, pyyaml, requests).

Note: This article requires a minimal knowledge of Docker, AWS Lambda and of course Python.
Disclaimer: All thoughts and opinion are my own.

AWS Lambda and Python

AWS Lambda currently supports three Python versions with specific runtime environments.

source: https://docs.aws.amazon.com/lambda/latest/dg/python-programming-model.html

From the above list, it’s clear that every Python runtime uses Amazon Linux, which is a distribution that evolved from CentOS and Red Hat Enterprise Linux (RHEL).

Why this information is useful? 
It is useful because if we are using an external package in our Python app, it has to be compatible with that specific Linux distro (see what’s behind pip here: https://pydist.com/blog/pip-install).

The main “problem” is we don’t have control of what can be installed on a Lambda environment due to its serverless nature (which is a huge benefit from a cloud management perspective).

The solution is simple: bundle and ship our script/s and the package/s together.

Considering the above information, the next natural step is to build and bundle the entire application using the AWS Docker Amazon Linux 2018.03 image (https://hub.docker.com/_/amazonlinux)

Create the developer environment

In this section, we will see how to set up from scratch our Python 3.7 environment using the AWS Docker image.

Let’s start by creating a Dockerfile based on the Amazon Linux image, then add some lines to install Python 3.7 and pip.

Finally, we can build our image with the following command where amazonlinux-py3.7 is an arbitrary image tag.

A complete example

Let’s try to create a simple function which calls an API and returns a response.

From the terminal, we can create a local folder for this project, then move inside it.

Inside our project directory we can create a file called lambda_function.pywith the following code:

The function named lambda_handler is the handler function that Lambda calls when the function is invoked.
(for more info see: https://docs.aws.amazon.com/lambda/latest/dg/python-programming-model.html)

The function as it stands won’t work because we need to install the requestpackage first.
To satisfy the import statement we should open an ssh session in our previously built Docker container with:

and install the requests package inside our project directory with:

The content of our project folder should look like this:

project directory structure

Let’s close the ssh connection (writing exit in the terminal) and go to the next step.

Bundle and deploy on AWS

To create our package we should compress in a .zip file all the content of our project directory, packages and .py script.

Let’s move inside our project folder (lambda_python) and run the following command:

Finally, we are ready to create and deploy our Lambda Function.
From the AWS Lambda dashboard, we can click on “Create function”.

Then, select “Author from scratch”, write a function name and select Python 3.7 as the runtime environment. Finally, we can press “Create function”.

Almost in the middle of the page in the “Function code” section, using the “upload” button we can upload the my_lambda.zip file previously created and then click “save” (on the top right).

Well done!! our Lambda function is ready to be tested.

Testing

From the AWS Lambda dashboard, we can click on “test” (top right) and configure a basic test event (see picture below).

Sample test event.

If we highlight the lambda_function.py file from the web editor and press “test” we should see an “Execution Result” tab with the output of the call.

Execution Result output for the API call.

Final consideration

With the approach described in this article, it is possible to build and package every type of Lambda function but…..

Before going ahead with any project let’s be aware of the current AWS Lambda main limitations:

Deployment package:

  • size 50 MB (zipped, for direct upload)
  • 250 MB (unzipped, including layers)
  • 3 MB (console editor)

Function timeout: 900 seconds (15 minutes)

[Full list here: https://docs.aws.amazon.com/lambda/latest/dg/limits.html]

If the .zip bundle is bigger than the supported size there are at least two other alternatives:

  1. Bundle the rest(or part) of the files/packages in an S3 bucket and deploy the main .py handler in Lambda. The main handler for each call must download and use the S3 files (example here). This is possible since the current Lamda function and layer storage is 75 GB but could cause some performance issue.
  2. Consider AWS Fargate (https://aws.amazon.com/fargate/).

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.