A step by step guide using AWS Lambda layers to manage installation of Python modules for custom AWS Boto3 SDK, requests, numpy, pandas and scipy libraries. Follow this if you are seeing “Unable to import module ‘lambda_function’: No module named” error on your function and the best practices!

Introduction

The use of Lambda layers makes developers’ jobs much easier by introducing modularity across multiple Lambda functions. The creation of layers and deployment packages is very similar except on a small change — the folder structure. Depending upon the runtime, this will change as listed here. Layers are added to the /opt
directory and are included in the $PATH. This means that no other edits to sys.path or environment variables are required.

The Prerequisites:

1. EC2 instance using Lambda AMI for the Lambda function. This step is important for python runtime; make sure we are using the correct AMI otherwise this will not work.
2. ssh into the instance
3. Complete: aws configure
4. Upgrade AWS CLI to the latest version: pip install –upgrade awscli –user

Install the minor version of python you want.
# Download python36
$ yum list | grep python36
python36-pip.noarch

$ sudo yum install python36-pip.noarch -y
$ python3 --version
> Python 3.6.11

# Download and install pip3
$ curl -O https://bootstrap.pypa.io/get-pip.py
$ python3 get-pip.py
$ rm get-pip.py
# Upgrade and configure the AWS CLI
$ pip3 install --upgrade awscli --user
$ aws configure

Let’s get started.

1. Install custom boto3 version in place of default SDK

Let’s SSH into the instance we created with the AMI, and get started with creating a custom boto3 layer to be used in the function. The .zip file must contain a folder named — python where all the dependencies must exist. Follow these CLI commands:

cd ~
mkdir boto3
cd boto3
pip3 install boto3 -t ./python # installs the latest
# Or use the required version from: https://github.com/boto/boto3/releases
pip install boto3==1.14.37 -t ./python

# Publish a new layer version named boto3 with a description.
aws lambda publish-layer-version --layer-name boto3 --zip-file fileb://boto3.zip --compatible-runtimes python3.6 python3.7 python3.8 --description "boto3 v1.14.37"

# Associate the layer to a function, or use the console
aws lambda update-function-configuration --function-name --layers

We can verify this inside Lambda once the layer is added by:

import boto3, json
def lambda_handler(event, context):
    print(boto3.__version__)
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

You should see the required boto3 version we just added: 1.14.37

2. Install the Requests module for Python

cd ~
mkdir requests
cd requests
# Install the module under a new python directory
pip3 install requests -t ./python
zip -r requests.zip python/

# Create a new layer version
aws lambda publish-layer-version --layer-name requests --zip-file fileb://requests.zip --compatible-runtimes python3.6 python3.7 python3.8 --description "Request module for Python runtime"

# Use the layer in an existing function
aws lambda update-function-configuration --function-name --layers

Within the function, start making API calls with the requests module!

import json, requests
def lambda_handler(event, context):
    response = requests.get('https://requests.readthedocs.io/en/master/')
    print(response.text)
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

3. Python: Pandas, NumPy, SciPy

You can also create the modules using a virtual environment, with something like:

cd ~
mkdir numpy-pandas-scipy
cd numpy-pandas-scipy/

# start virtual env for python3
python3 -m venv v-env

# activate it and install the modules, and deactivate the virtual environment
source v-env/bin/activate
pip3 install pandas numpy scipy
deactivate

# Copy the contents to a folder to zip it
cp -r ~/numpy-pandas-scipy/v-env/lib/python3.6/site-packages/ ~/numpy-pandas-scipy/python

# Zip
zip -r ~/numpy-pandas-scipy/pandasnumpy.zip ~/numpy-pandas-scipy/python

# Create a new layer
# If the layer is too large (> 65MBs), use S3 to upload the layer.
aws s3 cp ~/numpy-pandas-scipy/pandasnumpy.zip s3://
aws lambda publish-layer-version --layer-name numpy-pandas-scipy --content S3Bucket=,S3Key=pandasnumpy.zip --compatible-runtimes python3.6 python3.7 python3.8 --description "Pandas, Numpy & SciPy"

# Finally update the function to use the new layer
aws lambda update-function-configuration --function-name --layers

If the layer is too large, the API call to publish will fail and hence we need to send the .zip file to S3 and then reference the S3 file in the publish-layer-version CLI command as above.

Once done, you should be able to import all these modules without any errors as long as we are using the AMI for the runtime Lambda provides. If you run into permissions error, make sure the files and the directories have the correct read/write permissions before creating the .zip files

$ chown -R ec2-user:ec2-user python/
$ find . -type d -exec chmod 755 {} +
$ find . -type f -exec chmod 644 {} +

#Verify permissions and folder structure by:
$ zipinfo requests.zip

Archive: requests.zip
Zip file size: 992799 bytes, number of entries: 278
drwxrwxr-x 3.0 unx 0 bx stor 19-Oct-25 17:50 python/
drwxrwxr-x 3.0 unx 0 bx stor 19-Oct-25 17:50 python/urllib3/
-rw-rw-r-- 3.0 unx 5943 tx defN 19-Oct-25 17:50 python/urllib3/fields.py
-rw-rw-r-- 3.0 unx 5991 tx defN 19-Oct-25 17:50 python/urllib3/request.py

I’ve added a bunch of Lambda samples over GitHub for Python and CLI if you want to see more use-cases and sample .zip files for AWS Lambda layers. Hope this helps! 🎉