Deploying multiple Lambdas at once

[eneko]

I've been wondering what would be the best approach for deploying multiple lambdas at once, so I'm hoping to use this issue thread as a place to discuss the best strategy for this. If there is a better place, like forums.swift.org, let me know.

I don't have much experience with lambda, however, I can imagine the following scenarios:

• Swift package with single function
• Swift package with multiple, independent functions
• Swift package with multiple functions sharing resources (eg. same DynamoDB table, same AWS Gateway API, ...)

I'd like to assume for each of those scenarios, being able to deploy lambdas individually (this is, update a single lambda and deploy it) should remain an option.

Focusing on SAM for now, there are two approaches:

• One SAM template per function
• One SAM template per application, with multiple functions

What would be the recommended layout for projects with many functions? What are the pros and cons of one template for multiple lambdas vs one template per function?

[tomerd]

@eneko you are correct to point out that the deployment examples focus on single-function/single-executable per package

users of this library may of course choose to build "larger" packages that include many executables in them. we did not emphasize this option in the documentation and example as it leads to lager packages that have slower start time (AWS needs to download a larger zipfile, etc)

for modularity and code sharing, one can build an executable that is a router of sorts and can delegate the event to the relevant private modules.

#57 has some relevant discussion as well

[eneko]

Thanks for the feedback, @tomerd

I read #57, and found it interesting. For sure, it would be nice to support the "handler" name entered in Lambda configuration page, so a single binary could be used for multiple functions.

In the cases stated above in this issue, I was thinking on the scenario where a Swift Package contains multiple binaries (multiple executable products), one executable per Lambda function. This is, following how Lambda Functions Examples are set up.

The question I had is on what would be the best approach for budding, packaging and deploying multiple binaries at the same time, using SAM for example. Hope that makes sense.

[eneko]

Might be a good idea to document terminology. For example, here we are talking about two different packages:

• Swift Packages (the “project” used in Xcode)
• Packages downloaded by Lambda during a cold start.

Using a single SAM template to deploy multiple lambdas in a Swift Package shouldn’t make each packaged lambda larger in size.

Hope I’m making sense.

[fabianfett]

Hi @eneko,

I read #57, and found it interesting. For sure, it would be nice to support the "handler" name entered in Lambda configuration page, so a single binary could be used for multiple functions.

As stated in #57 you can use the _HANDLER env variable without any problem and you're free to do so.

The question I had is on what would be the best approach for budding, packaging and deploying multiple binaries at the same time, using SAM for example. Hope that makes sense.

You can just throw your sam templates into each other. If you want to you could deploy all example Lambas with just one sam template if you copy and pasted all the sam templates into one file.

Just make sure all your resources have different names.
Does that solve your issue?

[eneko]

@fabianfett thanks for the feedback. This is not really an issue, just trying to find out what the best practices are, so we could update scripts if needed, and/or document different approaches.

For example, on a demo project I'm working on, I have two lambda functions that use the same DynamoDB table. Having both functions in the same SAM template, together with the DynamoDB table definition, allows for using resource references instead of manually specifying the ARN:

oneFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: .build/lambda/FunctionA/lambda.zip
      [...]
      # Grants this function permission to perform actions on the table
      Policies:
        - Version: "2012-10-17"
          Statement:
          - Effect: "Allow"
            Action:
              [...]
            Resource: !GetAtt backendTable.Arn    <--- this

anotherFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: .build/lambda/FunctionB/lambda.zip
      [...]
      # Grants this function permission to perform actions on the table
      Policies:
        - Version: "2012-10-17"
          Statement:
          - Effect: "Allow"
            Action:
              [...]
            Resource: !GetAtt backendTable.Arn    <--- this

backendTable:
    Type: AWS::DynamoDB::Table
    DeletionPolicy: Retain
    Properties:
      [...]

However, in order to deploy this SAM template, both FunctionA and FunctionB have to be built and packaged.

This can easily be done with a deploy script like this:

echo "-------------------------------------------------------------------------"
echo "preparing docker build image"
echo "-------------------------------------------------------------------------"
docker build . -q -t builder

$DIR/build-and-package.sh FunctionA
$DIR/build-and-package.sh FunctionB

echo "-------------------------------------------------------------------------"
echo "deploying using SAM"
echo "-------------------------------------------------------------------------"

sam deploy --template "template.yml" $@

I'm not sure if this follows best practices, or if there is a better approach.

Thanks!

[fabianfett]

I'm not sure if this follows best practices, or if there is a better approach.

@eneko Well, I don't know if there is a better approach either, but that's pretty close to what I'm doing ;)

[tomerd]

@eneko would you like to PR some docs around this so we can close it out? maybe highlighting the tradeoffs and why normally you want to deploy one at a time?

[pokryfka]

@eneko

For example, on a demo project I'm working on, I have two lambda functions that use the same DynamoDB table. Having both functions in the same SAM template, together with the DynamoDB table definition, allows for using resource references instead of manually specifying the ARN:

one approach to address that is to have many Lambda function in the same SAM template (which also defines other AWS resources) but to package and deploy each function separately

oneFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: .build/lambda/FunctionA.zip

anotherFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: .build/lambda/FunctionB.zip

[pokryfka]

@eneko

As far as I know SAM CLI packages each function separately.
This used to work for supported runtimes only.
With recent SAM CLI release (1.1.0) you can configure makefile build method in SAM template and, as long as you define build targets (which could be just one with wildcard) sam build will package all each function for you.

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: hello-world/
      Handler: my.bootstrap.file
      Runtime: provided.al2
    Metadata:
      BuildMethod: makefile

References:

• https://aws.amazon.com/blogs/compute/migrating-aws-lambda-functions-to-al2/
• https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-cli-command-reference-sam-build.html

[eneko]

Thanks everyone. I haven't looked into this recently, but I'm looking forward to do more research and document it.

Some example of possible goals or nice-to-have's:

• Having a Swift package with multiple lambdas on it:
  • Build and deploy all lambdas at once
  • Specify one single lambda (from the ones included) to build and deploy
  • Automatically detect lambdas with code changes, build and upload only those

[saltzmanjoelh]

Hi, I have project called aws-deploy-kit that might help with this. It has a lot of cli help, readme help and some examples.

If you end up using separate SAM templates per Lambda, you could explicitly deploy them with a command like aws-deploy -d /path/to/package --post-build-command "aws sam deploy" function-one function-two

In the project, I converted all the shell build scripts from the examples in this repo to Swift code. The above command will build each function in Docker, then run aws sam deploy in the source directory for each target.

If you can switch from SAM, please take a look at the AdvancedExample to see how to programmatically deploy. The example shows a similar idea as what you described with the shared resources.

I just wrapped up with the examples. If you have any questions, comments or requests, please let me know.

[idelfonsog2]

Keeping this thread alive

I came to pondered upon the same doubts and implementations naturally as I was developing a bigger project (the ones proposed by @eneko and @pokryfka)

• Is this the way of building your AWS Serverless Application?
• By implementing it this way, It allow me to share AWS resources with my serverless "application"
• Packaging each lambda as a zip, should not increase the size/time of cold starts right?... as long as we keep the number of package dependencies to a minimum...
• There has been a couple of updates that has effects on the purposes of this thread which is... lower the costs..
  • AWS Lambda Container images
  • Graviton 2
• How to share service classes among your lambdas do we package those as libraries or executables and append those to the lambda swift code as dependencies?

A few "fine lines" I also found were:

• You cannot convert an existing container image function to use a .zip file archive. You must create a new function. here
• Containerized images cannot use layers