Setup Serverless Applications With AWS CloudFormation
13 min read

Setup Serverless Applications With AWS CloudFormation

Kickstart: Initiation to AWS CloudFormation

A hands-on guide to using AWS CloudFormation  for setting up your serverless application

Photo by Sigmund / Unsplash

Let's learn about CloudFormation templates, and how we can use them to create serverless applications. This guide is very hands-on and will let you become much more familiar with manipulating CloudFormation templates.

This is an example of a typical company/enterprise setup. For professional projects, developers often do not have access to all the features in the AWS console and have to deploy the infrastructure changes using CloudFormation. If you are ever stuck during the hands-on guide, the full code can be found at the end of the article.

If you haven't worked with serverless applications yet, have a look at Write Your First Serverless Application using AWS Lambda. That article explains the why and the how of serverless applications, with a manual setup. This guide, however, is focused on writing serverless applications together with CloudFormation templates.

Why CloudFormation?

CloudFormation is AWS' service for creating infrastructure as code. Developers store a configuration file describing all the resources necessary for the application, AWS then automatically creates those resources for your application.

Why would we want that? First of all, by having your infrastructure as code, you can enjoy all the advantages of git. All changes to the infrastructure are logged, and backed up at all times. If the current infrastructure breaks, you can simply re-run the CloudFormation template. With manual setups, developers have to manually delete and re-create every infrastructure item for an application.

Enjoy the advantages of Git. Photo by Yancy Min / Unsplash

Professional Environments

Finally, in professional environments, developers often do not have the required access for manually setting up an infrastructure. This is where CloudFormation templates come in handy. This is how it usually works:

  1. The cloud administrator creates an IAM user, with all the necessary rights for your required infrastructure. Together with the cloud administrator, the access rights can be tweaked.
  2. Developers create the infrastructure configuration file, also called CloudFormation template.
  3. In many cases, developers setup automated builds and deployments using CICD tools, pushing the CloudFormation template to AWS.
  4. Finally, the CloudFormation template is run on AWS using the credentials of the IAM user, which was created by the cloud administrators.

We will work in a similar way, in this guide we will focus on getting familiar with manipulating CloudFormation templates. We will define our infrastructure, write it in CloudFormation templates, and deploy it. Let's go!

Creating Our Serverless Application

First of all, we'll need to have a serverless application that we want to deploy. Let's create a fun application which draws speaking cows on the screen! We will use the cowsay NPM package for this purpose.

Create a package.json with the dependency for cowsay. Don't forget to run npm install

{
  "name": "cowsay",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "cowsay": "^1.4.0"
  }
}
c

Let's create the application's index.js file where we want to run the cowsay package.

const cowsay = require('cowsay');

exports.handler = function(event, context) {
    // Get a cow from cowsay!
    let cow = cowsay.say({
        text: "I'm a lambda moooodule",
        e: "oO",
        T: "U "
    });

    // Replace line breaks to HTML line breaks
    cow = cow.replace(/[\n|\r|\n\r]/g, '<br />');

    // Create the HTML
    var html = '<html><head><title>Cowsay Lambda!</title></head>' + 
        `<body><h1>Welcome to Cowsay Lambda!</h1>${cow}</body></html>`;
    
    // Return the HTML response
    context.succeed(html);
};

Finally, let's run this lambda locally using the following command. Make sure you have the AWS SAM CLI installed on your machine. If this is new to you, have a look at our previous article, Writing Your First Serverless Application.

Before we can run it in our command line, we'll also need to define the template.yaml file, to explain which runtime engine our lambda method is using.

AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'
Description: An AWS Serverless Specification template describing your function.
Resources:
  myfirstlambda:
    Type: 'AWS::Serverless::Function'
    Properties:
      Handler: index.handler
      Runtime: nodejs14.x
      CodeUri: .
      Description: ''
      MemorySize: 128
      Timeout: 3

Finally, we can run the sam local invoke command to run the lambda locally.

$ sam local invoke
Invoking index.handler (nodejs14.x)

START RequestId: 47128cc5-4c83-47eb-89e7-d5f73f22a72b Version: $LATEST
END RequestId: 47128cc5-4c83-47eb-89e7-d5f73f22a72b
REPORT RequestId: 47128cc5-4c83-47eb-89e7-d5f73f22a72b  Init Duration: 0.99 ms  Duration: 713.01 ms     Billed Duration: 800 ms Memory Size: 128 MB     Max Memory Used: 128 MB
"<html><head><title>Cowsay Lambda!</title></head><body><h1>Welcome to Cowsay Lambda!</h1> ________________________<br />< I'm a lambda moooodule ><br /> ------------------------<br />        \\   ^__^<br />         \\  (oO)\\_______<br />            (__)\\       )\\/\\<br />             U  <br /><br />----w <br /><br />                <br /><br />     <br /><br /></body></html>"      

We can see our cow in the output! However, this is HTML so it does not look very nice in the console. Once we have it running on our lambda, we will be able to see our cow!

Manual CloudFormation Templates

Setting Up an AWS Lambda Function

Let's first create a CloudFormation stack manually, afterwards we'll go more in-depth on using automatic deployment tools, so you can be familiar with both ways of work.

Go to the AWS Console, and select CloudFormation. We will create a new stack for our application. A stack is a group of services required for an application, or part of an application to run. Select the with new resources (standard) option.

We will then select to create a new CloudFormation template using the designer. Once you get the hang of CloudFormation templates, you could start making them manually as well.

Select, create template in Designer

You will find the following screen appear. It looks intimidating, because Amazon has a wide range of services to choose from. Stick with the services you know you will need and this will be fine. We also notice that at the bottom, a JSON output is provided. This will be your final CloudFormation template.

Let's drag in the AWS Lambda resource onto the right. Under Lambda, find Function and drag it onto the editor. Now, you will notice a piece of configuration has been created for you once you press on the Lambda.

The properties of this AWS Lambda however seem to be an empty object {}. In order to know what to fill in, let's head over to the AWS Lambda documentation. The documentation specifies which properties are required, and what they do. When scrolling down on the documentation, you'll also be able to find useful examples. Let's copy over parts of their example.

CloudFormation templates are very verbose, so please bear with me for a moment. You'll find the more you work with these, they will look less and less intimidating.

"AMIIDLookup": {
    "Type": "AWS::Lambda::Function",
    "Properties": {
        "Handler": "index.handler",
        "Role": {
            "Fn::GetAtt": [
                "LambdaExecutionRole",
                "Arn"
            ]
        },
        "Code": {
            "S3Bucket": "lambda-functions",
            "S3Key": "amilookup.zip"
        },
        "Runtime": "nodejs12.x",
        "Timeout": 25,
        "TracingConfig": {
            "Mode": "Active"
        }
    }
}
Example configuration provided by AWS

First of all, let's change the name of our lambda, to cowsayLambda. We will then need the Code property, where we specify where AWS can find the code of our Lambda application. Let's set the value of S3Bucket to cowsaybucket. We'll have to generate this s3 bucket as well in our CloudFormation file. The S3Key can be set to cowsay.zip which will contain our application code. We will also copy over the Runtime value as we want to define our code to be run in nodejs, set it to nodejs. Finally, set Handler to index.handler. This means, the lambda will run the handler method in the index.js file.

We should now have a configuration similar to the below, which is a good start for the Lambda.

Pro Tip: If you want to beautify your JSON template, press the YAML button and go back to the JSON view. Amazon will automatically beautify it. It also works the way around, if you prefer writing the configuration in the YAML format.

{
    "Resources": {
        "cowsayLambda": {
            "Type": "AWS::Lambda::Function",
            "Properties": {
                "Code": {
                    "S3Bucket": "cowsaybucket",
                    "S3Key": "cowsay.zip"
                },
                "Handler": "index.handler",
                "Runtime": "nodejs14.x"
            }
        }
    }
}
Our Lambda configuration

Creation of IAM Role for Lambda

The Lambda needs to have an IAM execution role defined. The lambda will assume this role when running, and will have the associated permissions. For our lambda, we don't require any specific permissions. Let's set up a default IAM role that we can then refer to within the Lambda configuration. Drag an IAM:Role item onto the visual editor and update it to the following template.

{
    "Resources": {
        "cowsayIamRole": {
            "Type": "AWS::IAM::Role",
            "Properties": {
                "AssumeRolePolicyDocument": {
                    "Version": "2012-10-17",
                    "Statement": [
                        {
                            "Effect": "Allow",
                            "Principal": {
                                "Service": [
                                    "lambda.amazonaws.com"
                                ]
                            },
                            "Action": [
                                "sts:AssumeRole"
                            ]
                        }
                    ]
                },
                "Path": "/",
                "Policies": [
                    {
                        "PolicyName": "root",
                        "PolicyDocument": {
                            "Version": "2012-10-17",
                            "Statement": [
                                {
                                    "Effect": "Allow",
                                    "Action": [
                                        "logs:*"
                                    ],
                                    "Resource": "arn:aws:logs:*:*:*"
                                }
                            ]
                        }
                    }
                ]
            }
        }
    }
}

This template comes from an example from the AWS Documentation. It creates an IAM role which the Lambda can assume, and grants access to logs, in case the Lambda will create logs.

Now we'll need to update the Role on the Lambda, to use this new IAM role. Ad the following piece of configuration to the properties of the Lambda function.

"Role": {
   "Ref": "cowsayIamRole"
}

Creation of S3 Bucket

We have specified an S3 Bucket named cowsaybucket in our Lambda configuration. Let's drag in an S3 bucket into our visual editor. Open up S3 and drag a Bucket onto the editor. Click it, and edit the properties to rename it to cowsaybucket. Find a pink circle on the Lambda Function in the visual editor. Drag it towards the Bucket in order to create a dependency. The Lambda cannot be run without the Bucket, since the Bucket will contain the code. Note, the Bucket must light up pink when you drag the arrow towards it, to make the dependency link.

Now, we can set up the S3 Bucket properties, click the bucket and let's head over to the Amazon S3 CloudFormation documentation page. We can find an example usage again.

{
    "Resources": {
        "S3Bucket": {
            "Type": "AWS::S3::Bucket",
            "DeletionPolicy": "Retain",
            "Properties": {
                "BucketName": "DOC-EXAMPLE-BUCKET"
            }
        }
    }
}
Example S3 Bucket Configuration from AWS Documentation

We won't need that specific DeletionPolicy, but the rest can remain the same. We will have to add the BucketName property. Set it to cowsaybucket.

{
    "Resources": {
        "cowsaybucket": {
            "Type": "AWS::S3::Bucket",
            "Properties": {
                "BucketName": "cowsaybucket"
            }
        }
    }
}
Our final S3 Bucket configuration

Granting S3 Access

We have our Lambda and our Bucket, but the Lambda is not yet allowed to access the files on the bucket. Let's update our S3 Bucket's policies, to allow the lambda to retrieve application code from the bucket. The AWS documentation has an example policy described for this.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service":  "serverlessrepo.amazonaws.com"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::<your-bucket-name>/*",
            "Condition" : {
                "StringEquals": {
                    "aws:SourceAccount": "123456789012"
                }
            }
        }
    ]
}

We will need a similar policy for our bucket. Let's drag an S3 BucketPolicy into our visual editor. Our final BucketPolicy will look similar to the following. We define access to the cowsaybucket resources, if the ARN, the identifier of the calling resource, is the same ARN as the Lambda. An arrow of the dependency to the bucket will automatically appear in the visual editor.

{
    "Resources": {
        "cowsayBucketPolicy": {
            "Type": "AWS::S3::BucketPolicy",
            "Properties": {
                "Bucket": "cowsaybucket",
                "PolicyDocument": {
                    "Statement": [
                        {
                            "Effect": "Allow",
                            "Principal": {
                                "Service": "serverlessrepo.amazonaws.com"
                            },
                            "Action": "s3:GetObject",
                            "Resource": "arn:aws:s3:::cowsaybucket/*",
                            "Condition": {
                                "StringEquals": {
                                    "aws:SourceArn": {
                                        "Fn::GetAtt": [
                                            "cowsayLambda",
                                            "Arn"
                                        ]
                                    }
                                }
                            }
                        }
                    ]
                }
            }
        }
    }
}
Our Final BucketPolicy

Refactoring CloudFormation Templates

At this point, our full CloudFormation template is looking very good! However, there is one issue left. Our Lambda is expecting an S3 Bucket, check. This Lambda is expecting a file called cowsay.zip in the S3 bucket. Here is what will fail. It will try to find the source code for the lambda and create an error.

The issue is, the source code is not existing on the S3 bucket before retrieving it for the Lambda's properties. We can fix this by first creating the S3 bucket, then uploading the source code, and finally run a second CloudFormation stack which will create the Lambda. We can see this as a 3-step process.

  1. Run a CloudFormation stack to create an S3 Bucket
  2. Upload the source code of our Lambda to the S3 bucket
  3. Run a second CloudFormation stack to create the Lambda

Also very interesting to note is, we can find the full CloudFormation template at the bottom of the visual editor, on the template tab. This is one template containing all the resources we have defined.

Creating the S3 Bucket Stack

Let's start by creating a new stack called cowsay-bucket. In this stack we will only define the S3 bucket. Copy the previous S3 Bucket properties into it.

{
    "Resources": {
        "cowsaybucket": {
            "Type": "AWS::S3::Bucket",
            "Properties": {
                "BucketName": "cowsaybucket"
            }
        }
    }
}

In the visual editor, press on Create Stack. Then, continue until it is created. It will prompt for a stack name, let's call it cowsay-bucket.

In the stack events, you will be able to find the status of the stack creation. It should be CREATE_COMPLETE.

Our S3 bucket should be created and ready for use!

Deploying the Lambda Application Source Code

We already have the source code from the cowsay application we developed previously. Let's deploy this code to the S3 bucket. The AWS SAM CLI provides some commands to automatically zip up the source code and upload it to an S3 bucket. This time, let's do it manually, so that you can understand what happens behind the scenes.

Zip up all the files in your source folder. (!) Make sure not to zip up the folder containing the source code, as this will not work. With our current configuration, AWS expects the index.js file to be in the root of your source code package. Call the compressed file, cowsay.zip, as defined in our Lambda S3Key property.

Let's head over to the S3 bucket using the AWS console. Upload the cowsay.zip file to the root of your cowsay S3 bucket.

Upload the cowsay.zip to the cowsay S3 bucket

Creating the Lambda Function Stack

Our code is available and ready to be run for our Lambda! Let's create a new CloudFormation stack for our Lambda. Head over to the CloudFormation stack designer. Drag the S3:BucketPolicy, Lambda:Function, and IAM:Role objects onto the editor.

As done previously, press the Create stack button and follow the steps until it is created. Call the stack cowsay-lambda.

Successful creation of the cowsay-lambda stack

We can now go to Lambda in the AWS console, find the cowsay Lambda and do a test invocation. This should be working as expected.

That looks great indeed! But where's the URL to our Lambda? Lambdas don't have URLs by default. Instead, we'll need to setup an API Gateway on AWS which will route to our Lambda function. Our next guide will be hands-on with API Gateways, how to set them up manually and with CloudFormation.

Summary

CloudFormation templates are very verbose ways for configuring infrastructure with text instead of manually. It makes it easier to automate infrastructure or create multiple similar infrastructures. Next to that, the templates can be committed to Git, to enjoy the advantages of a versioning system.

Often developers do not have access to manually create infrastructures in professional environments, this is where CloudFormation comes in handy. It can even be used to create automated pipelines for CICD build and deployment plans.

In order to set up CloudFormation templates, the AWS Documentation pages come in handy, explaining which properties are expected and how they can be configured.

Full Application Code

If you ever get stuck during the hands-on guide, you can find the full application code below.

Application Code

./package.json

{
  "name": "cowsay",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "cowsay": "^1.4.0"
  }
}

./index.js

const cowsay = require('cowsay');

exports.handler = function(event, context) {
    // Get a cow from cowsay!
    let cow = cowsay.say({
        text: "I'm a lambda moooodule",
        e: "oO",
        T: "U "
    });

    // Replace line breaks to HTML line breaks
    cow = cow.replace(/[\n|\r|\n\r]/g, '<br />');

    // Create the HTML
    var html = '<html><head><title>Cowsay Lambda!</title></head>' + 
        `<body><h1>Welcome to Cowsay Lambda!</h1>${cow}</body></html>`;
    
    // Return the HTML response
    context.succeed(html);
};

S3 Bucket Stack

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Metadata": {
        "AWS::CloudFormation::Designer": {
            "77235b77-da08-4103-993c-a41c3fa91784": {
                "size": {
                    "width": 60,
                    "height": 60
                },
                "position": {
                    "x": 230,
                    "y": 250
                },
                "z": 0,
                "embeds": []
            }
        }
    },
    "Resources": {
        "cowsaybucket": {
            "Type": "AWS::S3::Bucket",
            "Properties": {
                "BucketName": "cowsaybucket"
            },
            "Metadata": {
                "AWS::CloudFormation::Designer": {
                    "id": "77235b77-da08-4103-993c-a41c3fa91784"
                }
            }
        }
    }
}

Lambda Stack

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Metadata": {
        "AWS::CloudFormation::Designer": {
            "0fa2ecf2-821f-46ab-ac83-2d84e97c0c96": {
                "size": {
                    "width": 60,
                    "height": 60
                },
                "position": {
                    "x": 20,
                    "y": 250
                },
                "z": 0,
                "embeds": []
            },
            "0b834445-1d41-4189-a6fc-b233ff71813a": {
                "size": {
                    "width": 60,
                    "height": 60
                },
                "position": {
                    "x": 160,
                    "y": 250
                },
                "z": 0,
                "embeds": []
            },
            "b2461de9-1013-4eff-a448-9a6f04f3acab": {
                "size": {
                    "width": 60,
                    "height": 60
                },
                "position": {
                    "x": 20,
                    "y": 140
                },
                "z": 0,
                "embeds": []
            }
        }
    },
    "Resources": {
        "cowsayLambda": {
            "Type": "AWS::Lambda::Function",
            "Properties": {
                "Code": {
                    "S3Bucket": "cowsaybucket",
                    "S3Key": "cowsay.zip"
                },
                "Handler": "index.handler",
                "Runtime": "nodejs14.x",
                "Role": {
                    "Fn::GetAtt": [
                        "cowsayIamRole",
                        "Arn"
                    ]
                }
            },
            "Metadata": {
                "AWS::CloudFormation::Designer": {
                    "id": "0fa2ecf2-821f-46ab-ac83-2d84e97c0c96"
                }
            }
        },
        "cowsayBucketPolicy": {
            "Type": "AWS::S3::BucketPolicy",
            "Properties": {
                "Bucket": "cowsaybucket",
                "PolicyDocument": {
                    "Statement": [
                        {
                            "Effect": "Allow",
                            "Principal": {
                                "Service": "serverlessrepo.amazonaws.com"
                            },
                            "Action": "s3:GetObject",
                            "Resource": "arn:aws:s3:::cowsaybucket/*",
                            "Condition": {
                                "StringEquals": {
                                    "aws:SourceArn": {
                                        "Fn::GetAtt": [
                                            "cowsayLambda",
                                            "Arn"
                                        ]
                                    }
                                }
                            }
                        }
                    ]
                }
            },
            "Metadata": {
                "AWS::CloudFormation::Designer": {
                    "id": "0b834445-1d41-4189-a6fc-b233ff71813a"
                }
            }
        },
        "cowsayIamRole": {
            "Type": "AWS::IAM::Role",
            "Properties": {
                "AssumeRolePolicyDocument": {
                    "Version": "2012-10-17",
                    "Statement": [
                        {
                            "Effect": "Allow",
                            "Principal": {
                                "Service": [
                                    "lambda.amazonaws.com"
                                ]
                            },
                            "Action": [
                                "sts:AssumeRole"
                            ]
                        }
                    ]
                },
                "Path": "/",
                "Policies": [
                    {
                        "PolicyName": "root",
                        "PolicyDocument": {
                            "Version": "2012-10-17",
                            "Statement": [
                                {
                                    "Effect": "Allow",
                                    "Action": [
                                        "logs:*"
                                    ],
                                    "Resource": "arn:aws:logs:*:*:*"
                                }
                            ]
                        }
                    }
                ]
            },
            "Metadata": {
                "AWS::CloudFormation::Designer": {
                    "id": "b2461de9-1013-4eff-a448-9a6f04f3acab"
                }
            }
        }
    }
}

Resources