Building a Serverless Architecture with AWS Part 3

In my two previous blog posts, I’ve covered serverless hosting and serverless authentication through the use of Amazon tools such as S3 and Cognito User Pools. In this post, we’re going to continue the serverless theme by adding a few more technologies to give us server-like functionalities.

In this post, I’ll explore micro-services through the use of Lambda and Dynamo DB. I will also explore security options on AWS by implementing federated security through the use of Cognito Federated Identities. If you have not read parts one and two of this series, strongly suggest going back and doing so before you begin. The code in this post will be building off what has already been done in those posts. Also, this post assumes you have access to an AWS account.

Note: launching services such as Lambda and DynamoDB incur a per-use charge. Please be aware of the charges as you follow this demo.

The Project

As I began to plan for this post, I was trying to think of how best to show the power of Lambda. The first thing that came to mind was an image processor. But, apparently, the first thing that comes to everyone’s mind is an image processor. So instead, we are going to build a document processor. This processor will allow us to create simple documents using markdown. We can then create pdf’s of these documents and save them to an S3 bucket. Upon saving these documents, we will then trigger a Lambda to add these new documents to a database under the user’s name.

Serverless Architecture

Preliminaries

As always, there is some preliminary work we need to do to get our infrastructure ready. We are first going to get all of our services set up on AWS, and then we will start walking through our code.

Lambdas

Serverless Architecture 3The first thing we are going to do is get our Lambdas ready. Let’s create three Lambdas called serverless-pdf, serverless-monitor and serverless-query. To do this, we need to log in to our AWS dashboard. On the AWS dashboard choose Lambda, which is found in the Compute section. If this is your first time here, click Get Started Now, if you already have some Lambdas, then just click the Create a Lambda function.

On the next screen, choose Skip in the lower right-hand corner. We are not going to use an existing template, instead, we’re going to build our own. Fill out the appropriate information as follows.

Serverless Architecture 3

And for the Lambda Code Function, we are just going to put a comment for now. Later we will be updating our code. For now, this is just a placeholder.

Serverless Architecture 3

For the function handler section, leave everything as is except for the role.

Serverless Architecture 3

Serverless Architecture 3To get the proper role in place, we need to take a few steps. First, we need to understand what this Lambda will do. The purpose of this Lambda is to create a PDF from the submitted HTML, and then save that PDF to an S3 bucket. So we need a Lambda that can put an object (putObject) to an S3 bucket.

On the Role drop-down menu, choose S3 execution role. When you do that, another tab is going to open up and ask you to allow Lambda to access S3. Take all the defaults on this page and click Allow. You will now be taken back to your Lambda creation page. Click Next to review your details and then click Create Function. There you have it! You have created a Lambda placeholder that we will later update with code to create a PDF.

We still need two more Lambda placeholders, one for our serverless-monitor and one for serverless-query. Let me give you a brief overview of those and then you can follow the same instructions to create these Lambdas as well.

The serverless-monitor Lambda will be triggered from S3. When triggered, it will create an entry in a DynamoDB table with the new document information. So this Lambda will need access to create records in a DynamoDB table. When choosing the role, choose Basic with DynamoDB.

The serverless-query Lambda will act as an API endpoint for our website. Its job is to query the DynamoDB table for records and return them to the client. So this Lambda will need access to get records from a DynamoDB table. Use the same role that is created when you created your serverless-monitor Lambda. Make sure you do not choose a role under Create new role in the dropdown. Further down in the dropdown will be the roles you have created. This confused me the first time I saw it.

Serverless Architecture 3One thing to understand when creating these generic roles is that they are pretty wide open. Let’s take a look at the first generic role that we created when we created the serverless-pdf Lambda. We needed to be able to write to an S3 bucket.

Notice the second statement record in the Resource array. The listed resource is “arn:aws:s3:::*”. With this resource, our Lambda has access to write any S3 bucket in our account. This is fine for the purposes of our demo, but I can tighten this up by changing it to “arn:aws:s3:::mybucketname/*”. By changing this, it limits my Lambda to only the bucket I want.

Before we move on, take a moment and grab the arn for the serverless-pdf and serverless-query Lambdas.

Serverless Architecture 3

Cognito Federated Identities

Now that we have our Lambda’s in place, let’s go ahead and build out our federated identities through Cognito. In my previous blog post, I showed you how to use Cognito User Pools as a serverless authentication solution for your site. However, we now need to extend that authentication to include AWS resources as well. Cognito Federated Identities works hand in hand with Cognito User Pools to do just that.

In these next steps, I am going to show you how to link the user pool we created in the last blog with a federated identity pool provided by Cognito. By doing this, our user pool becomes the authentication provider for our identity pool which allows us to assume an authenticated role that gives us access to our AWS resources. Clear as mud? Yeah, it’s a little confusing. Let’s look at the authentication flow.

Serverless Architecture 3

Ok, let’s walk through setting this up. Click the Services menu on the top left and choose Cognito. The first thing we need to do is get a little information about the user pool you set up. Click Manage your User Pools and click on the pool you created in the last blog. Mine is serverless. On the Pool Details tab, copy the Pool Id and save it somewhere handy. On the Apps tab, grab the App client id and save it as well.

Next, we move over to the Federated Identities. Click on the Federated Identities tab in the upper left. If you don’t have any pools already, click Manage Federated Identities to create your first. If you have one already, click Create new identity pool to create a new one. For the identity pool name put serverless. Be sure to leave the Enable access to unauthenticated identities unchecked. We want all our traffic authenticated.

Serverless Architecture 3

Click on the Authentication Providers to expand it. On the Cognito tab, fill in the User Pool Id and App Client ID with the information we copied from our user pool.

Serverless Architecture 3

Serverless Architecture 3 Click Create Pool. The next screen is going to ask you to setup some roles. We are going to setup two roles here. An authenticated role and un-unauthenticated role. If you remember the diagram from above, in Cognito we can assume either of these. It is here that we decide what an authorized user can or cannot get to, and what an unauthorized user can or cannot get to. Click View Details to expand the policy section.

You may be asking yourself, “why do we have to do an unauthenticated identity?”. Good question, above we made sure to uncheck unauthenticated access, however, AWS still requires that we create the role. We are going to take the defaults on both IAM Roles and Role Names. The only thing we are going to edit is the policy for the authenticated identity. Click View Policy Document to expand the policy editor.  Click Edit Replace everything in the policy with the following JSON document. Be sure and add the arns for your serverless-pdf and serverless-query Lambdas under resources.

Click Allow to finish. That’s it, your Cognito User Pool and Cognito Federated Identity Pool are now tied together. Take note of your Cognito Identity Pool ID, as you will need that later.

Dynamo DB

The last piece of the preliminaries puzzle is to create a DynamoDB table to hold a record of our created documents. In the top left corner click the Services dropdown and choose DynamoDB. Click Create Table to create your first table.

Serverless Architecture 3

Fill in the information as shown above. Be sure and check Add sort key to add the sort key field. Click Create to create your table. That’s it, fairly painless.

S3 Bucket

The last step we need to take is to create a bucket for our Lambda to place the pdf files. This bucket is going to need an original name. Mine is called serverless-objects, so that is taken. Let’s walk through the steps if you have never done this before. Click on the Services dropdown and choose S3. Click Create Bucket and fill in the form. Choose your unique folder name and pick where it will be located. Click Create to finish.

Serverless Architecture 3

Now that we have a bucket setup to hold our pdf files, we need to set it up so that people can read the PDF files. For the purposes of this demo, we are going to take the easy route. We are going to create a hosted website. If you remember, we did this in the first post to host our client website.

Click on the box next to your bucket name. Click Properties on the left side. Expand the Static Website Hosting section and set your Index Document to be index.html. Understand, this is not the most secure way to do this. Buckets can be securely accessed with a signed certificate. However, that falls outside the scope of this blog post. Be sure to click Save to commit your bucket changes.

Serverless Architecture 3

Serverless Architecture 3Ok, our bucket is set up and publicly available. Let’s go ahead and set up our trigger. The trigger we are going to create will call our serverless-monitor Lambda every time a document is added or updated. Further down on this same page is a section called Events. Click Events to expand the section. Name your event and in the dropdown choose ObjectCreated(All). After choosing the Events you can then fill in the rest of the information

Serverless Architecture 3For the Prefix we will use ugc. This will keep the trigger limited to the ugc subfolder our Lambda will create. We want to Send To a Lambda and choose our serverless-monitor lambda. Click Save to apply your changes.

Ok, that wraps up the AWS preliminary setup. We’ve added our Lambda placeholders, updated our authentication model to include federated authentication into AWS, created a table to hold data about our documents and created an S3 bucket to hold our documents and trigger our lambda. Not a bad day. Our next step is to get our code in place.

The Code

In our previous blog posts, we worked with client-side code that we’re hosting in an S3 bucket. In his post, we are going to continue working with that code as well as introduce new code to run as serverless Lambda functions. Because of the amount of code, I will not be breaking it down as much as I will be giving overall examples. However, I will point out specific complexities.

Lambda Code

We have three Lambdas that require code updates. Let’s look at the first, which is the serverless-pdf lambda. The code for this is a modification of project aws-lambda-wkhtmltopdf found at https://github.com/lubos/aws-lambda-wkhtmltopdf.

This package includes some advanced topics of Lambda, including recompiling code for Amazon Linux, and resetting the Lambda Path. For now, it is not necessary to understand these concepts, but I wanted to give you a heads up.

Because of the size of this code, we’re going to choose one of the lesser-used options to get our code to our Lambda. As a first step, download the package to your local computer from http://serverless-assets.s3-website-us-east-1.amazonaws.com/serverless-pdf.zip.

Unzip the package and open the config.js. On the first line, there is a property called dstBucket. Set this property to be the name of your S3 bucket. Zip the files back up as they will need to be compressed for the upload.

Next, you are going to upload this compressed file to an S3 bucket on your account. Once you have the file uploaded to your bucket, get the S3 address to the file under the properties for that file. Make a copy of this address, we will need it shortly.

Serverless Architecture

Ok, let’s update the code of our serverless-pdf Lambda. Return to the Lambda dashboard and click on the serverless-pdf function to enter into it. Choose Upload a file from Amazon S3. In the field, place the address to the zip file you uploaded. Click Save. Lambda will grab the file from S3, unzip it and put everything in place. That’s it, your Lambda is now updated with the proper code.

Serverless Architecture 3

Now that we have knocked out the hardest one, updating the other two Lambdas will be easy. Let’s attack the serverless-monitor Lambda next. From the Lambda dashboard, click on the serverless-monitor function to enter the function. Choose Edit code inline and place the example code found below.

Serverless Architecture 3

The serverless-query is also a simple inline edit. Update the code for the serverless-query Lambda with this code.

The code for these Lambdas is pretty self-explanatory. Take some time to look through them and understand what they are doing. On the serverless-monitor, you will see event.Records. This is the array that a service sends when it is triggering a Lambda.

In the serverless-pdf and serverless-query Lambdas, you will see the use of context.identity.cognitoIdentityId. The context.identity object provides us with information about the unique user that is making the call. By using this information, the Lambda can determine the user from context and does not require the user to submit user info that can be falsified.

Client Code

Now that we have our micro-services in place in the form of Lambdas, we can turn to the client code and update it to consume our services. We need to deal with our expanded authentication first, and then write some code to take advantage of our new micro-services. Let’s tackle our authentication code.

Global Settings

The first change we need to make is in our global settings. In this change, we add in the AWS.config along with the AWSCognito.config. We also configure our credentials with our identityPoolId for both services. You can find this number in the Cognito Federated Identities dashboard under your identity pool.

The Auth Library

Our next step is to modify our Auth library to handle the updated authentication protocols. To handle the identity pools we will need a few extra parameters.

You’ll notice we have added the identityPoolId and the region. And in the constructor, we construct a loginId. You will see the all of these in use when we set the credentials for the identity pool.

To handle authentication, we’ve added a function called setCredentials(). When logging in, or refreshing the session, this function is called upon in successful completion. This function takes the resulting token and builds a credentials package for the federated login.

One thing worth noting is that this does not actually ping Cognito to verify the federated credentials. If you are monitoring your traffic when you run this, you will not see an API call after the call to the user pool for the original authentication. It is not until you call your first service that the federated authentication will be verified.

The next modification we make is in the loginUser() function. We have not changed this to make a call to the setCredentials() function once the token has been returned.

The same modifications have been made to the getSession() function. In fact, the getSession() has been rewritten. The purpose of the rewrite was to ensure the function returns a promise.

The reason for this is a bit complicated, but I think it is worth discussing. To best explain this, we need to look at it in the scope of a SPA application. In a SPA application, be it Angular, Knockout or Aurelia, you generally load up a main page, and the rest of the pages are loaded into this shell. This is a bit oversimplified but works for our purposes. If the user navigates around in a SPA, only part of the page is being updated and the rest stays the same. This allows us to have a state, of sorts.

However, when a SPA is refreshed by the browser, it needs to load everything back up again. This is when we would use the getSession() function to get the current user, if they exist. However, the call to refresh the token is an asynchronous call. Meaning, the call is made and the script moves on. So, if you refresh a page that makes a call to an API, it may or may not fail, depending on if your session is back or not. This is what we call a race issue and can be difficult to track down.

To ensure we do not have a race issue, we need a way to make sure the session is back before any other functions are called. This is where the promise comes in really handy. This is also one of the reasons I use Aurelia. Aurelia has a mechanism to handle this very issue quite beautifully. In the App.js of my demo SPA, which is the main page of the SPA, there is a function called activate() that looks like this.

The activate() is part of the Aurelia lifecycle and is called before the page is rendered to the browser. In this function, I am returning a Promise. Aurelia knows not to continue until my promise has either resolved or been rejected. Once it is complete, then the page is rendered and sub-pages are built. The bottom line is, no other API call is made until we know if we have a session or not.

Seeing it In Action

To demo this site and try it out, please go to http://serverless-architecture-federated.s3-website-us-east-1.amazonaws.com and register for an account. Screen captures used here will be from this site.

You can also download the entire code base at https://github.com/rackerlabs/serverless-demo/releases/tag/Blog-3. Follow the direction in the readme.md to get started.

Now that files updated. Let’s take a look at some of the client code that uses the micro-services. In the attached repo there is a file at src/documents/index.js. Let’s take a look at this to see what is going on.

This class has several functions in it. In the translate() function, the markdown value is converted to html using the showdown library. As the user types, their text will be translated on the fly.

The createPDF() is the main function I want to discuss. This is an example of calling a Lambda using the AWS-SDK for JavaScript. When invoking we have to pass a FunctionName and a JSON.strinigified payload. When called, the SDK will first ensure that this call is authorized. It will then call the invoke API and pass the function information. When the serverless-pdf Lambda receives this call, it converts the HTML to a pdf document and saves it to S3 storage. The API then responds with the id and document name. These are then inserted into our files array. This is a temporary array of the files created in this session. If you leave the page and return, those files will no longer be reflected here.

Serverless Architecture 3

To see a historical list of created documents we need to head on over to our profile page. Clicking my username in the upper right will take me there.

Serverless Architecture 3

Based on the last file we just looked at, the code for this page will be pretty straight forward and can be viewed at src/profile/index.js in the attached repo.

Wrap Up

There you have it. In the last three posts, we’ve covered a full-stack of serverless technologies on AWS. I encourage you to download the repos and play around. I hope this has wet your appetite for developing against AWS and doing it without servers!

https://github.com/rackerlabs/serverless-demo/releases/tag/Blog-3

LEAVE A REPLY

Please enter your comment!
Please enter your name here