When developing a non-trivial application, an important early step is to decide what to do with your application secrets.
These can be API keys, database passwords, or other special configuration values your application needs to function, but that you don’t want everyone to have access to. Very often, developers wind up taking an insecure or difficult-to-manage (or both) approach to application secret storage, either due to time constraints or uncertainty around best practices.
In this article, we’ll learn about the best way to secure your application secrets — EC2 Parameter Store. But first, let’s take a look at a few less secure methods for managing application secrets that are still commonly used.
Storing plaintext secrets in source control
While this may seem like a convenient way to manage secrets, its crucial flaw is that even in a closed, private project, it exposes your application secrets to anybody with read access to the repository. This approach would certainly not pass an audit, and the security implications are considerable even if your project doesn’t have any formal compliance requirements.
Storing plaintext secrets on the server
Another approach is to store secrets on the server where your application is running. This can be in the form of an application configuration file that doesn’t get committed to source control. In the case of a web application, you can store secrets in the web server configuration to be injected into your application’s runtime as environment variables.
This is a safer approach than having plaintext secrets in source control, as it limits the number of people who would have access to them, but it has a big weakness that the previous approach doesn’t: it’s inconvenient.
Let’s say you have a dozen servers running the same application. How are you going to modify existing application secrets or create new ones? Logging in to each server individually would take a while and is very inconvenient. You could introduce some automation around configuration management, which would address how your application is going to access the secrets it needs to run and how you’re going to keep them up to date, but it doesn’t address where you’re going to store those secrets in the first place.
In other words, how is your automation going to know what secrets it needs to deposit on each server?
We already covered why you shouldn’t store them in your source control repository, and it wouldn’t make sense to only store them at their destination. So where would you store them? Somewhere else on your computer? That’s not necessarily secure either, but even if it were, how would other members of the team who should have access to those secrets get them? Perhaps everyone could get a copy when they change, but then how do you distribute them securely and make sure they’re all in sync? There are a lot of logistical problems with this approach.
So, what can you do?
Storing encrypted secrets in source control
If the problem is due to plaintext, then let’s start encrypting things, right? Well, yes, but it’s not that simple. If you used a modern, secure algorithm, then you alleviate a lot of problems with storing plaintext secrets in source control, but you encounter another problem — how do you decrypt the secrets? To decrypt an encrypted string, you’ll need the right key, but the key itself is also a secret that needs to be kept safe!
You can’t store the key in your source control for the same reason you can’t store the secrets themselves in source control. If you store the key on your servers, then you encounter the same maintenance complications you’d see if you were to store the secrets on the server. Even if you did have an approach that allowed you to store and easily maintain the decryption key (or the secrets themselves) on your servers, there’s still a problem — what if there are no servers?
Lambda, EMR, CodeDeploy, etc., are all “serverless” technologies available to you in the AWS portfolio of features and services. None of them are meant to run on any sort of permanent server that you would access in order to store your encrypted application secrets or the keys that unlock them. If your application stack involves any of these services, then all the approaches to application secret management we’ve covered so far are not going to work.
Managing secrets with Amazon EC2 Parameter Store
Now that we’ve gotten a few suboptimal approaches out of the way, let’s dive into the best approach – the Amazon EC2 Parameter Store, which provides a centralized store to manage your configuration data, whether it’s plain-text data such as database strings or secrets such as passwords, encrypted through AWS Key Management Service.
For this tutorial, let’s assume we have a web application that interacts with an external API endpoint, which requires an API key for use. The API key is meant to be secret and should not be shared, so you’re the only one who knows it right now. It needs to be stored securely, updated efficiently and easily accessible to your application.
Secrets managed by the EC2 Parameter Store are encrypted using AWS Key Management Service (KMS) keys, so the first thing we need to do is create a key. We could use the default aws/ssm key that AWS would create automatically for us, but creating our own key specific to the application that will be using it gives us more granular control over who can manage and access it. It also allows larger organizations with multiple applications to more easily limit visibility and management of application secrets to those teams responsible for those particular applications.
We’ll be adhering to AWS best practices and using IAM Roles on our EC2 instances to interface with the AWS API, so we need to create a new IAM Role. We could also simply attach the AmazonSSMReadOnlyAccess IAM Policy, but doing so would give the IAM Role access to read all SSM-related items, not just Parameters.
To start creating this custom IAM Policy, open the AWS Console and navigate to IAM -> Policies and click Create policy. Next, select Create Your Own Policy, and use the following Policy Document:
If you have additional parameters that you want to grant access to in the future, you can add their ARNs to the Resource list, or prefix all of their names with your application name. You can then specify
arn:aws:ssm:YOUR_REGION_HERE:YOUR_ACCOUNT_NUMBER_HERE:parameter/MyApplicationName-* as the ARN. Otherwise, enter MyApplicationPolicy for the Policy Name and click Create Policy.
Now, let’s create the IAM Role. Head over to IAM -> Roles and click Create new role, then choose Amazon EC2 for the AWS Service Role.
Choose Customer Managed as your policy Filter and select MyApplicationPolicy as the IAM Policy to attach, then click Next step.
Finally, name your role MyApplicationRole and click Create role.
Now it’s time to create our own key in KMS. Navigate to IAM -> Encryption Keys and click Create Key.
Next, enter an Alias for your key and an optional Description. For Key Material Origin, leave the default KMS selected.
Add some tags, if needed. For this example, I’ll tag this key with the application name and the environment in which it would be used.
Now we need to assign one or more key administrators. The key administrator is an IAM User or Role who is allowed to make changes to the key itself or grant others access to use and administer it. For this example, I’ve chosen my own user, but I could have also created an IAM Role that my IAM User could assume in order to execute administrative functions on the key. Depending on your organization, this might make management of key administrators easier, but for this example we’ll keep it simple and just use my IAM User. I’ve also left the Key Deletion option checked, which will allow me to delete the key in the future.
Next, we’ll Define Key Usage Permissions so that we can actually use our key. Here, I’ve selected the MyApplicationRole IAM Role as an authorized key user, which is the role I’ll be assigning to the EC2 instances my application is going to run on. I’ve also selected myself as an authorized user, as by default, the key administrator permissions do not include permissions to actually use the key. If desired, we could also give other AWS accounts permissions to access our key, which those account administrators could grant to individual IAM users and roles on their end.
The next page will display a preview of what your KMS key policy will look like. After looking it over and concluding that it’s fine, click Finish.
You will now be returned to the Encryption Keys page and find that the key was successfully created.
Encrypting our secret
Now that we have a KMS key that we can use to encrypt our secrets, let’s go ahead and actually create one!
We can do this using the AWS Console, but at this time, any secrets stored in that fashion will be encrypted with the default aws/ssm key. We don’t want to do this, because that would enable anyone who has access to that default key to view our secrets. Instead, we’re going to use the AWS Command Line Interface, which gives us the option of specifying which KMS key we’d like to use.
Note that if you already have an encrypted application secret with the selected name, the command will fail. If that’s the case (as it would be in the future whenever you need to change the value of the secret), you must add the –overwrite flag.
This command only displays output if there was an error, so if you see nothing after running it, you’ve successfully encrypted your first application secret!
Retrieving the plaintext value
Viewing our secret’s plaintext value in the AWS Console is quite easy. First, navigate to the EC2 Parameter Store dashboard and select the parameter.
Then, just click Show to expose the plaintext value.
Despite how easy it is to reveal the plaintext value, this information is only available to users and roles who have permission to use MyApplicationKey to decrypt values. If a user who doesn’t have the appropriate key permissions tries to view the plaintext value, the AWS Console will not reveal it to them.
Do note that there are some IAM managed policies, such as AdministratorAccess and PowerUserAccess that will automatically grant access to view EC2 Parameters and use the decryption keys. You can prevent PowerUserAccess users and roles from doing so by adding an Inline Policy to their assigned permissions specifically denying access to the EC2 Parameters or the KMS key (or both). This is not possible for AdministratorAccess, as that access level by definition also enables any such users to remove that restriction from themselves. As such, AdministratorAccess should be assigned judiciously.
The AWS Console makes it easy for humans to view a secret’s plaintext value, but what about an application? Let’s take a look at a quick example using the AWS SDK for PHP.
This example assumes that your AWS default profile credentials on your workstation have been correctly configured, or that it’s running on an EC2 instance with an IAM Role assigned to it that has access to the encrypted secret. You will also need to have the AWS SDK for PHP installed. Refer to the User Guide for more information on getting started.
Executing this sample code will retrieve the APPLICATION_SECRET_API_KEY stored EC2 Parameter, decrypt it using the associated KMS key, and return the plaintext string to you. For use in a real, Production application, we recommend caching these values in a local configuration file on the application server that can be refreshed as part of your code deployment process. Doing so eliminates the added initialization time at the start of a request used to retrieve the parameters from the EC2 Parameter Store. This ends up yielding the same benefits as the storing plaintext secrets on the server approach without the inconveniences.
If you don’t use PHP, there are AWS SDKs for a variety of other programming languages. Visit the AWS Developer Tools documentation for more information.
Naturally, you can also use the AWS Command Line Interface to retrieve plaintext values, much like it can be used to store them in the first place.
This command will output the key in the following format, and can be used to retrieve multiple keys simultaneously:
Visually, the workflow looks like this:
In this article, we covered some of the common, less-than-ideal ways to manage application secrets. We then learned how to do so in a secure, easily-manageable way with EC2 Parameter Store, and how to make this information available to our applications. Now you should be ready to head over to the AWS Web Console, fire up your favorite code editor and get started on integrating this approach into your applications. Good luck!
Do you have additional questions about EC2 Parameter Store or other AWS features and services? Visit Rackspace to find out more about how our AWS-certified experts are helping businesses get the most out of AWS.