When planning your architecture for AWS, choosing where to put development environments can often be a tricky question. In this post, we present some benefits of local development environments, some misconceptions/myths of local and cloud development environments, some examples of mixing the two (such as site to site VPNs) and a final, strong recommendation for keeping your development environment in AWS.
We then present some best practices that can help manage an AWS-based development environment, such as how to structure VPCs, tools to manage AMIs, and how to use CloudFormation templates to give developers pushbutton access to create new environments.
To conclude the post, we present some next steps you could take with a development environment in the cloud, such as how you would be well-positioned to build CI/CD pipelines, Blue/Green deployments or even bespoke environments for user-acceptance testing or performance testing of a specific feature.
As mentioned above, choosing where to put development environments when planning your architecture for AWS can often be a tricky question. There are many different opinions and competing best practices to evaluate when considering the question.
We propose using local developer machines to write code and run unit tests, while doing end-to-end testing of your application in the cloud on AWS. With the rise of desktop virtualization and automation tools like Vagrant in the late 2000s, local development environments on an individual developer’s workstation have become the de-facto standard. loud computing also began to take off in the late 2000s, and it did not take long for developers to combine cloud computing with many of these existing local development tools. Today, building development environments within cloud providers offers developers the best of both worlds.
“Anyone who’s spent time studying or thinking about design won’t be surprised to hear that architecture has a powerful impact on software quality and developer productivity. Our survey analysis shows that specific architectural characteristics correlate with high IT performance: ability to test without an integrated environment; ability for developers to get comprehensive feedback from automated tests; ability to deploy an application independent of other services it depends on; and use of a microservices architecture.” – Puppet Labs 2015 State of DevOps Report
The analysis goes on to show that specific architectural characteristics correlate with high IT performance: “ability to test without an integrated environment; ability for developers to get comprehensive feedback from automated tests; ability to deploy an application independent of other services it depends on; and use of a microservices architecture.”
Our most effective customers implement both continuous integration and continuous delivery, and treat all infrastructure as code. We’ve seen similar conclusions from the 2014 and 2015 State of DevOps report, indicating that high performing organizations incorporate these same best practices. While it can be difficult to implement continuous integration and delivery from scratch, this is another area where cloud providers offer some pre-built solutions that rival what most people are able to build. Specifically, tools like Code Pipeline, “shepherd your code through the staging, testing, and release process.”
If your organization does not already have source control (another major pillar of high performing IT organizations according to the State of DevOps report), you’ll find it’s also very easy to add source control hosting via Code Commit. The value of these tools cannot be understated, as they embody the idea and facilitate the implementation of treating infrastructure as code and implementation of code testing and deployment pipelines. These tools allow you to shift your focus from IT engineering to delivering value to your customers.
It’s important to recognize that most of the familiar local development environment automation tools already support pluggable backends. From Vagrant to Eclipse, it’s actually quite easy to configure them to build a cloud server and deploy to a remote cloud server. Developers can still use their familiar tools and back them with cloud APIs instead of local resources. Using these same tools means that developers can still push code just as quickly, only limited by bandwidth and connectivity.
And development environments in Amazon also enable developers to use some of the newest tools in our industry, such as building microservices using containers (and docker-compose), creating reproducible VM image with Packer, and even the ability to leverage some of these tools to move between local and cloud development environments in a way that simply wasn’t possible until now.
Using a cloud development environment can also improve provisioning speed as many dependencies can be transferred faster from data center to data center than to a workstation on an office LAN, and with tools like Amazon’s Virtual Private Networks, a developer workstation can actually be placed on the same network segment as AWS resources. Network architecture and provisioning is entering a new era with cloud providers starting to offer more software defined networking options. With the ability to template many of these network architectures, provisioning development environments has changed from a manual, error-prone task to a push button, repeatable and easy task.
Amazon’s best practices also include the use of the Relational Database Service (RDS), ElastiCache, and many other cloud services. These are difficult to truly reproduce in local development environments, which is why we strongly recommend unit testing when implementing development environments in the cloud. Unit testing catches errors earlier, makes changes easier, and can even serve as a formal design or documentation process.
Problems not caught earlier in the development cycle tend to cost much, much more to solve later after the bugs are released into production code. While this may also mean you must stub out more functionality locally with unit tests, this is actually a benefit as you will be spending more time focused on your software than on how to configure dependent services on a developer workstation.
Finally, unit testing doesn’t require connectivity; well-designed software should allow for much of the initial design and implementation to be performed by developing code that can pass unit tests offline.
“AWS has long offered services that address the need to reliably and efficiently automate the creation of an environment. Services like Amazon EC2 and AWS CloudFormation allow infrastructure to be managed as code. Through the CloudFormation service, AWS resources can be provisioned declaratively using JSON. CloudFormation templates can be versioned right alongside the application code itself. Combined with the automation capabilities of EC2, this allows for a complex environment to be spun up and torn down quickly and reliably. These are just some of the reasons why AWS is such a good choice for development and test workloads.”– Amazon Web Services Startup Program
For end to end testing and building your production environments, we recommend the use of CloudFormation templates across all environments. Cloud Formation templates build on existing ideas about infrastructure automation and extend them to the cloud, from computing to networking to SAAS services. Templates should be provided internally to developers and integrate with any configuration management systems in use.
Combining many of the tools and services we’ve already covered here, Cloud Formation allows anyone in your organization to stamp out multiple, identical environments. These open up all kinds of new possibilities – from local development environments to additional integration testing environments to things like facilitating more developer innovation and experimentation or A/B user testing of a particular feature. A fully templated environment also enables blue-green deployment strategies, that enable you to build an entirely immutable infrastructure. These templates allow you to build production-identical environments quickly and accurately to enable more true-to-life performance testing. We strongly recommend adopting CloudFormations as a best practice for all environments.
With Fanatical Support for AWS, we can also help you avoid many of the pitfalls of taking your development environments into the cloud. We can help you control cost with special instance types and by helping you remove infrastructure that you maintain in-house.
These best practices save cost to onboard additional developers, yield better performance testing and find bugs before software is released. And when you need to get under the hood to troubleshoot, we have certified AWS Solutions Architects available to assist you.
Visit http://www.rackspace.com/aws for more information about Fanatical Support for AWS and how it can help your business. And download our free white paper Best Practices for Fanatical Support for AWS, which offers further detail regarding implementation options for AWS, including identity management, auditing and billing.