Performance is a key consideration in any infrastructure decision and an important topic for our customers, and therefore an important subject for Rackspace. Keeping a focus on performance and providing a high performance-to-price ratio is important for any cloud provider.
This post examines the role of benchmarks as tools to evaluate the performance of a system and the importance of proper design and best practices to give applications the performance that is required by the business.
A number of companies have published benchmarks that try to establish the performance of public clouds with varying conclusions. Some of these results show Rackspace infrastructure delivers higher performance than other options, while others show the opposite. It’s not that some benchmarks are wrong and some are right, it’s simply that benchmarking is complex and measures specific aspects of performance under very particular conditions.
In a way, benchmarks are like statistics: they can be manipulated to support almost any argument. In general, tests will produce different results depending on what is being tested and the design of the test. For example, an independent test on cloud storage services found networking bottlenecks that impacted the results heavily; yet customers are highly unlikely to hit these networking loads in any real world scenario. These test results are not useful for most practical scenarios.
Another example of how the design of a test invariably produces certain bias is in the testing of object storage; the architecture for the product being tested is optimized for multi-container performance, while the test was designed for a single container. This does not mean that the test was wrong or invalid; it simply means the test produced a performance indicator for a specific configuration and use case; therefore benchmark conclusions should not be generalized.
Another challenge with benchmarks is that they can imply an assumption that performance is the most important aspect of a technology, which is not always the case – it depends on the use case. Performance benchmarks are useful not in providing an absolute point of view, but a general ballpark of what to expect in a specific setup. This brings up a more important point: performance is impacted greatly by architecture – designing and optimizing an application can be complex, and requires skills and expertise that not all organizations have at hand.
If the goal is to maximize performance, evaluating performance benchmarks can be useful, yet it is more important to make sure you have the right architecture, that you are using the right tool for the job and that you have the right configuration. You may be thinking of using the wrong technology to solve your problem.
This is especially true today for two reasons: First, the technology alternatives that can be used to solve a particular problem are growing. Just look at database engines as an example: MongoDB, Cassandra, CouchDB and Redis have joined the SQL RDBMS we have trusted for years, and each one of these new offerings presents new opportunities, advantages and tradeoffs.
The cloud has brought more deployment models: dedicated (physical), virtualized, public cloud and private cloud – and a combination of all these in a hybrid cloud. These provide more options and new opportunities to get higher performance for your applications.
Second, the cloud has introduced new ways of building IT solutions that often require a new skill set. It is always an advantage to get help from someone who has the experience of having designed similar systems before.
This is why many organizations trust Rackspace to help them design the infrastructure that supports their applications properly, and choose the right technology for their particular needs. Fanatical Support is about taking our years of experience and expertise in building high-performance systems to help our customers design, build and operate their applications for the right performance and we guide customers though these decisions every day.
In conclusion, here are my recommendations for optimizing performance:
- When looking at benchmarks recognize the conditions under which the data is valid. Understand the test parameters and the design.
- When possible, look for application benchmarks, as they provide more contextual validation of infrastructure under a particular load.
- Take advantage of best practices in architecting for performance – for example by relying on the expertise of Rackspace and services such as monitoring and optimization.
- Focus on performance value: the performance ratio you get from resources invested. You can learn more about this topic on this post from my colleague J.R. Arrredondo.
You can expect Rackspace to continue our focus on optimizing performance for applications in the hybrid cloud and using benchmarks as tools to assess our progress. We plan to to publish results and have a deeper conversation on performance at www.rackspace.com/performance.