Comparing Cloud Compute Services.

Jason Read at CloudHarmony does a great job sorting through all of the different cloud computer offerings by vendors and then compares their performance in a like-to-like benchmark.  This is no small task and most reports I have read comparing cloud compute do not do a good enough job of comparing similar services when conducting their performance tests.  You can obtain a copy of the full report from here which is 100+ pages long.

Below are some the test results along with my own comments as well.

Compute Service Provider 
Web Server ComparisonSmall Web ServerMedium Web ServerLarge Web Server
Amazon EC2 (Instance Types Explained)c3.large + t2.mediumc3.xlargec3.2xlarge
DigitalOcean (Types Explained)4 GB / 2 Cores8 GB / 4 Cores16 GB / 8 Cores
Google Compute Engine (Types Explained)n1-highcpu-2n1-highcpu-4n1-highcpu-8
Microsoft Azure (Tiers Explained)Medium (A2)Large (A3)Extra Large (A4)
Rackspace (Types Explained)Performance 1 2GBPerformance 1 4GBPerformance 1 8GB
SoftLayer (Types Explained)2 GB / 2 Cores4 GB / 4 Cores8 GB / 8 Cores
Database Server ComparisonSmall Database ServerMedium Database ServerLarge Database Server
Amazon EC2c3.xlargec3.2xlargec3.4xlarge
DigitalOcean8 GB / 4 Cores16 GB / 8 Cores48 GB / 16 Cores
Google Compute Enginen1-standard-4n1-standard-8n1-standard-16
Microsoft AzureLarge (A3)Extra Large (A4)A9
RackspacePerformance 2 15GBPerformance 2 30GBPerformance 2 60GB
SoftLayer8 GB / 4 Cores16 GB / 8 Cores32 GB / 16 Cores

Test Results

CPU Performance Results

In the web server test Amazon EC2 was a little better than the rest of the competition.  And in the database server test Rackspace was slightly better until the testing of large database servers where Azure’s new A9 server won out.

Also included in this test was CPU variability.  In these multi-tenant, shared resource environments the performance over time can be a risk to a customers if that is the case this test should be taken into consideration as well as the other variable tests included in the report.  For this test the lower the score the better, ideally you want the same performance over the life of the service.

In both tests Amazon had the best overall score across all server types.  The changing CPU types as testing across different server types should be considered when looking at these two scores as well.

Disk Performance Testing

In these tests Amazon EC2 and Rackspace were consistently faster and more reliable.  Since they are SSD based storage they should be faster and more consistent than the other services.  DigitalOcean is also SSD based but their performance was not on par with the other SSD based services and also had the highest rate of variability.  SoftLayer is not SSD based but their overall disk performance and consistency were very good.  This could be from using SSD caching but no matter how they are doing it the performance speaks for itself.  Microsoft Azure and Google do not offer SSD storage which is reflected in their testing.

 Web Server Tests

 

Memory Performance

In these test they conduct them only on the database servers and because of Azure’s older hardware based on AMD CPU and MOBO they did not perform well until the large database server test where they are using the newer Intel Sandy Bridge based platform.  Amazon and Google both perform well through all testing and outperformed the others.  Newer hardware is always going to win in these types of tests and that is reflected here.

External and Internal Network Performance

Network testing is always subjective and by testing cloud service providers networking throughput it just adds to the complexity of the configuration as some add additional ways to improve performance through different setups.  Amazon, Google and Rackspace seem to provide higher throughput then Microsoft Azure, DigitalOcean and Softlayer.  Also throughput by some vendors is limited depending on the size of the compute instance your purchase.

They also cover the value of each provider in their report which ultimately should be the deciding factor in choosing which cloud computing provider to use for your solution. After all there is no need to overspend on a service if you can accomplish everything you need with a cheaper solution.

You can download the full report here.