Kubernetes Benchmark: Bigstep vs AWS [part 1]

Setup and Methodology

The purpose of this benchmark is to compare Kubernetes performance on two different cloud providers, one offering a bare metal IaaS while the other a virtualized IaaS. We will not get into details on micro-services architectures, benefits of container-based application and virtualization impact on performance, but we recommend these resources for research.

The entire benchmark and tests should be easily reproducible on any other cloud provider or on premise, and for this purpose, we uploaded all the materials needed for this experiment. 

Hardware specification:

*hyperthreaded
**Dedicated EBS Bandwidth: 2000(Mbps)

For Bigstep we used the UK data centre and the CPU is Intel(R) Xeon(R) CPU E5-2630 v4 (@2.20GHz) while for AWS we used the Ohio region and the CPU used is 2.4 GHz Intel Xeon E5-2676 v3 (@ 2.40GHz). Despite the fact that it’s not the same type of processor and we have 4 additional threads in Bigstep, the CPU frequency is higher in AWS instances. Even though the instances are not identical, the purpose is not to compare processor generation but rather equivalent instances in terms of resources and price.

Cluster deployment

Both providers offer Kubernetes as a managed service, taking away the burden of managing docker, kubernetes and operating systems. Here we measure the time it takes to create a container platform (we assume the account is already created, and in AWS a VPC and at least 2 subnets are also created):

The amount of time and skills required is greater for AWS setup. In Bigstep the setup implies to choose the instance type, set the docker image and hit deploy, all under one minute and the rest of the time is spent booting the actual servers and starting the OS.

Network Latency

Network latency is the most important aspect in a high-performance container environment. Without the hypervisor’s overhead, the bare metal environment should have better performance compared to virtual environments, the question is by how much.

We used the open source netperf utility for testing latency. It is maintained by the Kubernetes community and easy to deploy on a cluster.

It uses kubectl to start the containers on both AWS communication and Bigstep Bare Metal nodes of the cluster. 

This is a sample of the results:

  Container Platform

  Elastic Container Service

As we can see in the above charts there is significantly lower network latency (almost x6) and 5 times more network throughput on Bigstep bare metal container platform. Most applications depend on network throughput and latency to offer a good user experience: web apps, streaming platforms or distributed applications all depend on the nature of inter-container communication. From this perspective, one should really take into consideration running containers on bare metal servers as it offers up to 5 times more network performance.

Filesystem Write Performance

Another important aspect in a high-performance container platform is the filesystem read/write capacity. One simplistic approach to testing filesystem performance is using dd to write a bulk of zeros in chunks of 1M or more. We used simple-container-benchmarks container for the test and monitor it using dstat.

This is a sample of the results (2 runs for 1G and 2G):

 Container Platform

# dd bs=1M count=1024 if=/dev/zero of=~/simple-container-benchmarks-writetest conv=fdatasync

1024+0 records in

1024+0 records out

1073741824 bytes (1.1 GB) copied, 5.38783 s, 199 MB/

# dd bs=2M count=2048 if=/dev/zero of=~/simple-container-benchmarks-writetest conv=fdatasync

2048+0 records in

2048+0 records out

2147483648 bytes (2.1 GB) copied, 14.0173 s, 153 MB/s

  Elastic Container Service

# dd bs=1M count=1024 if=/dev/zero of=~/simple-container-benchmarks-writetest conv=fdatasync

1024+0 records in

1024+0 records out

1073741824 bytes (1.1 GB) copied, 7.33642 s, 146 MB/s

# dd bs=2M count=2048 if=/dev/zero of=~/simple-container-benchmarks-writetest conv=fdatasync

2048+0 records in

2048+0 records out

4294967296 bytes (4.3 GB) copied, 32.3296 s, 133 MB/s

As we can observe, the performance of containers running on bare metal is between 15% and 35% higher when we measure throughput in MB per second and twice as fast when measuring the time to write a 2GB file to disk.

Another relevant test for most applications is Sysbench, a multi-threaded benchmark tool used for database workloads.

Closing Note

The scope of these benchmarks remains to create an objective opinion on running containers on bare metal servers or virtualized servers with a focus on performance and cost. The scripts, containers and detailed explanations are available on Github and it should be easy to reproduce the tests on other environments.

As we gathered from the tests, running containers on bare metal servers offers up to 5 times more network performance; when we measure throughput in MB per second, the performance of containers running on bare metal is also between 15% and 35% higher.

In the second part we will cover sysbench tests on mysql database and complex workload that involve database and applications communication as well as provide a cost/month analysis.