ibm.com: An Updated Performance Comparison of Virtual Machines and Linux Containers

ibm.com: An Updated Performance Comparison of Virtual Machines and Linux Containers

Wes Felter, Alexandre Ferreira, Ram Rajamony, Juan Rubio
IBM Research Division
Austin Research Laboratory

Cloud computing makes extensive use of virtual machines (VMs) because they permit workloads to be isolated from one another and for the resource usage to be somewhat controlled. However, the extra levels of abstraction involved in virtualization reduce workload performance, which is passed on to customers as worse price/performance. Newer advances in container-based virtualization simplifies the deployment of applications while continuing to permit control of the resources allocated to different applications.

In this paper, we explore the performance of traditional virtual machine deployments, and contrast them with the use of Linux containers. We use a suite of workloads that stress CPU, memory, storage, and networking resources. We use KVM as a representative hypervisor and Docker as a container manager. Our results show that containers result in equal or better performance than VMs in almost all cases. Both VMs and containers require tuning to support I/O-intensive applications. We also discuss the implications of our performance results for future cloud architectures.

All of our tests were performed on an IBM System x3650
M4 server with two 2.4-3.0 GHz Intel Sandy Bridge-EP Xeon
E5-2665 processors for a total of 16 cores (plus HyperThreading)
and 256 GB of RAM. The two processors/sockets are
connected by QPI links making this a non-uniform memory
access (NUMA) system. This is a mainstream server configuration
that is very similar to those used by popular cloud
providers. We used Ubuntu 13.10 (Saucy) 64-bit with Linux
kernel 3.11.0, Docker 1.0, QEMU 1.5.0, and libvirt 1.1.1. For
consistency, all Docker containers used an Ubuntu 13.10 base
image and all VMs used the Ubuntu 13.10 cloud image.
Power management was disabled for the tests by using
the performance cpufreq governor. Docker containers were not
restricted by cgroups so they could consume the full resources
of the system under test. Likewise, VMs were configured
with 32 vCPUs and adequate RAM to hold the benchmark’s
working set. In some tests we explore the difference between
stock KVM (similar to a default OpenStack configuration)
and a highly-tuned KVM configuration (similar to public
clouds like EC2). We use microbenchmarks to individually
measure CPU, memory, network, and storage overhead. We
also measure two real server applications: Redis and MySQL.

We see several general trends in these results. As we expect
given their implmentations, containers and VMs impose almost
no overhead on CPU and memory usage; they only impact
I/O and OS interaction. This overhead comes in the form of
extra cycles for each I/O operation, so small I/Os suffer much
more than large ones. This overhead increases I/O latency and
reduces the CPU cycles available for useful work, limiting
throughput. Unfortunately, real applications often cannot batch
work into large I/Os.

E-Bike: Risiko bei Pedelec-Antrieben von Bosch

E-Bike: Risiko bei Pedelec-Antrieben von Bosch

Bei den Pedelec-Antrieben der ersten Generation (Classic+ Line) von Bosch können gravierende Probleme auftreten, wie der Hersteller einräumt. In einer „Service­aktion“ tauscht Bosch defekte Antriebe aus – aber erst, wenn die Störung bereits aufgetreten ist. Besser wäre ein vorsorglicher Rück­ruf, bei dem der Antrieb im Fach­betrieb kontrolliert und vorsorglich ausgetauscht wird. Der Defekt kann nämlich zu gefähr­lichen Stürzen oder Kollisionen führen.

Als wäre die Kette gerissen

Pedelecs fahren nicht von alleine. Der Fahrer muss für sein Vorwärts­kommen auch selber in die Pedale treten. Doch beim Bosch Classic+ Line Pedelec-Antrieb kann der Tritt ins Leere gehen. Wie Bosch in einem Info-Blatt für Fachhändler bestätigt, rutscht der Antrieb dann durch, als wäre die Kette gerissen. Auf so etwas ist niemand gefasst. Die Kontrolle über das Elektrofahr­rad kann verloren gehen. Es drohen Stürze oder Kollisionen.

Schmier­mittel stört die Mechanik

Das Problem ist das Schmier­mittel. Es soll eigentlich die Beweglich­keit mecha­nischer Teile sicher­stellen, stört aber die Funk­tion. Wer während des Radelns mal eine Tret­pause macht und dann wieder losstrampelt, spürt zunächst keinen Widerstand. Irgend­wann greift die Mechanik wieder. Das kann nach einem kurzen Augen­blick oder auch erst nach mehreren Kurbel­umdrehungen sein. Selbst erfahrene Fahr­radfahrer werden durch solche Situationen aus der Fassung gebracht. Vor allem, wenn der Pedaleur an einer Steigung aufsteht, um im Wiegetritt berg­auf zu fahren – und mit voller Kraft ins Leere tritt.

Austausch erst im Schadenfall

Bosch empfiehlt den Fachhänd­lern, erst dann aktiv zu werden, wenn der Fahr­radfahrer beim Pedalieren Unter­brechungen spürt und das reklamiert. Ein vorsorglicher Austausch ist bislang nicht vorgesehen. test.de rät Nutzern von Fahr­rädern mit dem abge­bildeten Classic+ Line-Antrieb auf Geräusche zu achten. Knackt der Antrieb, sollte das Pedelec umge­hend zum Fahr­radhändler gebracht werden – erst recht, wenn der Antrieb „ruckelt“. Wo sich die nächste Vertretung für Bosch-angetriebene Pedelecs befindet, erfahren Sie unter der E-Mail-Adresse contact@bosch-ebike.com.

Folgende Räder aus dem E-Bike-Test der Stiftung Wartentest vom Juni 2013 haben einen Antrieb der ersten Generation: Giant Twist Elegance C1 28, Kreidler Vitality Elite VE 3, KTM Macina Eight, Pegasus Premio E8, Sinus B3- 8-G Nexus und Stevens E-Courier SX. Die Pedelecs mit Bosch-Antrieb aus dem E-Bike-Test vom Juli 2014 besitzen bereits den Antrieb der zweiten Generation, bei dem laut Anbieter dieser Fehler nicht auftritt.

Kunden müssen mit Warte­zeit rechnen

Der Austausch der Antriebs­einheit dauert laut Bosch voraus­sicht­lich fünf bis zehn Arbeits­tage. Während der Fahr­radsaison möglicher­weise etwas länger. Die Kosten für den Austausch trägt Bosch. Die Aktion ist zunächst bis Ende 2016 befristet.

Im Falle eines Unfalls

Wer sich wegen eines plötzlich durch­rutschenden Bosch-Antriebs verletzt, kann vom Hersteller des Fahr­rads oder von Bosch den Ersatz der Behand­lungs­kosten sowie die Zahlung eines angemessenen Schmerzens­gelds verlangen. Aus Sicht von test.de ist das Durch­rutschen des Pedals durch nicht richtig greifende Sperr­klinken wegen der Konsistenz der Schmierung ein Konstruktions­fehler. Tech­nische Geräte müssen so konstruiert sein, dass von ihnen im Betrieb unter normalen Umständen keine Gefahren ausgehen. Auf ein Verschulden kommt es nicht an. Hersteller müssen für Verletzungen und Schäden durch Produkt­fehler auch dann gerade­stehen, wenn ihnen kein Vorwurf zu machen ist. Einschränkungen: Sach­schäden bis 500 Euro bleiben entschädigungs­frei. Schäden am Produkt selbst erfasst die Produkthaftung von vorn­herein nicht.

Tipp: Weitere Informationen rund um das Thema E-Bikes erhalten Sie in unseren FAQ E-Bike.

opensource.com: Dynamic tracing in Linux user and kernel space

opensource.com: Dynamic tracing in Linux user and kernel space

Did you forget to insert probe points in your code? No problem. Learn how to insert them dynamically with uprobe and kprobe.

Have you ever been in a situation where you realize that you didn’t insert debug print at a few points in your code, so now you won’t know if your CPU hits a particular line of code for execution until you recompile the code with debug statements? Don’t worry, there’s an easier solution. Basically, you need to insert dynamic probe points at different locations of your source code assembly instructions.

For advanced users, kernel documentation/trace and man perf provide a lot of information about different types of kernels and user space tracing mechanisms; however, average users just want a few simple steps and an example to get started quickly. That’s where this article will help.

Programming and development
New Python content
Our latest JavaScript articles
Recent Perl posts
Red Hat Developers Blog
Let’s start with definitions.

Probe point

A probe point is a debug statement that helps explore execution characteristics of software (i.e., the execution flow and state of software data structure at the moment a probe statement executes). printk is the simplest form of probe statement and one of the fundamental tools used by developers for kernel hacking.

Static vs. dynamic probing

Because it requires recompilation of source code, printk insertion is a static probing method. There are many other static tracepoints at important locations in the kernel code that can be enabled or disabled dynamically. The Linux kernel has a few frameworks that can help a developer probe either the kernel or user space application without recompiling the source code. Kprobe is one such dynamic method of inserting probe points within kernel code, and uprobe does so within a user application.

Tracing user space using uprobe

An uprobe tracepoint can be inserted into any user space code by using either the sysfs interface or the perf tool.

Inserting uprobe using the sysfs interface

Consider the following simple test code with no print statements in which we want to insert a probe at some instruction:

[source,c]
.test.c
#include
#include
#include

static int func_1_cnt;
static int func_2_cnt;

static void func_1(void)
{
func_1_cnt++;
}

static void func_2(void)
{
func_2_cnt++;
}

int main(int argc, void **argv)
{
int number;

while(1) {
sleep(1);
number = rand() % 10;
if (number < 5)
func_2();
else
func_1();
}
}
Compile the code and find the instruction addresses you want to probe:

# gcc -o test test.c
# objdump -d test
Let's say we have the following object code on the ARM64 platform:

0000000000400620 :
400620: 90000080 adrp x0, 410000
400624: 912d4000 add x0, x0, #0xb50
400628: b9400000 ldr w0, [x0]
40062c: 11000401 add w1, w0, #0x1
400630: 90000080 adrp x0, 410000
400634: 912d4000 add x0, x0, #0xb50
400638: b9000001 str w1, [x0]
40063c: d65f03c0 ret

0000000000400640 :
400640: 90000080 adrp x0, 410000
400644: 912d5000 add x0, x0, #0xb54
400648: b9400000 ldr w0, [x0]
40064c: 11000401 add w1, w0, #0x1
400650: 90000080 adrp x0, 410000
400654: 912d5000 add x0, x0, #0xb54
400658: b9000001 str w1, [x0]
40065c: d65f03c0 ret
and we want to insert a probe at offset 0x620 and 0x644. Execute the following commands:

# echo ‘p:func_2_entry test:0x620’ > /sys/kernel/debug/tracing/uprobe_events
# echo ‘p:func_1_entry test:0x644’ >> /sys/kernel/debug/tracing/uprobe_events
# echo 1 > /sys/kernel/debug/tracing/events/uprobes/enable
# ./test&
In the first and second second echo statement above, p tells us that it is a simple probe. (A probe can be either simple or return.) func_n_entry is the name we see in the trace output. Name is an optional field; if it is not provided, we should expect a name like p_test_0x644. test is the executable binary in which we want to insert the probe. If test is not in the current directory, we need to specify path_to_test/test. The 0x620 or 0x640 is the instruction offset from the start of the program. Please note >> in the second echo statement, as we want to add one more probe. So, when we enable uprobe tracing after inserting probe points in the first two commands, it will enable all the uprobe events when we write to events/uprobes/enable. We can enable individual events, as well, by writing to the specific event file created in the events directory. Once the probe points have been inserted and enabled, we can see a trace entry whenever the probed instruction is executed.

Read the trace file to see the output:

# cat /sys/kernel/debug/tracing/trace
# tracer: nop
#
# entries-in-buffer/entries-written: 8/8 #P:8
#
# _—–=> irqs-off
# / _—-=> need-resched
# | / _—=> hardirq/softirq
# || / _–=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
test-2788 [003] …. 1740.674740: func_1_entry: (0x400644)
test-2788 [003] …. 1741.674854: func_1_entry: (0x400644)
test-2788 [003] …. 1742.674949: func_2_entry: (0x400620)
test-2788 [003] …. 1743.675065: func_2_entry: (0x400620)
test-2788 [003] …. 1744.675158: func_1_entry: (0x400644)
test-2788 [003] …. 1745.675273: func_1_entry: (0x400644)
test-2788 [003] …. 1746.675390: func_2_entry: (0x400620)
test-2788 [003] …. 1747.675503: func_2_entry: (0x400620)
We can see what task was done by what CPU and at what time it executed the probed instruction.

A return probe can also be inserted into any instruction. This will log an entry when the function with that instruction is returned:

# echo 0 > /sys/kernel/debug/tracing/events/uprobes/enable
# echo ‘r:func_2_exit test:0x620’ >> /sys/kernel/debug/tracing/uprobe_events
# echo ‘r:func_1_exit test:0x644’ >> /sys/kernel/debug/tracing/uprobe_events
# echo 1 > /sys/kernel/debug/tracing/events/uprobes/enable
Here we use r instead of p, and all the other parameters are the same. Note that we need to disable uprobe events if we want to insert a new probe point:

test-3009 [002] …. 4813.852674: func_1_entry: (0x400644)
test-3009 [002] …. 4813.852691: func_1_exit: (0x4006b0 <- 0x400644)
test-3009 [002] …. 4814.852805: func_2_entry: (0x400620)
test-3009 [002] …. 4814.852807: func_2_exit: (0x4006b8 <- 0x400620)
test-3009 [002] …. 4815.852920: func_2_entry: (0x400620)
test-3009 [002] …. 4815.852921: func_2_exit: (0x4006b8 /sys/kernel/debug/tracing/events/uprobes/enable
# echo ‘p:func_2_entry test:0x630’ > /sys/kernel/debug/tracing/uprobe_events count=%x1
# echo 1 > /sys/kernel/debug/tracing/events/uprobes/enable
# echo > /sys/kernel/debug/tracing/trace
# ./test&
Here we are printing the value of ARM64 x1 register as count= when the instruction at offset 0x630 is executed.

The output would look like this:

test-3095 [003] …. 7918.629728: func_2_entry: (0x400630) count=0x1
test-3095 [003] …. 7919.629884: func_2_entry: (0x400630) count=0x2
test-3095 [003] …. 7920.629988: func_2_entry: (0x400630) count=0x3
test-3095 [003] …. 7922.630272: func_2_entry: (0x400630) count=0x4
test-3095 [003] …. 7923.630429: func_2_entry: (0x400630) count=0x5
Inserting uprobe using perf

Always finding the offset of an instruction or function where a probe needs to be inserted is cumbersome, and it’s even more complicated to know the name of a CPU register assigned to any local variable. perf is a useful tool to help prepare and insert uprobe into any line in source code.

Besides perf, there are a few other tools, such as SystemTap, DTrace, and LTTng, that can be used for kernel and user space tracing; however, perf is fully coupled with the kernel, so it’s favored by kernel developers.

# gcc -g -o test test.c
# perf probe -x ./test func_2_entry=func_2
# perf probe -x ./test func_2_exit=func_2%return
# perf probe -x ./test test_15=test.c:15
# perf probe -x ./test test_25=test.c:25 number
# perf record -e probe_test:func_2_entry -e probe_test:func_2_exit -e probe_test:test_15 -e probe_test:test_25 ./test
As the above example shows, we can insert a probe point directly into the function start and return, a specific line number of the source file, etc. You can get a local variable printed. You can have many other options, like all the instances of a function calling (see man perf probe for details). perf probe is used to create a probe point event, then those events can be probed using perf record while executing a ./test executable. When we create a perf probe point, we can have other recording options like perf stat, and we can have many post-analysis options like perf script or perf report.

Using perf script, our output for the above example looks like:

# perf script
test 2741 [002] 427.584681: probe_test:test_25: (4006a0) number=3
test 2741 [002] 427.584717: probe_test:test_15: (400640)
test 2741 [002] 428.584861: probe_test:test_25: (4006a0) number=6
test 2741 [002] 428.584872: probe_test:func_2_entry: (400620)
test 2741 [002] 428.584881: probe_test:func_2_exit: (400620 <- 4006b8)
test 2741 [002] 429.585012: probe_test:test_25: (4006a0) number=7
test 2741 [002] 429.585021: probe_test:func_2_entry: (400620)
test 2741 [002] 429.585025: probe_test:func_2_exit: (400620 /sys/kernel/debug/tracing/events/enable
; disable kprobe events until probe points are inseted.
# echo 0 > /sys/kernel/debug/tracing/events/kprobes/enable
; clear out all the events from kprobe_events, to insure that we see output for
; only those for which we have enabled
# echo > /sys/kernel/debug/tracing/kprobe_events
; insert probe point at kfree
# echo “p kfree” >> /sys/kernel/debug/tracing/kprobe_events
; insert probe point at kfree+0x10 with name kfree_probe_10
# echo “p:kree_probe_10 kfree+0x10” >> /sys/kernel/debug/tracing/kprobe_events
; insert probe point at kfree return
# echo “r:kfree_probe kfree” >> /sys/kernel/debug/tracing/kprobe_events
; enable kprobe events until probe points are inseted.
# echo 1 > /sys/kernel/debug/tracing/events/kprobes/enable
[root@pratyush ~]# more /sys/kernel/debug/tracing/trace
# tracer: nop
#
# entries-in-buffer/entries-written: 9037/9037 #P:8
#
# _—–=> irqs-off
# / _—-=> need-resched
# | / _—=> hardirq/softirq
# || / _–=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
sshd-2189 [002] dn.. 1908.930731: kree_probe: (__audit_syscall_exit+0x194/0x218 <- kfree)
sshd-2189 [002] d… 1908.930744: p_kfree_0: (kfree+0x0/0x214)
sshd-2189 [002] d… 1908.930746: kree_probe_10: (kfree+0x10/0x214)
Inserting kprobe using perf

As with uprobe, we can use perf to insert a kprobe within the kernel code. We can insert a probe point directly into the function start and return, the specific line number of the source file, etc. We can either provide vmlinux to the -k option, or we can provide the kernel source code path to the -s option:

# perf probe -k vmlinux kfree_entry=kfree
# perf probe -k vmlinux kfree_exit=kfree%return
# perf probe -s ./ kfree_mid=mm/slub.c:3408 x
# perf record -e probe:kfree_entry -e probe:kfree_exit -e probe:kfree_mid sleep 10
Using a perf script, we see this output for the above example:

# perf script
sleep 2379 [001] 2702.291224: probe:kfree_entry: (fffffe0000201944)
sleep 2379 [001] 2702.291229: probe:kfree_mid: (fffffe0000201978) x=0x0
sleep 2379 [001] 2702.291231: probe:kfree_exit: (fffffe0000201944 <- fffffe000019f67c)
sleep 2379 [001] 2702.291241: probe:kfree_entry: (fffffe0000201944)
sleep 2379 [001] 2702.291242: probe:kfree_mid: (fffffe0000201978) x=0xfffffe01db8f6000
sleep 2379 [001] 2702.291243: probe:kfree_exit: (fffffe0000201944 <- fffffe000019f67c)
sleep 2379 [001] 2702.291249: probe:kfree_entry: (fffffe0000201944)
sleep 2379 [001] 2702.291250: probe:kfree_mid: (fffffe0000201978) x=0xfffffe01db8f6000
sleep 2379 [001] 2702.291251: probe:kfree_exit: (fffffe0000201944 <- fffffe000019f67c)
I hope that this tutorial has explained how to hack your executable code and insert a couple of probe points into them. Happy tracing!

Topics
LinuxProgramming
About the authorPratyush Anand – Pratyush is working with Red Hat as a Linux Kernel Generalist. Primarily, he takes care of several kexec/kdump issues being faced by Red Hat product and upstream. He also handles other kernel debugging/tracing/performance issues around Red Hat supported ARM64 platforms. Apart from Linux Kernel, he has also contributed in upstream kexec-tools and makedumpfile project. He is an open source enthusiast and delivers volunteer lectures in educational institutes for the promotion of FOSS.
More about me
Learn how you can contribute