xenrec

SystemDataRecorder is offering several data recorders for different jobs: overall system utilization, per-CPU, per-NIC utilization along with many others. On systems where we use virtualization in general we can monitor the guests directly or if we want more accurate numbers, we will need to monitor the host. The purpose of this short article is to show how you can use SystemDataRecorder to record Xen performance metrics.

Xen Hypervisor

Xen is an open-source type-1 or baremetal hypervisor which has the following structure: 

The Xen Hypervisor is an exceptionally lean, less than 150,000 lines of code, software layer that runs directly on the hardware and is responsible for managing CPU, memory, and interrupts. It is the first program running after the boot-loader exits. The hypervisor itself has no knowledge of I/O functions such as networking and storage.

Xen dom0

The Control Domain, or Domain 0, is a specialized Virtual Machine that has special privileges like the capability to access the hardware directly, handles all access to the system’s I/O functions and interacts with the the other Virtual Machines. It also exposes a control interface to the outside world, through which the system is controlled. The Xen hypervisor is not usable without Domain 0, which is the first VM started by the system.

Xen domU

Guest Domains/Virtual Machines are virtualized environments, each running their own operating system and applications. Xen supports two different virtualization modes: Paravirtualization (PV) and Hardware-assisted or Full Virtualization (HVM). Both guest types can be used at the same time on a single Xen system. It is also possible to use techniques used for Paravirtualization in an HVM guest: essentially creating a continuum between PV and HVM. This approach is called PV on HVM. Xen guests are totally isolated from the hardware: in other words, they have no privilege to access hardware or I/O functionality. Thus, they are also called unprivileged domain, or DomU. 

xenrec

SystemDataRecorder agents can be installed on dom0 or domU guests. To have the best visibility and accuracy we need to place all data recorders on dom0. There have been some comparative measurements between dom0 and domU and SDR, where we can see the difference between guest and host regarding data recording.

However during these measurements, one part, missing was the possibility to report per guests metrics directly from dom0. Welcome xenrec, a simple utility based on xentop , a standard Xen administrative tool.

Why xenrec ?

In short, because xentop does not record time series data, in a CSV format, simple to be consumed by systems like RRDtool or R Statistical. More xentop utility is an interactive tool, designed in general to be run from the terminal and visual check the results. You could run xentop and use other tools like awk, sed etc to parse and store data on disk. But we want something simple and easy to be used.

So, we did use xentop to record domain statistics and handle all output using Perl, with final results:

1397842672:c5932:b:2178:0.0:1048576:4.2:1049600:4.2:1:1:397715:41767:0:0:0:0:0:0:0
1397842672:c5964:b:23443:0.1:1048576:4.2:1049600:4.2:1:1:8049983:9220016:0:0:0:0:0:0:0
1397842672:c6464:b:1838:0.0:1048576:4.2:1049600:4.2:1:1:219136:5394:0:0:0:0:0:0:0
1397842672:deb7464:b:14628:0.0:1048576:4.2:1049600:4.2:1:1:1182646:411741:0:0:0:0:0:0:0
1397842672:dom0:r:431606:8.4:6656000:26.5:NA:NA:2:0:0:0:0:0:0:0:0:0:0
1397842672:lobby:b:4855:0.0:774144:3.1:775168:3.1:1:1:5462996:754207:0:0:0:0:0:0:0
1397842672:r5732:b:85953:1.4:1044452:4.2:1049600:4.2:1:2:220886:7420:0:0:0:0:0:0:0
1397842672:r5764:b:180371:2.9:1044452:4.2:1049600:4.2:1:2:210161:7228:0:0:0:0:0:0:0
1397842672:s10u8:b:13712:0.2:1044320:4.2:1049600:4.2:1:1:222133:4808:0:0:0:0:0:0:0
1397842672:sdrcom:b:1403:0.0:786432:3.1:787456:3.1:1:1:329407:370297:0:0:0:0:0:0:0
1397842672:sdrorg:b:2402:0.0:786432:3.1:787456:3.1:1:1:5307867:4998688:0:0:0:0:0:0:0
1397842672:u100432:b:1255:0.0:1048576:4.2:1049600:4.2:1:1:218320:2378:0:0:0:0:0:0:0
1397842672:u100464:b:1164:0.0:1048576:4.2:1049600:4.2:1:1:217599:1806:0:0:0:0:0:0:0
1397842672:u120464:b:74359:0.0:1048576:4.2:1049600:4.2:1:1:3183711:3990520:0:0:0:0:0:0:0
1397842672:win764:b:42255:1.8:1048548:4.2:1049600:4.2:2:2:99484:10380:0:0:0:0:0:0:0

Tip: we used xentop -b -d1 -i2, inside our Perl agent. Why not -d0 ? Take a look:

xentop, 9 columns output, delay 0 seconds

Using a delay of 0 seconds will add an overhead on the dom0 for xentop and xenrec. So we don't want that. Increasing to 1second for example will make things different:

xentop, 9 columns output, delay 1 second

Very clear that xentop -b -d1 -i2 will do nicely the job.

Performance Metrics

xenrec will record all xentop reported parameters, which unfortunatelly are not proper documented under xentop manual page nor help system. Thats another reason we wanted to add xenrec to SystemDataRecorder, better documentation.

The metrics below, using help function, -h:

xenrec, help usage funtion

We will soon add xenrec to our data recorders and make available the source code under SystemDataRecorder repository.

References

		http://wiki.xenproject.org/wiki/Xen_Overview