Benchmarking

Contents

Gelato@UNSW Work

Gelato@UNSW is undertaking a project on Ia64 LinuxScalability.

As part of this project we will be aiding the Scalability group in several areas of benchmarking, such as:

What benchmarks are relevant to Ia64 scalability.
Benchmark configuration including tuning and porting benchmarks for various Ia64 systems.
Identifying individual benchmark compatibility issues with respect to Ia64.
Maintaining a repository of patches that resolve any issues found above.
Result extraction, and interpretation.
Results will be hosted on the Gelato Scalability Wiki.

The following is a list of Benchmarks that have been identified as being useful for general benchmarking.

SPECweb99
lmbench - lmbench2
lmbench3 - lmbench3
reaim7 -
OSDL DB benchmark -
NFSSTONE - Measures network file system performance, independent of the underlying protocols and file system.
LTP - (Not really a benchmark) Linux Test Project, the aim of this project is to provide a common test suite for the Linux operating system.
IRQLatency -- ways to stress the system while performing latency measurements.

hackbench and debench are useful for finding worse-case latencies (see also IRQLatency)

HPC Benchmarks
- Stream - Measures data transfer bandwidth between a processor and main memory.
- Ptrans - Measures the communication speed between individual processors in a multiprocessor system.
- b_eff - Measures the response time and bandwidth of a data network.
- DGEMM - Measures matrix multiplications NOTE: For a reason, give it!.

NetworkBenchmarking : resources on network benchmarking.
Bonnie++ - Is a disk performance benchmark.
NHFSSTONE - An alternative to NFSSTONE, using the Remote Procedure Call(RPC) interface instead of the standard file system interface.

Run times are calculated on a dual 1500Mhz Itianium2 rx2600 running a 2.6.8-rc2 SMP Linux kernel.

Benchmark	Setup	Approx. Setup Time	Run Command	Approximate Runtime	Extract Results	Requirements
NFSSTONE	unpack, compile, run	1m	./nfsstone <path to nfs> <path to lock file>	0m 3.645869s	printed to stdout	Network file system access
Stream	unpack, make, run	1m	This script will run 10 iterations of all the gcc and icc generated binaries and pipe the output to file and stdout	1m 21s	printed to stdout	Intel's C compiler would be nice.
reaim7	unpack, bootstrap, configure, make, run	2m	reaim -c src/reaim.config -f src/workfile.alltests -g -i 10 -r 10	180m10.151s	file	Can be run from user space
LTP	unpack, make, run		export TOUTPUT=CONDENSE; sh runalltests.sh -o `uname -r`-ltp-01-output -l `uname -r`-ltp-01-log	55m 12.441s	file and stdout	root access to run all test cases, if root access is not available run times will be shorter.
LMBench2 Scale	unpack, make		make results; make rerun; make rerun; make rerun; make rerun	8m51.978s (single rerun)	cd results && make summary percent LIST=ia64-linux-gnu-first-run/*	none
lmbench3	unpack, make		make results; make rerun; make rerun; make rerun; make rerun	222m37.013s (single rerun)	cd results && make summary percent LIST=ia64-linux-gnu-first-run/*	none
SPECweb99			Start http server on system under test, . ./shrc, ./client -d, specperl manager	105m (defualt run)		http server with cgi access, user access to a cgi_bin

Benchmark	Setup	Run Command	Approximate Runtime	Extract Results	Requirements
OSDL-DBT-1	see WiKi page				postgresql, postgresql development libraries and headers, user and group pgsql with password pgsql
OSDL DB benchmark	see WiKi page				postgresql, postgresql development libraries and headers, user and group pgsql with password pgsql

These requirements are a base to get all the benchmarks to run on an external machine without root privileges.
Since root is not a requirement not all benchmarks can be run or Linux Test Project tests be performed.

There are several consideration required when tuning and configuring a system for benchmarking:

The TuningBenchmarks page has more information on these issues.

Reporting your data correctly is very important

Gelato@UNSW is sponsored by

Please contact us with any questions or comments.